COSCUP2018
x
Outline
What network functions about k8s
How does above function implement
What is the challenge about k8s

COSCUP2018
x
What network function kubernetes
providers?

COSCUP2018
x
What
Container Network
○ Connectivity
○ DNS
Kubernetes services

COSCUP2018
x
Do You Know How Container Works?

COSCUP2018
x
Containersvs.VMs
Containers are isolated, but share OS
and where appropriate bins/libraries

COSCUP2018
x
HowDockerWorks
We know docker is isolated, but how
does it works?
Linux kernel support the Namespaces
mechanisms to partition kernel
resources to different processes

COSCUP2018
x
HowDockerWorks
Mount namespaces
IPC namespaces
PID namespaces
Network namespaces
User namespaces
UTS namespaces
○ Unix Time System

COSCUP2018
x
NetworkNamespace
Isolate the network functions.
Including the
○ Network interfaces
○ Routing rules
○ Netfilter (iptables)

COSCUP2018
x
Linux Host

COSCUP2018
x
docker0
Linux Host Linux Host

COSCUP2018
x
docker0 docker0
ns1
Linux Host Linux Host Linux Host

COSCUP2018
x
docker0 docker0
docker0
ns1
ns1
veth1
veth0
Linux Host

COSCUP2018
x
docker0 docker0
docker0docker0
ns1
ns1ns1
vth1
vth0
veth
eth0
Linux HostLinux Host

COSCUP2018
x
Before we talk about service, we must
know why service exist.

COSCUP2018
x
Pods/Deployments
We can deploy our applications as a
containers in the kubernetes.
There’re many kind of the container
we can deploy
○ Pod
○ Deployment
○ Statefulset
○ DaemonSet

COSCUP2018
x
Deployment
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
Deployment:
○ Ngnix
○ Replica: 3
10.123.234.56 10.123.234.57 10.123.234.58

COSCUP2018
x
Access
How we application access those
Nginx servers?
By IP address
○ 10.123.234.56:80
○ 10.123.234.57:80
○ 10.123.234.58:80
What’s the problem

COSCUP2018
x
Deployment
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
Deployment:
○ Ngnix
○ Replica: 3
10.123.234.56 10.123.234.57 10.123.234.75

COSCUP2018
x
Access
How we application access those
Nginx servers?
By IP address
○ 10.123.234.56:80
○ 10.123.234.57:80
○ 10.123.234.58:80
○ 10.123.234.75:80
It’s not easy for our application to
handle those ip-changed situation.

COSCUP2018
x
The Service is used to solve this problem.

COSCUP2018
x
Service
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
10.123.234.56 10.123.234.57 10.123.234.58
App
Service Nginx

COSCUP2018
x
Service
Application to Service
○ We use the DNS to access the service.
○ $(service).$(namespace).cluster.local
Service to Pods
○ Service maintains all IP addresses of all
Pods.
○ We call it endpoints

COSCUP2018
x
Service
Node1
Nginx
Node2
Nginx
Node3
Nginx
Kubernetes Cluster
10.123.234.56 10.123.234.57 10.123.234.58
App
Service Nginx
nginx.default
endpoints

COSCUP2018
x
How does kubernetes implements those
functions?

COSCUP2018
x
What
Container Network Connectivity
○ Container Network Interface(CNI)
Kubernetes Services
○ There’re many implementation we can
choose

COSCUP2018
x
ContainerNetworkInterface

COSCUP2018
x
Cloud Native Computing Foundation
Project
Consists of a specification and
libraries.
Configure network interfaces in Linux
containers
Concerns itself only with network
connectivity of containers
○ Create/Remove

COSCUP2018
x
Removing allocated resources when
the container is deleted

COSCUP2018
x
Who is using CNI?

COSCUP2018
x
FromtheGITHUB
l rkt - container engine
l Kubernetes - a system to simplify container operations
l OpenShift - Kubernetes with additional enterprise
features
l Cloud Foundry - a platform for cloud applications
l Apache Mesos - a distributed systems kernel
l Amazon ECS - a highly scalable, high performance
container management service

COSCUP2018
x
So, How to use the CNI?

COSCUP2018
x
StepbyStep
Create a kubernetes cluster
Setup your CNI plugin
Deploy your first Pod

COSCUP2018
x
Just follow the installation to install the
kubernetes

COSCUP2018
x
How do we install the CNI?

COSCUP2018
x

COSCUP2018
x
Handbyhand
In the kubelet, we have the following
parameters for CNI.
--cni-bin-dir
○ /opt/cni/bin
--cni-conf-dir
○ /etc/cni/net.d/
We should config the CNI for every
k8s nodes.

COSCUP2018
x
Let Deploy a Pod

COSCUP2018
x
Beforewestart
Pod
○ A collection of containers

COSCUP2018
x
Steps
Load the Pod config
○ Multiple containers
Find a node to deploy the pod
Create a Pause container

COSCUP2018
x
Linux Host
Pause
Container

COSCUP2018
x
Steps
Load the Pod config
Load the CNI config

COSCUP2018
x
Linux Host
Pause
Container
Load the CNI config from /etc/cni/net.d/…

COSCUP2018
x
Steps
Load the Pod config
Load the CNI config
Execute the CNI

COSCUP2018
x
Callthebinary
Load the binary from the config
Find the binary from the /opt/cni/bin/

COSCUP2018
x
Linux Host
Pause
Container
Call the /opt/cni/bin/flannel
Network
Connectivity

COSCUP2018
x
Steps
Load the Pod config
Load the CNI config
Execute the CNI
Create target containers and attach to
Pause container

COSCUP2018
x
Linux Host
Pause
Container
Network
Connectivity
Busybox

COSCUP2018
x
Attachtoothercontainer
docker run –net=…
○ bridge
○ host
○ containerID

COSCUP2018
x
Linux Host
Pause
Container
Network
Connectivity
Busybox
Pod

COSCUP2018
x
Kubernetesservice
There’re three implementation now.
○ User-space Kube-Proxy
○ Kernel-space iptables (default)
○ Kernel-space ipvs
We use the iptables to explain how
service(ClusterIP) works

COSCUP2018
x
LAB
Get the Service
○ kubectl get service

COSCUP2018
x
LAB
Get the endpoints
○ kubectl get endpoints

COSCUP2018
x
LAB
Get the pod ip address
○ kubectl get pods –o wide

COSCUP2018
x
Now.Trytofetchthenginx
Service name is k8s-nginx-cluster
Use the nslookup to lookup the IP
○ nslookup k8s-nginx-cluster
○ nslookup k8s-nginx-cluster.default
■ default is the namespace of the service

COSCUP2018
x
Trytofetchthenginx
In the pod: curl k8s-nginx-cluster

COSCUP2018
x
Howitworks?
First, we can get the VIP from the
hostname.
○ It’s just a DNS request.
Second, we can access the nginx from
that VIP address.
○ iptables!!

COSCUP2018
x
Lte’swatchtheiptables
First, we can use the service name to
filter the iptables rules.
sudo iptables-save | grep ”k8s-
nginx-cluster”

COSCUP2018
x
Lte’swatchtheiptables
Remember? There’re three endpoints
for the service now.

COSCUP2018
x
workflowsPackets
Packets
Packets
Match Services’
ClusterIP
Find the
endpoints
DNAT
KUBE-SERVICES
KUBE-SVC-XXXX
KUBE-SEP-XXXX
Enter iptables PREROUTING
Jump to other chain Jump to other chain
Jump to other chain

COSCUP2018
x
Howdowechoosewhichonetouse?
When we match the clusterIP:Port,
goto another custom chain.
○ 10.105.100.214:80

COSCUP2018
x
Use the random to choose what
endpoint we use.

COSCUP2018
x
EP1 EP2 EP3
P < 0.33
P < 0.5
EP1
EP2 EP3
P= 1/3
P= 2/3 * 1/2
= 1/3
P= 2/3 * 1/2
= 1/3

COSCUP2018
x
EP1 EP2 EP3
P < 0.2
P < 0.25
EP1
EP2
EP3
EP4 EP5
P < 0.33
P < 0.5
EP4 EP5
P = 0.2
P = 4/5 * 1/4
= 1/5
P = 4/5 * 3/4 * 1/3
= 1/5
P = 4/5 * 3/4 * 2/3 * 1/2
= 1/5
P = 4/5 * 3/4 * 2/3 * 1/2
= 1/5

COSCUP2018
x
K8S create a custom chain for each
endpoints.
First rule is a SNAT
○ The Ngnix want to access outside.
Second is DNAT
○ Change the IP to one of the endpoints

COSCUP2018
x
Now, We Know The Basic Function Of
Kubernetes Network.

COSCUP2018
x
What is the next step of kubernetes
network

COSCUP2018
x
challenges
For different use cases
○ 5G/NFV/IoT
Network features
○ High performance
○ Low latency
Network infrastructure
○ Multiple network
■ Separate the data/control network.

COSCUP2018
x
Networkfeaturesc
We use the hardware/smart NIC for
those requirements before.
We also have some mechanism in the
software.
○ DPDK
○ SR-IOV
○ …
○ etc

COSCUP2018
x
Network Interface Card
Linux Kernel
Network Stack
Network Driver
Application
Network Interface Card
Linux Kernel
Network Stack
Network Driver
Application
Kernel Space
User Space
DPDK

COSCUP2018
x
How integrate those with kubernetes?

COSCUP2018
x
CNI
We use some CNI for those functions.
Intel had developed the CNI for those
functions.
○ Call sriov-cni
○ https://github.com/intel/sriov-cni

COSCUP2018
x
Node1 Node2 Node3
PodA PodA PodA PodB PodA PodB
Flannel
(Control Network)
br0 br0 br0
Data Network
(192.168.0.0/16)

COSCUP2018
x
Problem
For those container using the
DPDK/SR-IOV, it can’t use any
kubernetes service now.
Since the network function is handled
by DPDK/SR-IOV now.
How to solve this?

COSCUP2018
x
Multus
There’s a discussion in the github
about that requirement.
Intel develop a CNI plugin to support
multiple network for a Pod.
○ It’s called Multus CNI
Multus call CNIs one by one.

COSCUP2018
x
Pod
Container
eth0
Pod
Container eth0
eth1
eth2
Previous What Multus Provide

COSCUP2018
x
,
,
You need to create first

COSCUP2018
x
Node1 Node2 Node3
PodA PodA PodA PodB PodA PodB
Flannel
(Control Network)
br0 br0 br0
Data Network
(192.168.0.0/16)
Data Network
(10.56.10/24)

COSCUP2018
x
Q&A

Overview of kubernetes network functions

In this document