Use Multus CNI to create multi-homed Pods

Available since MKE 3.7.0

In Kubernetes, by default, a Pod is only connected to a single network interface, which is the default network. Using Multus CNI, however, you can create a multi-home Pod that has multiple network interfaces.

The following example procedure attaches two network interfaces to a Pod, net1 and net2.

  1. Configure kubectl for your MKE cluster.

  2. Enable Multus CNI in the MKE cluster when you install MKE, using the --multus-cni flag with the MKE install CLI command.

  3. Install a different CNI plugin.

    Run the following command on all nodes in the cluster:

    CNI_PLUGIN_VERSION=v1.3.0
    CNI_ARCH=amd64
    curl -sL
    https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGIN_VERSION}/cni-plugins-linux-${CNI_ARCH}-${CNI_PLUGIN_VERSION}.tgz
    | sudo tar xvz -C /opt/cni/bin/
    
  4. Determine the primary network interface for the node. You will need this information to create the NetworkAttachmentDefinitions file.

    Note

    The name of the primary interface can vary with the underlying network adapter.

    Run the following command on the nodes locate the default network interface. The Iface column in the line with destination default indicates which interface to use.

    route
    

    Note

    eth0 is the primary network interface in most Linux distributions.

    Example output:

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref  Use Iface
    default         ip-172-31-32-1. 0.0.0.0         UG    100    0    0   ens3
    172.17.0.0      0.0.0.0         255.255.0.0     U     0      0    0   docker0
    172.18.0.0      0.0.0.0         255.255.0.0     U     0      0    0   docker_gwbridge
    172.19.0.0      0.0.0.0         255.255.0.0     U     0      0    0   br-05e6200d101b
    172.31.32.0     0.0.0.0         255.255.240.0   U     0      0    0   ens3
    ip-172-31-32-1. 0.0.0.0         255.255.255.255 UH    100    0    0   ens3
    192.168.219.0   0.0.0.0         255.255.255.192 U     0      0    0   *
    

    Important

    Run all ensuing commands on the machine upon which kubectl is configured.

  5. Create NetworkAttachmentDefinitions to specify other networks.

    cat <<EOF | kubectl create -f -
    apiVersion: "k8s.cni.cncf.io/v1"
    kind: NetworkAttachmentDefinition
    metadata:
    name: macvlan-conf
    spec:
    config: '{
          "cniVersion": "0.3.0",
          "type": "macvlan",
          "master": "eth0",
          "mode": "bridge",
          "ipam": {
          "type": "host-local",
          "subnet": "192.168.1.0/24",
          "rangeStart": "192.168.1.200",
          "rangeEnd": "192.168.1.216",
          "routes": [
             { "dst": "0.0.0.0/0" }
          ],
          "gateway": "192.168.1.1"
          }
       }'
    EOF
    
  6. Create a multi-homed Pod:

    cat <<EOF | kubectl create -f -
    apiVersion: v1
    kind: Pod
    metadata:
    name: samplepod
    annotations:
       k8s.v1.cni.cncf.io/networks: macvlan-conf
    spec:
    containers:
       - name: samplepod
       command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
       image: alpine
    EOF
    
  7. Check the network interfaces of the Pod:

    kubectl exec -it samplepod -- ip a
    

    Example output:

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
       inet 127.0.0.1/8 scope host lo
          valid_lft forever preferred_lft forever
    3: eth0@if490: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
       link/ether 9e:02:e2:cf:4f:7e brd ff:ff:ff:ff:ff:ff
       inet 192.168.58.70/32 scope global eth0
          valid_lft forever preferred_lft forever
    4: net1@if492: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
       link/ether 0e:b0:45:13:94:f7 brd ff:ff:ff:ff:ff:ff
       inet 192.168.1.200/24 brd 192.168.1.255 scope global net1
          valid_lft forever preferred_lft forever
    

    Interface

    Description

    lo

    A loopback interface.

    eth0

    Default network interface of the Pod.

    net1

    New interface created with the macvlan configuration.

  8. Add multiple additional interfaces:

    cat <<EOF | kubectl create -f -
    apiVersion: v1
    kind: Pod
    metadata:
    name: samplepod-multi
    annotations:
       k8s.v1.cni.cncf.io/networks: macvlan-conf,macvlan-conf
    spec:
    containers:
       - name: samplepod
       command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
       image: alpine
    EOF
    

    The operation results in the addition of the net1 and net2 network interfaces to the Pod.