Installing an unmanaged CNI plugin

Installing an unmanaged CNI plugin

For MKE, Calico provides the secure networking functionality for container-to-container communication within Kubernetes. MKE handles the lifecycle of Calico and packages it with MKE installation and upgrade. Additionally, the Calico deployment included with MKE is fully supported with Docker providing guidance on the container network interface (CNI) components.

At install time, MKE can be configured to install an alternative CNI plugin to support alternative use cases. The alternative CNI plugin may be certified by Docker and its partners, and published on Docker Hub. MKE components are still fully supported by Docker and respective partners. Docker will provide pointers to basic configuration, however for additional guidance on managing third-party CNI components, the platform operator will need to refer to the partner documentation or contact that third party.

MKE does not manage the version or configuration of alternative CNI plugins. MKE upgrade will not upgrade or reconfigure alternative CNI plugins. To switch between managed and unmanaged CNI plugins or vice versa, you must uninstall and then reinstall MKE.

Install an unmanaged CNI plugin on MKE

Once a platform operator has complied with MKE system requirements and taken into consideration any requirements for the custom CNI plugin, you can run the MKE install command with the --unmanaged-cni flag to bring up the platform.

This command will install MKE, and bring up components like the user interface and the RBAC engine. MKE components that require Kubernetes Networking, such as Metrics, will not start and will stay in a Container Creating state in Kubernetes until a CNI is installed.

Enable an unmanaged CNI for Windows Server nodes

When MKE is installed with –unmanaged-cni, the ucp-kube-proxy-win container on Windows nodes will not fully start, instead, logging the following suggestion in a loop:

example : [System.Environment]::SetEnvironmentVariable("CNINetworkName", "ElangoNet", [System.EnvironmentVariableTarget]::Machine)
example : [System.Environment]::SetEnvironmentVariable("CNISourceVip", "192.32.31.1", [System.EnvironmentVariableTarget]::Machine)

This occurs because kube-proxy needs more information to program routes for Kubernetes services. You have two options:

  • Deploy your own kube-proxy along with the CNI, as implemented by this manifest documented in the Kubernetes 1.18 Windows Install Guide.

  • If using a VXLAN-based CNI, set the the following variables:

    • `CNINetworkName` should match the name of the Windows Kubernetes HNS

      network, which you should be able to find in the installation documentation for the third party CNI, or via hnsdiag list networks.

    • `CNISourceVip` is the value of the source VIP for this node, which should be identifiable from the installation documentation for the third party CNI. Since Source VIP will be different for each node and may change across host reboots, it may be best to set this variable using a utility script.

      Here’s an example of setting these variables using powershell:

    [System.Environment]::SetEnvironmentVariable("CNINetworkName", "vxlan0", [System.EnvironmentVariableTarget]::Machine)
    
    [System.Environment]::SetEnvironmentVariable("CNISourceVip", "192.32.31.1", [System.EnvironmentVariableTarget]::Machine)
    

Install MKE without a CNI plugin

Once connected to a manager node with MCR installed, you are ready to install MKE with the --unmanaged-cni flag.

docker container run --rm -it --name ucp \
  -v /var/run/docker.sock:/var/run/docker.sock \
  mirantis/ucp:3.4.4 install \
  --host-address <node-ip-address> \
  --unmanaged-cni \
  --interactive

Once the installation is complete, you can access MKE from a web browser. Note that the manager node will be unhealthy as the kubelet will report NetworkPluginNotReady. Additionally, the metrics in the MKE dashboard will also be unavailable, as this runs in a Kubernetes pod.

Configure CLI access to MKE

Next, a platform operator should log in to MKE, download a MKE client bundle, and configure the Kubernetes CLI tool, kubectl.

With kubectl, you can see that the MKE components running on Kubernetes are still pending, waiting for a CNI driver before becoming available.

$ kubectl get nodes
NAME         STATUS     ROLES     AGE       VERSION
manager-01   NotReady   master    10m       v1.11.9-docker-1
$ kubectl get pods -n kube-system -o wide
NAME                           READY     STATUS              RESTARTS   AGE       IP        NODE         NOMINATED NODE
compose-565f7cf9ff-gq2gv       0/1       Pending             0          10m       <none>    <none>       <none>
compose-api-574d64f46f-r4c5g   0/1       Pending             0          10m       <none>    <none>       <none>
kube-dns-6d96c4d9c6-8jzv7      0/3       Pending             0          10m       <none>    <none>       <none>
ucp-metrics-nwt2z              0/3       ContainerCreating   0          10m       <none>    manager-01   <none>

Install an unmanaged CNI plugin

You can usekubectl to install a custom CNI plugin on MKE. Alternative CNI plugins are Weave, Flannel, Canal, Romana and many more. Platform operators have complete flexibility on what to install, but Docker will not support the CNI plugin.

The steps for installing a CNI plugin typically include:

  1. Downloading the relevant upstream CNI binaries.

  2. Placing them in /opt/cni/bin.

  3. Downloading the relevant CNI plugin’s Kubernetes Manifest YAML file.

  4. Running $ kubectl apply -f <your-custom-cni-plugin>.yaml

Follow the CNI plugin documentation for specific installation instructions.

Note

While troubleshooting a custom CNI plugin, you may wish to access logs within the kubelet. Connect to a MKE manager node and run $ docker logs ucp-kubelet.

Verify the MKE installation

Upon successful installation of the CNI plugin, the related MKE components should have a Running status as pods start to become available.

$ kubectl get pods -n kube-system -o wide
NAME                           READY     STATUS    RESTARTS   AGE       IP            NODE         NOMINATED NODE
compose-565f7cf9ff-gq2gv       1/1       Running   0          21m       10.32.0.2     manager-01   <none>
compose-api-574d64f46f-r4c5g   1/1       Running   0          21m       10.32.0.3     manager-01   <none>
kube-dns-6d96c4d9c6-8jzv7      3/3       Running   0          22m       10.32.0.5     manager-01   <none>
ucp-metrics-nwt2z              3/3       Running   0          22m       10.32.0.4     manager-01   <none>
weave-net-wgvcd                2/2       Running   0          8m        172.31.6.95   manager-01   <none>

Note

The above example deployment uses Weave. If you are using an alternative CNI plugin, look for the relevant name and review its status.