Skip to content

Configure CNI Providers#

Information is provided herein on how to configure the Container Network Interfaces that MKE 4 supports.

Provider list#

The network.providers field in the mke4.yaml configuration file is a list of CNI provider configurations. Each entry supports the following fields:

Field Description Values
provider The CNI provider name. calico, custom
enabled Determines whether this provider is the active CNI for the cluster. true/false
extraConfig Provider-specific configuration, as a map of string keys to string values. Map of string to string

Important

  • At most one provider in the list can be enabled: true at a time.
  • The active CNI provider is immutable after the cluster is deployed.
  • When the list is empty, or every entry has enabled: false, MKE 4 resolves the CNI to custom: no CNI is managed by MKE, and you must deploy one yourself before the cluster becomes operational.

Supported providers:

  • calico — Managed by MKE 4 through the Tigera Operator.
  • custom — Unmanaged; you install and manage the CNI yourself.

Calico OSS#

Info

Starting with MKE 4.2.0, all management components must be pinned to the control plane nodes. In your Calico OSS Installation specification, include controlPlaneNodeSelector to restrict such components as calico-kube-controller and calico-typha to control plane nodes only:

controlPlaneNodeSelector:
  node-role.kubernetes.io/control-plane: ""

The following table includes details on the configurable extraConfig settings for the Calico provider.

Field                           Description             Values                   Default
cidrV4 Sets the IP pool in the Kubernetes cluster from which Pods are allocated. This is also the cluster's pod CIDR. Valid IPv4 CIDR 192.168.0.0/16 You can easily modify cidrV4 prior to cluster deployment. Contact Mirantis Support, however, if you need to modify cidrV4 once your cluster has been deployed.
mode Sets the Calico encapsulation type. Not case-sensitive. vxlan, ipip, empty vxlan
overlay Sets when encapsulation is applied. Case-sensitive. Always, CrossSubnet, Never Always
vxlanVNI Sets the VXLAN Network Identifier. Integer, as a string 10000
vxlanPort Sets the UDP port used for VXLAN. Integer, as a string 4789
loglevel Sets the log level for the CNI components. Any Calico log severity is accepted; MKE 4 does not restrict this value. For example Info, Debug, Warning, Error Info
linuxDataplane Sets the dataplane for Calico CNI. This is the only dataplane supported through extraConfig; to use the BPF dataplane, use values.yaml instead. Iptables Iptables
ipautodetect Sets the method Calico uses to autodetect each node's IPv4 address. first-found, can-reach=<ip>, interface=<regex>, skip-interface=<regex>, cidr=<cidr>[,<cidr>...], kubernetes-internal-ip Not set
values.yaml Advanced: full Tigera Operator Helm values, as YAML. When set, the individual extraConfig parameters above are ignored, except that the pod CIDR falls back to cidrV4 if the supplied values define no IPv4 IP pool. Refer to Using values.yaml below. YAML Not set

Info

Calico clusters upgraded from MKE 3 use values.yaml. Refer to Using values.yaml below.

Encapsulation: the mode and overlay values combine as follows. BGP is enabled for the IP-in-IP encapsulations and disabled for all others.

mode overlay Encapsulation BGP
vxlan Always VXLAN Disabled
vxlan CrossSubnet VXLAN, cross-subnet only Disabled
vxlan Never None Disabled
ipip Always IP-in-IP Enabled
ipip CrossSubnet IP-in-IP, cross-subnet only Enabled
ipip Never None Disabled
Empty Any None Disabled

Using values.yaml#

The default network configuration described herein offers a serviceable, low maintenance solution. If, however, you want more control over your network configuration environment, MKE 4 exposes maximal configuration for the Calico CNI through which you can configure your networking to the fullest extent allowed by the provider. For this, you will use the values.yaml key, in which case an example networking would resemble the following:

network:
   cplb:
     disabled: true
   kubeProxy:
     iptables:
       minSyncPeriod: 0s
       syncPeriod: 0s
     ipvs:
       minSyncPeriod: 0s
       syncPeriod: 0s
       tcpFinTimeout: 0s
       tcpTimeout: 0s
       udpTimeout: 0s
     metricsBindAddress: 0.0.0.0:10249
     mode: iptables
     nftables:
       minSyncPeriod: 0s
       syncPeriod: 0s
   multus:
     enabled: false
   nllb:
     disabled: true
   nodePortRange: 32768-35535
   serviceCIDR: 10.96.0.0/16
   providers:
   - enabled: true
     extraConfig:
       loglevel: Info
       values.yaml: |-
         installation:
           kubeletVolumePluginPath: <mke4.yaml kubelet.kubeletRootDir setting>
           logging:
             cni:
               logSeverity: Debug
           cni:
             type: Calico
           calicoNetwork:
             linuxDataplane: Iptables
             ipPools:
             - cidr: 192.168.0.0/15
               encapsulation: VXLAN
         resources:
           requests:
             cpu: 250m
         defaultFelixConfiguration:
           enabled: true
           wireguardEnabled: false
           wireguardEnabledV6: false
     provider: calico
   - enabled: false
     provider: custom

Important

  • You must choose whether to specify an exact YAML specification for the Helm installation of Tigera Operator during the initial cluster installation.
  • Set installation.kubeletVolumePluginPath to the cluster's kubelet.kubeletRootDir setting (refer to kubelet custom profiles), or to None. Do not set the root-level kubeletVolumePluginPath key instead.
  • values.yaml accepts at most one IPv4 IP pool under installation.calicoNetwork.ipPools; the pod CIDR is taken from that pool, or falls back to cidrV4 when no pool is supplied.
  • Omit tigeraOperator.version from values.yaml, as any version inserted will be rejected. For more information, refer to Tigera Operator version selection.

Info

Refer to the official Tigera Operator documentation for:

You can view the full values.yaml specification for the Helm chart needed to install Tigera Operator at the Project Calico GitHub.

The network configuration generated as a result of upgrading to MKE 4 from an existing MKE 3 cluster always uses YAML. As such clusters have at least one existing IP pool, however, the CIDR and dataplane values are specified outside of the YAML, as illustrated below:

network:
  cplb:
    disabled: true
  kubeProxy:
    iptables:
      minSyncPeriod: 0s
      syncPeriod: 0s
    ipvs:
      minSyncPeriod: 0s
      syncPeriod: 0s
      tcpFinTimeout: 0s
      tcpTimeout: 0s
      udpTimeout: 0s
    metricsBindAddress: 0.0.0.0:10249
    mode: iptables
    nftables:
      minSyncPeriod: 0s
      syncPeriod: 0s
  multus:
    enabled: false
  nllb:
    disabled: true
  nodePortRange: 30000-32768
  serviceCIDR: 10.96.0.0/16
  providers:
  - enabled: true
    extraConfig:
      cidrV4: 192.168.0.0/15
      linuxDataplane: Iptables
      loglevel: DEBUG
      values.yaml: |-
        installation:
          kubeletVolumePluginPath: None
          registry: ghcr.io/mirantiscontainers/
          cni:
            type: Calico
          calicoNetwork:
            bgp: Disabled
            linuxDataplane: Iptables
        resources:
          requests:
            cpu: 250m
        tigeraOperator:
          registry: ghcr.io/mirantiscontainers/
        defaultFelixConfiguration:
          enabled: true
          bpfConnectTimeLoadBalancing: TCP
          bpfHostNetworkedNATWithoutCTLB: Enabled
          bpfLogLevel: Debug
          floatingIPs: Disabled
          logSeverityScreen: Debug
          logSeveritySys: Debug
          reportingInterval: 0s
          vxlanPort: 4789
          vxlanVNI: 10037
    provider: calico
  - enabled: false
    provider: custom

Info

  • MKE 4 uses a static port range for Kubernetes NodePorts, from 32768 to 35535.
  • Following a successful MKE 3 to MKE 4 upgrade, a list displays that details the ports that no longer need to be opened on manager or worker nodes. These ports can be blocked.
  • Once set in the network configuration, the serviceCIDR value is immutable.

Tip

Refer to kubelet custom profiles for information on the kubelet.kubeletRootDir setting.

Tigera Operator version selection#

MKE 4 selects the Tigera Operator version based on the MKE build. That operator version then fixes the version of every Calico component: calico-node, Typha, the Calico API server, and the Calico controllers. Pinning the operator version in your configuration therefore pins the entire Calico data plane, not just the operator.

Starting with MKE 4.2.1, MKE rejects any Calico values.yaml that sets tigeraOperator.version. Two states are possible:

  • Key absent (supported). MKE tracks the Tigera Operator version internally and updates it automatically on every MKE upgrade. No edit to your configuration is required.
  • Key present (rejected). MKE rejects any value, including the version your current MKE build ships. A pin that matches today's version is rejected for the same reason a stale pin is: the next MKE upgrade moves the shipped Tigera Operator version, and the pin then holds the operator and every Calico component at the old version.

tigeraOperator.registry is unaffected. Set it as needed — for example, to point air-gapped clusters at a registry mirror.

Fix a rejected configuration#

Delete only the version line, making sure to keep the rest of the tigeraOperator block:

# tigeraOperator.version must be absent -- MKE owns that version
tigeraOperator:
  registry: ghcr.io/mirantiscontainers/

Identify pre-flight check failure#

The mkectl apply command fails this pre-flight check, before any host is modified:

pre-flight checks failed (1):
  ✗ tigera-operator version:
      invalid Calico CNI configuration: tigeraOperator.version must not be set (found "v1.37.1"): remove the line -- MKE derives the tigera-operator version from the chart it ships (currently v1.40.15)

The version in the message reflects your MKE build. It is informational, and not a value to copy back into values.yaml.

To resolve the failure, delete the version line from values.yaml.

Notes

  • The check runs on every mkectl apply, and not only at installation.
  • The same rule applies to child clusters through MkeChildConfig, enforced both when a child configuration is created and when it is updated. An update can carry a pin that the child configuration already had, so a child created before MKE 4.2.1 remains editable while it waits to be upgraded. An update that changes the MKE version of the child cluster removes the pin automatically when the child cluster is being upgraded from MKE 4.2.0 or earlier. From MKE 4.2.1 or later, a pin that is still present is rejected. Note that MkeChildConfig uses a single spec.network.provider field and not a providers list.
  • The mkectl node add and mkectl node remove commands are exempt from this check.
  • Upgrading from MKE 4.2.0 or earlier removes an existing pin from the configuration stored in the cluster for you. A local mke4.yaml that still carries the line is a separate case: the next mkectl apply from that file is rejected, so delete the line, or run mkectl config get to obtain a refreshed copy.