Improve network performance with Route Reflectors

MKE uses Calico as the default Kubernetes networking solution, configured to create a BGP mesh between all nodes in the cluster.

Note

MKE deployments running on Microsoft Azure use Azure SDN for multi-host networking, rather than Calico.

Networking performance decreases as you add more nodes to the cluster. If your cluster has more than 100 nodes, you should reconfigure Calico to use Route Reflectors rather than a node-to-node mesh.

Select dedicated nodes

For production-grade systems, deploy at least two Route Reflectors, with each running on a dedicated node that is not running any other workloads.

If Route Reflectors are running on a same node as other workloads, swarm ingress and NodePorts may not work in these workloads. As such, Mirantis suggests that you taint the nodes to ensure that they do not run other workloads.

  1. For each dedicated node, run:

    kubectl taint node <node-name> \
    com.docker.ucp.kubernetes.calico/route-reflector=true:NoSchedule
    
  2. Add labels to the newly tainted nodes:

    kubectl label nodes <node-name> \
    com.docker.ucp.kubernetes.calico/route-reflector=true
    

Deploy the Route Reflectors

  1. Create a calico-rr.yaml file with the following content:

    kind: DaemonSet
    apiVersion: apps/v1
    metadata:
      name: calico-rr
      namespace: kube-system
      labels:
        app: calico-rr
    spec:
      updateStrategy:
        type: RollingUpdate
      selector:
        matchLabels:
          k8s-app: calico-rr
      template:
        metadata:
          labels:
            k8s-app: calico-rr
          annotations:
            scheduler.alpha.kubernetes.io/critical-pod: ''
        spec:
          tolerations:
            - key: com.docker.ucp.kubernetes.calico/route-reflector
              value: "true"
              effect: NoSchedule
          hostNetwork: true
          containers:
            - name: calico-rr
              image: calico/routereflector:v0.6.1
              env:
                - name: ETCD_ENDPOINTS
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: etcd_endpoints
                - name: ETCD_CA_CERT_FILE
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: etcd_ca
                # Location of the client key for etcd.
                - name: ETCD_KEY_FILE
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: etcd_key # Location of the client certificate for etcd.
                - name: ETCD_CERT_FILE
                  valueFrom:
                    configMapKeyRef:
                      name: calico-config
                      key: etcd_cert
                - name: IP
                  valueFrom:
                    fieldRef:
                      fieldPath: status.podIP
              volumeMounts:
                - mountPath: /calico-secrets
                  name: etcd-certs
              securityContext:
                privileged: true
          nodeSelector:
            com.docker.ucp.kubernetes.calico/route-reflector: "true"
          volumes:
          # Mount in the etcd TLS secrets.
            - name: etcd-certs
              secret:
                secretName: calico-etcd-secrets
    
  2. Deploy the DaemonSet:

    kubectl create -f calico-rr.yaml
    

Configure calicoctl

To reconfigure Calico to use Route Reflectors rather than a node-to-node mesh, you must direct calicoctl to the etcd key-value store that is managed by MKE. Use the CLI with an MKE client bundle to create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

UCP_VERSION=$(docker version --format '{{index (split .Server.Version "/") 1}}') alias calicoctl="\
docker run -i --rm \
  --pid host \
  --net host \
  -e constraint:ostype==linux \
  -e ETCD_ENDPOINTS=127.0.0.1:12378 \
  -e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
  -e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
  -e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
  -v /var/run/calico:/var/run/calico \
  -v ucp-node-certs:/ucp-node-certs:ro \
  mirantis/ucp-dsinfo:${UCP_VERSION} \
  calicoctl \
"

Disable node-to-node BGP mesh

Once you have configured calicoctl, verify the current Calico BGP configuration to disable node-to-node BGP mesh:

calicoctl get bgpconfig
  • If no configuration presents, create a new one with node-to-node mesh BGP disabled:

    calicoctl create -f - <<EOF
    apiVersion: projectcalico.org/v3
    kind: BGPConfiguration
    metadata:
      name: default
    spec:
      logSeverityScreen: Info
      nodeToNodeMeshEnabled: false
      asNumber: 63400
    EOF
    
  • If a configuration and the meshenabled setting is true:

    1. Update your configuration:

      calicoctl get bgpconfig --output yaml > bgp.yaml
      
    2. Edit the bgp.yaml file, changing the nodeToNodeMeshEnabled setting to false.

    3. Update the Calico configuration:

      calicoctl replace -f - < bgp.yaml
      

Configure Calico to use Route Reflectors

  1. Determine the Autonomous System (AS) number for your network:

    calicoctl get nodes --output=wide
    
  2. Create the Calico configuration by customizing and running the following snippet for each route reflector:

    calicoctl create -f - << EOF
    apiVersion: projectcalico.org/v3
    kind: BGPPeer
    metadata:
      name: bgppeer-global
    spec:
      peerIP: <IP_RR>
      asNumber: <AS_NUMBER>
    EOF
    
    • IP_RR = IP of the node on which the Route Reflector Pod is deployed

    • AS_NUMBER = the AS number for your nodes

Stop calico-node pods

To ensure that there are no instances in which pods and Route Reflectors are running on the same node, use your MKE client bundle to manually delete any calico-node pods that are running on nodes dedicated to Route Reflectors.

  1. Determine the Pod name:

    kubectl -n kube-system \
      get pods --selector k8s-app=calico-node -o wide | \
      grep <node-name>
    
  2. Delete the Pod:

    kubectl -n kube-system delete pod <pod-name>
    

Validate peers

  1. Verify that any calico-node pods that are running on other nodes are peering with the Route Reflector.

  2. From a CLI with an MKE client bundle, use a Swarm affinity filter to run calicoctl node status on any node running calico-node:

    UCP_VERSION=$(docker version --format '{{index (split .Server.Version "/") 1}}')
    docker run -i --rm \
      --pid host \
      --net host \
      -e affinity:container=='k8s_calico-node.*' \
      -e ETCD_ENDPOINTS=127.0.0.1:12378 \
      -e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
      -e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
      -e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
      -v /var/run/calico:/var/run/calico \
      -v ucp-node-certs:/ucp-node-certs:ro \
      mirantis/ucp-dsinfo:${UCP_VERSION} \
      calicoctl node status
    

    The results delivered should resemble the following example output:

    IPv4 BGP status
    +--------------+-----------+-------+----------+-------------+
    | PEER ADDRESS | PEER TYPE | STATE |  SINCE   |    INFO     |
    +--------------+-----------+-------+----------+-------------+
    | 172.31.24.86 | global    | up    | 23:10:04 | Established |
    +--------------+-----------+-------+----------+-------------+
    
    IPv6 BGP status
    No IPv6 peers found.