Improve network performance with Route Reflectors¶

MKE uses Calico as the default Kubernetes networking solution, configured to create a BGP mesh between all nodes in the cluster.

Note

MKE deployments running on Microsoft Azure use Azure SDN for multi-host networking, rather than Calico.

Networking performance decreases as you add more nodes to the cluster. If your cluster has more than 100 nodes, you should reconfigure Calico to use Route Reflectors rather than a node-to-node mesh.

Select dedicated nodes¶

For production-grade systems, deploy at least two Route Reflectors, with each running on a dedicated node that is not running any other workloads.

If Route Reflectors are running on a same node as other workloads, swarm ingress and NodePorts may not work in these workloads. As such, Mirantis suggests that you taint the nodes to ensure that they do not run other workloads.

For each dedicated node, run:

kubectl taint node <node-name> \
com.docker.ucp.kubernetes.calico/route-reflector=true:NoSchedule

Add labels to the newly tainted nodes:

kubectl label nodes <node-name> \
com.docker.ucp.kubernetes.calico/route-reflector=true

Deploy the Route Reflectors¶

Create a calico-rr.yaml file with the following content:

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-rr
  namespace: kube-system
  labels:
    app: calico-rr
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      k8s-app: calico-rr
  template:
    metadata:
      labels:
        k8s-app: calico-rr
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
        - key: com.docker.ucp.kubernetes.calico/route-reflector
          value: "true"
          effect: NoSchedule
      hostNetwork: true
      containers:
        - name: calico-rr
          image: calico/routereflector:v0.6.1
          env:
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            - name: IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /calico-secrets
              name: etcd-certs
          securityContext:
            privileged: true
      nodeSelector:
        com.docker.ucp.kubernetes.calico/route-reflector: "true"
      volumes:
      # Mount in the etcd TLS secrets.
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets

Deploy the DaemonSet:
```
kubectl create -f calico-rr.yaml
```

Configure calicoctl¶

To reconfigure Calico to use Route Reflectors rather than a node-to-node mesh, you must direct calicoctl to the etcd key-value store that is managed by MKE. Use the CLI with an MKE client bundle to create a shell alias to start calicoctl using the mirantis/ucp-dsinfo image:

UCP_VERSION=$(docker version --format '{{index (split .Server.Version "/")}}')

alias calicoctl="\
docker run -i --rm \
--pid host \
--net host \
-e constraint:ostype==linux \
-e ETCD_ENDPOINTS=127.0.0.1:12378 \
-e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
-e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
-e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
-v /var/run/calico:/var/run/calico \
-v ucp-node-certs:/ucp-node-certs:ro \
mirantis/ucp-dsinfo:3.5.6 \
calicoctl --allow-version-mismatch \
"

Disable node-to-node BGP mesh¶

Once you have configured calicoctl, verify the current Calico BGP configuration to disable node-to-node BGP mesh:

calicoctl get bgpconfig

If no configuration presents, create a new one with node-to-node mesh BGP disabled:

calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 63400
EOF

If a configuration and the meshenabled setting is true:
1. Update your configuration:
```
calicoctl get bgpconfig --output yaml > bgp.yaml
```
2. Edit the bgp.yaml file, changing the nodeToNodeMeshEnabled setting to false.
3. Update the Calico configuration:
```
calicoctl replace -f - < bgp.yaml
```

Configure Calico to use Route Reflectors¶

Determine the Autonomous System (AS) number for your network:
```
calicoctl get nodes --output=wide
```
Create the Calico configuration by customizing and running the following snippet for each route reflector:
```
calicoctl create -f - << EOF
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global
spec:
  peerIP: <IP_RR>
  asNumber: <AS_NUMBER>
EOF
```
- IP_RR = IP of the node on which the Route Reflector Pod is deployed
- AS_NUMBER = the AS number for your nodes

Stop calico-node pods¶

To ensure that there are no instances in which pods and Route Reflectors are running on the same node, use your MKE client bundle to manually delete any calico-node pods that are running on nodes dedicated to Route Reflectors.

Determine the Pod name:

kubectl -n kube-system \
  get pods --selector k8s-app=calico-node -o wide | \
  grep <node-name>

Delete the Pod:

kubectl -n kube-system delete pod <pod-name>

Validate peers¶

Verify that any calico-node pods that are running on other nodes are peering with the Route Reflector.

From a CLI with an MKE client bundle, use a Swarm affinity filter to run calicoctl node status on any node running calico-node:

UCP_VERSION=$(docker version --format '{{index (split .Server.Version "/") 1}}')
docker run -i --rm \
  --pid host \
  --net host \
  -e affinity:container=='k8s_calico-node.*' \
  -e ETCD_ENDPOINTS=127.0.0.1:12378 \
  -e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
  -e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
  -e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
  -v /var/run/calico:/var/run/calico \
  -v ucp-node-certs:/ucp-node-certs:ro \
  mirantis/ucp-dsinfo:${UCP_VERSION} \
  calicoctl node status

The results delivered should resemble the following example output:

IPv4 BGP status
+--------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE |  SINCE   |    INFO     |
+--------------+-----------+-------+----------+-------------+
| 172.31.24.86 | global    | up    | 23:10:04 | Established |
+--------------+-----------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.