Deploy route reflectors for improved network performance¶

Before you begin¶

For production-grade systems, you should deploy at least two Route Reflectors, each running on a dedicated node. These nodes should not be running any other workloads.

If Route Reflectors are running on a same node as other workloads, swarm ingress and NodePorts might not work in these workloads.

Choose dedicated nodes¶

1. Taint the nodes to ensure that they are unable to run other workloads.

2. For each dedicated node, run:

   kubectl taint node <node-name> \
   com.docker.ucp.kubernetes.calico/route-reflector=true:NoSchedule
   

3. Add labels to those nodes:

   kubectl label nodes <node-name> \
   com.docker.ucp.kubernetes.calico/route-reflector=true
   

Deploy the Route Reflectors¶

1. Create a calico-rr.yaml file with the following content:

   kind: DaemonSet
   apiVersion: extensions/v1beta1
   metadata:
     name: calico-rr
     namespace: kube-system
     labels:
       app: calico-rr
   spec:
     updateStrategy:
       type: RollingUpdate
     selector:
       matchLabels:
         k8s-app: calico-rr
     template:
       metadata:
         labels:
           k8s-app: calico-rr
         annotations:
           scheduler.alpha.kubernetes.io/critical-pod: ''
       spec:
         tolerations:
           - key: com.docker.ucp.kubernetes.calico/route-reflector
             value: "true"
             effect: NoSchedule
         hostNetwork: true
         containers:
           - name: calico-rr
             image: calico/routereflector:v0.6.1
             env:
               - name: ETCD_ENDPOINTS
                 valueFrom:
                   configMapKeyRef:
                     name: calico-config
                     key: etcd_endpoints
               - name: ETCD_CA_CERT_FILE
                 valueFrom:
                   configMapKeyRef:
                     name: calico-config
                     key: etcd_ca
               # Location of the client key for etcd.
               - name: ETCD_KEY_FILE
                 valueFrom:
                   configMapKeyRef:
                     name: calico-config
                     key: etcd_key # Location of the client certificate for etcd.
               - name: ETCD_CERT_FILE
                 valueFrom:
                   configMapKeyRef:
                     name: calico-config
                     key: etcd_cert
               - name: IP
                 valueFrom:
                   fieldRef:
                     fieldPath: status.podIP
             volumeMounts:
               - mountPath: /calico-secrets
                 name: etcd-certs
             securityContext:
               privileged: true
         nodeSelector:
           com.docker.ucp.kubernetes.calico/route-reflector: "true"
         volumes:
         # Mount in the etcd TLS secrets.
           - name: etcd-certs
             secret:
               secretName: calico-etcd-secrets
   

2. Deploy the DaemonSet using:

   kubectl create -f calico-rr.yaml
   

Configure calicoctl¶

To reconfigure Calico to use Route Reflectors instead of a node-to-node mesh, you’ll need to tell calicoctl where to find the etcd key-value store managed by MKE. From a CLI with a MKE client bundle, create a shell alias to start calicoctl using the docker/ucp-dsinfo image:

UCP_VERSION=$(docker version --format {% raw %}'{{index (split .Server.Version "/") 1}}'{% endraw %})
alias calicoctl="\
docker run -i --rm \
  --pid host \
  --net host \
  -e constraint:ostype==linux \
  -e ETCD_ENDPOINTS=127.0.0.1:12378 \
  -e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
  -e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
  -e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
  -v /var/run/calico:/var/run/calico \
  -v ucp-node-certs:/ucp-node-certs:ro \
  docker/ucp-dsinfo:${UCP_VERSION} \
  calicoctl \
"

Disable node-to-node BGP mesh¶

After configuring calicoctl, check the current Calico BGP configuration:

calicoctl get bgpconfig

If you don’t see any configuration listed, create one:

calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 63400
EOF

This action creates a new configuration with node-to-node mesh BGP disabled.

If you have a configuration, and meshenabled is set to true:

1. Update your configuration:

   calicoctl get bgpconfig --output yaml > bgp.yaml
   

2. Edit the bgp.yaml file, updating nodeToNodeMeshEnabled to false.

3. Update the Calico configuration:

   calicoctl replace -f - < bgp.yaml
   

Configure Calico to use Route Reflectors¶

To configure Calico to use the Route Reflectors you need to know the AS number for your network first. For that, run:

calicoctl get nodes --output=wide

Using the AS number, create the Calico configuration by customizing and running the following snippet for each route reflector:

calicoctl create -f - << EOF
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global
spec:
  peerIP: <IP_RR>
  asNumber: <AS_NUMBER>
EOF

Where:

• IP_RR is the IP of the node where the Route Reflector pod is deployed.
• AS_NUMBER is the same AS number for your nodes.

Stop calico-node pods¶

1. Manually delete any calico-mode pods that are running on nodes dedicated to the running of route reflectors, as this will ensure that there are no instances in whic pods and route reflectors are running on the same node.

2. Using your MKE client bundle:

   # Find the Pod name
   kubectl -n kube-system \
     get pods --selector k8s-app=calico-node -o wide | \
     grep <node-name>
   
   # Delete the Pod
   kubectl -n kube-system delete pod <pod-name>
   

Validate peers¶

1. Verify that calico-node pods running on other nodes are peering with the Route Reflector.

2. From a CLI with a MKE client bundle, use a Swarm affinity filter to run calicoctl node status on any node running calico-node:

   UCP_VERSION=$(docker version --format {% raw %}'{{index (split .Server.Version "/") 1}}'{% endraw %})
   docker run -i --rm \
     --pid host \
     --net host \
     -e affinity:container=='k8s_calico-node.*' \
     -e ETCD_ENDPOINTS=127.0.0.1:12378 \
     -e ETCD_KEY_FILE=/ucp-node-certs/key.pem \
     -e ETCD_CA_CERT_FILE=/ucp-node-certs/ca.pem \
     -e ETCD_CERT_FILE=/ucp-node-certs/cert.pem \
     -v /var/run/calico:/var/run/calico \
     -v ucp-node-certs:/ucp-node-certs:ro \
     docker/ucp-dsinfo:${UCP_VERSION} \
     calicoctl node status
   

The delivered results should resemble the following sample output:

IPv4 BGP status
+--------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE |  SINCE   |    INFO     |
+--------------+-----------+-------+----------+-------------+
| 172.31.24.86 | global    | up    | 23:10:04 | Established |
+--------------+-----------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.