Deploy Prometheus on worker nodes

MKE deploys Prometheus by default on the manager nodes to provide a built-in metrics backend. For cluster sizes over 100 nodes, or if you need to scrape metrics from Prometheus instances, Mirantis recommends that you deploy Prometheus on dedicated worker nodes in the cluster.

To deploy Prometheus on worker nodes:

1. Source an admin bundle.

2. Verify that ucp-metrics pods are running on all managers:

   $ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide
   
   NAME               READY  STATUS   RESTARTS  AGE  IP            NODE
   ucp-metrics-hvkr7  3/3    Running  0         4h   192.168.80.66 3a724a-0
   

3. Add a Kubernetes node label to one or more workers. For example, a label with key ucp-metrics and value "" to a node with name 3a724a-1.

   $ kubectl label node 3a724a-1 ucp-metrics=
   
   node "test-3a724a-1" labeled
   

   SELinux Prometheus Deployment

   If you use SELinux, label your ucp-node-certs directories properly on the worker nodes before you move the ucp-metrics workload to them. To run ucp-metrics on a worker node, update the ucp-node-certs label by running:

   sudo chcon -R system_u:object_r:container_file_t:s0 /var/lib/docker/volumes/ucp-node-certs/_data.

4. Patch the ucp-metrics DaemonSet’s nodeSelector with the same key and value in use for the node label. This example shows the key ucp-metrics and the value "".

   $ kubectl -n kube-system patch daemonset ucp-metrics --type json -p
   '[{"op": "replace", "path": "/spec/template/spec/nodeSelector", "value":
   {"ucp-metrics": ""}}]' daemonset "ucp-metrics" patched
   

5. Confirm that ucp-metrics pods are running only on the labeled workers.

   $ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide
   
   NAME               READY  STATUS       RESTARTS  AGE IP           NODE
   ucp-metrics-88lzx  3/3    Running      0         12s 192.168.83.1 3a724a-1
   ucp-metrics-hvkr7  3/3    Terminating  0         4h 192.168.80.66 3a724a-0
   

See also

Kubernetes