In correlation with the end of life (EOL) for MKE 3.5.x, maintenance of this documentation set was discontinued as of 2023-NOV-22. Click here for the latest MKE 3.x version documentation.

Deploy Prometheus on worker nodes¶

MKE deploys Prometheus by default on the manager nodes to provide a built-in metrics back end. For cluster sizes over 100 nodes, or if you need to scrape metrics from Prometheus instances, Mirantis recommends that you deploy Prometheus on dedicated worker nodes in the cluster.

To deploy Prometheus on worker nodes:

Source an admin bundle.

Verify that ucp-metrics pods are running on all managers:

$ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide

NAME               READY  STATUS   RESTARTS  AGE  IP            NODE
ucp-metrics-hvkr7  3/3    Running  0         4h   192.168.80.66 3a724a-0

Add a Kubernetes node label to one or more workers. For example, a label with key ucp-metrics and value "" to a node with name 3a724a-1.
```
$ kubectl label node 3a724a-1 ucp-metrics=

node "test-3a724a-1" labeled
```
SELinux Prometheus Deployment

If you use SELinux, label your ucp-node-certs directories properly on the worker nodes before you move the ucp-metrics workload to them. To run ucp-metrics on a worker node, update the ucp-node-certs label by running:

sudo chcon -R system_u:object_r:container_file_t:s0 /var/lib/docker/volumes/ucp-node-certs/_data.

Patch the ucp-metrics DaemonSet’s nodeSelector with the same key and value in use for the node label. This example shows the key ucp-metrics and the value "".

$ kubectl -n kube-system patch daemonset ucp-metrics --type json -p
'[{"op": "replace", "path": "/spec/template/spec/nodeSelector", "value":
{"ucp-metrics": ""}}]' daemonset "ucp-metrics" patched

Confirm that ucp-metrics pods are running only on the labeled workers.

$ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide

NAME               READY  STATUS       RESTARTS  AGE IP           NODE
ucp-metrics-88lzx  3/3    Running      0         12s 192.168.83.1 3a724a-1
ucp-metrics-hvkr7  3/3    Terminating  0         4h 192.168.80.66 3a724a-0