Configure external Prometheus to scrape metrics from MKE¶

To configure your external Prometheus server to scrape metrics from Prometheus in MKE:

Source an admin bundle.

Create a Kubernetes secret that contains your bundle TLS material.

(cd $DOCKER_CERT_PATH && kubectl create secret generic prometheus --from-file=ca.pem --from-file=cert.pem --from-file=key.pem)

Create a Prometheus deployment and ClusterIP service using YAML.

Note

On bare metal clusters, enable MetalLB so that you can create a service of the load balancer type, and then perform the following steps:

Replace ClusterIP with LoadBalancer in the service YAML.
Access the service through the load balancer.
If you run Prometheus external to MKE, change the domain for the inventory container in the Prometheus deployment from ucp-controller.kube-system.svc.cluster.local to an external domain, to access MKE from the Prometheus node.

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
data:
  prometheus.yaml: |
    global:
      scrape_interval: 10s
    scrape_configs:
    - job_name: 'ucp'
      tls_config:
        ca_file: /bundle/ca.pem
        cert_file: /bundle/cert.pem
        key_file: /bundle/key.pem
        server_name: proxy.local
      scheme: https
      file_sd_configs:
      - files:
        - /inventory/inventory.json
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 2
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - name: inventory
        image: alpine
        command: ["sh", "-c"]
        args:
        - apk add --no-cache curl &&
          while :; do
            curl -Ss --cacert /bundle/ca.pem --cert /bundle/cert.pem --key /bundle/key.pem --output /inventory/inventory.json https://ucp-controller.kube-system.svc.cluster.local/metricsdiscovery;
            sleep 15;
          done
        volumeMounts:
        - name: bundle
          mountPath: /bundle
        - name: inventory
          mountPath: /inventory
      - name: prometheus
        image: prom/prometheus
        command: ["/bin/prometheus"]
        args:
        - --config.file=/config/prometheus.yaml
        - --storage.tsdb.path=/prometheus
        - --web.console.libraries=/etc/prometheus/console_libraries
        - --web.console.templates=/etc/prometheus/consoles
        volumeMounts:
        - name: bundle
          mountPath: /bundle
        - name: config
          mountPath: /config
        - name: inventory
          mountPath: /inventory
      volumes:
      - name: bundle
        secret:
          secretName: prometheus
      - name: config
        configMap:
          name: prometheus
      - name: inventory
        emptyDir:
          medium: Memory
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  ports:
  - port: 9090
    targetPort: 9090
  selector:
    app: prometheus
  sessionAffinity: ClientIP
EOF

Determine the service ClusterIP:

$ kubectl get service prometheus

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
prometheus   ClusterIP   10.96.254.107   <none>        9090/TCP   1h

Forward port 9090 on the local host to the ClusterIP. The tunnel you create does not need to be kept alive as its only purpose is to expose the Prometheus UI.
```
ssh -L 9090:10.96.254.107:9090 ANY_NODE
```
Visit http://127.0.0.1:9090 to explore the MKE metrics that Prometheus is collecting.