Set up Grafana with MKE Prometheus¶

Important

The information offered herein on how to set up a Grafana instance connected to MKE Prometheus is derived from the official Deploy Grafana on Kubernetes documentation and modified to work with MKE. As it deploys Grafana with default credentials, Mirantis strongly recommends that you adjust the configuration detail to meet your specific needs prior to deploying Grafana with MKE in a production environment.

Source an MKE admin bundle.
Create the monitoring namespace on which you will deploy Grafana:
```
kubectl create namespace monitoring
```

Obtain the UCP cluster ID:

CLUSTER_ID=$(docker info --format '{{json .Swarm.Cluster.ID}}')

Apply the following YAML file to deploy Grafana in the monitoring namespace and to automatically configure MKE Prometheus as a data source:

kubectl apply -f - <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        runAsUser: 0
      containers:
        - name: grafana
          image: grafana/grafana:9.1.0-ubuntu
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /etc/grafana/
              name: grafana-config-volume
            - mountPath: /etc/ssl
              name: ucp-node-certs
      volumes:
        - name: grafana-config-volume
          configMap:
            name: grafana-config
            items:
              - key: grafana.ini
                path: grafana.ini
              - key: dashboard.json
                path: dashboard.json
              - key: datasource.yml
                path: provisioning/datasources/datasource.yml
        - name: ucp-node-certs
          hostPath:
            path: /var/lib/docker/volumes/ucp-node-certs/_data
      nodeSelector:
        node-role.kubernetes.io/master: ""
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-config
  namespace: monitoring
  labels:
    grafana_datasource: '1'
data:
  grafana.ini: |
  dashboard.json: |
  datasource.yml: |-
    apiVersion: 1
    datasources:
    - name: mke-prometheus
      type: prometheus
      access: proxy
      orgId: 1
      url: https://ucp-metrics.kube-system.svc.cluster.local:443
      jsonData:
        tlsAuth: true
        tlsAuthWithCACert: false
        serverName: $CLUSTER_ID
      secureJsonData:
        tlsClientCert: "\$__file{/etc/ssl/cert.pem}"
        tlsClientKey: "\$__file{/etc/ssl/key.pem}"
---
EOF

Use port forwarding to access the Grafana UI. Be aware that this may require that you install socat on your manager nodes.
```
kubectl port-forward service/grafana 3000:3000 -n monitoring
```

You can now navigate to the Grafana UI, which has the MKE Prometheus data source installed at http://localhost:3000/. Log in initially using admin for both the user name and password, taking care to change your credentials after successful log in.