Virtualization Monitoring Installation Guide#

Overview#

This document provides a step-by-step guide for the installation of k0rdent Virtualization monitoring in the Mirantis k0rdent Enterprise cluster with enabled KOF and k0rdent Virtualization. Please follow all instructions carefully to ensure a successful installation.

Requirements#

Before proceeding with the installation, ensure your system meets the following requirements:

Mirantis k0rdent Enterprise version 1.2.x
k0rdent Enterprise Observability & FinOps (KOF) is enabled in the target child cluster
Grafana is installed in the KOF regional cluster
k0rdent Virtualization 1.17.1-mira or later is installed on the target child cluster
[for airgap only] kubevirt airgap bundle uploaded to the local registry (see kubevirt airgap installation)

Pre-Installation Checklist#

Complete the following steps before starting the installation in case k0rdent Virtualization was installed in the target child cluster before KOF was enabled there (in case KOF was enabled first and only then k0rdent Virtualization was installed, these steps are redundant):

List all ServiceMonitor objects in the kubevirt namespace of the target child cluster:
```
kubectl -n kubevirt get servicemonitor
```
If service-monitor-cdi ServiceMonitor is missing in the list, recreate the cdi-operator pod:
```
kubectl -n kubevirt delete pod -l name=cdi-operator
```
If service-monitor-cluster-network-addons-operator ServiceMonitor is missing in the list, recreate the cluster-network-addons-operator pod:
```
kubectl -n kubevirt delete pod -l name=cluster-network-addons-operator
```
If kubevirt-hyperconverged-operator-metrics ServiceMonitor is missing in the list, recreate the hyperconverged-cluster-operator pod:
```
kubectl -n kubevirt delete pod -l name=hyperconverged-cluster-operator
```
If prometheus-kubevirt-rules ServiceMonitor is missing in the list, recreate the virt-operator pod(s):
```
kubectl -n kubevirt delete pod -l kubevirt.io=virt-operator
```
At the end, all 4 mentioned above ServiceMonitor objects must exist in the cluster.

Installation Steps#

Follow these steps to install the product.

Step 1: Install Grafana dashboards in the KOF regional cluster#

Install the release of the k0rdent-virtualization-monitoring-service-template helm chart from the k0rdent-enterprise-catalog repository to the kcm-system namespace on the mothership cluster:

Note

Please use helm v3.

For public registry:

helm upgrade -i --wait k0rdent-virtualization-monitoring-1-0-0 -n kcm-system oci://registry.mirantis.com/k0rdent-enterprise-catalog/k0rdent-virtualization-monitoring-service-template --version 1.0.0

For airgap:

helm upgrade -i --wait k0rdent-virtualization-monitoring-1-0-0 -n kcm-system oci://${REGISTRY_ADDRESS}/k0rdent-enterprise-catalog/k0rdent-virtualization-monitoring-service-template --version 1.0.0 --set helmRepository.url=oci://${REGISTRY_ADDRESS}/kubevirt

For example:

helm upgrade -i --wait k0rdent-virtualization-monitoring-1-0-0 -n kcm-system oci://172.19.117.80:5000/k0rdent-enterprise-catalog/k0rdent-virtualization-monitoring-service-template --version 1.0.0 --set helmRepository.url=oci://172.19.117.80:5000/kubevirt --plain-http

Verify the k0rdent-virtualization-monitoring-1-0-0 ServiceTemplate object is created and VALID (it can take some time, up to a minute):

kubectl get servicetemplate k0rdent-virtualization-monitoring-1-0-0 -n kcm-system

Expected output:

NAME                                      VALID
k0rdent-virtualization-monitoring-1-0-0   true

Add the k0rdent-virtualization-monitoring Service to the .spec.serviceSpec.services list of the target KOF Regional cluster ClusterDeployment object:

apiVersion: k0rdent.mirantis.com/v1alpha1
kind: ClusterDeployment
metadata:
labels:
  ...
  k0rdent.mirantis.com/kof-cluster-role: regional
name: <KOF regional cluster name>
...
spec:
serviceSpec:
  services:
  ...
  - name: k0rdent-virtualization-monitoring
    namespace: kof
    template: k0rdent-virtualization-monitoring-1-0-0
    values: |
      prometheus:
        alerts: null
  ...

Step 2: Install alerting rules in the mothership cluster#

Install the release of the k0rdent-virtualization-monitoring helm chart from the kubevirt repository to the kof namespace on the mothership cluster:

Note

Please use helm v3.

For public registry:

helm upgrade -i --wait k0rdent-virtualization-monitoring -n kof oci://registry.mirantis.com/kubevirt/k0rdent-virtualization-monitoring --version 0.1.0 --set grafana.dashboards=null

For airgap:

helm upgrade -i --wait k0rdent-virtualization-monitoring -n kof oci://${REGISTRY_ADDRESS}/kubevirt/k0rdent-virtualization-monitoring --version 0.1.0 --set grafana.dashboards=null

For example:

helm upgrade -i --wait k0rdent-virtualization-monitoring -n kof oci://172.19.117.80:5000/kubevirt/k0rdent-virtualization-monitoring --version 0.1.0 --set grafana.dashboards=null --plain-http

Make sure the STATUS is deployed in the helm upgrade command output.

Post-Installation#

Verification#

After installation, verify the monitoring is working correctly by following these steps:

Navigate to the k0rdent Virtualization folder in KOF regional cluster Grafana.
Verify that all 5 dashboards are presented in the folder and all panels on these dashboards contain data:
- KubeVirt Control Plane Latency & Errors
  Workqueue row panels could be empty for some Workqueues and Pods (but not for all).
- KubeVirt Control Plane Stability
  It is expected that all components are healthy, ready and up. No out-of-band modifications.
- KubeVirt Host & Capacity
  To get the CPU Requests panel data, at least one VM must be created in the cluster.
- KubeVirt Lifecycle & Instability
  To get the meaningful data on the dashboard, at least one VM must be created in the cluster.
- KubeVirt VMI Performance, Storage & Network Contention
  To get the meaningful data on the dashboard, at least one VM must be created in the cluster.
Navigate to the Alerts tab in KOF mothership cluster Promxy.
Verify that all 11 KubeVirt* alerts are presented in the list and not firing (inactive, green).