Upgrade Monitoring CRDs

Before you upgrade to MKE 4.2.0, Mirantis suggests that you manually upgrade any CRDs that were made available following the introduction of prometheus-operator v0.86.2. This is in line with the MKE 4.2.0 upgrade of the kube-prometheus-stack from chart 79.5.0 (prometheus-operator v0.86.2) to 86.2.3 (v0.91.0).

Source-version coverage

kube-prometheus-stack 79.5.0 is the chart that is included in every MKE 4.1.x release. As the upgrade to 86.2.3 is first present in MKE 4.2.0, a single source version, such as version 79.5.0, works for every 4.1.x cluster subject to upgrade.

Specific to air gap deployments

If you are running an air-gapped MKE 4 cluster, upload the offline bundle for 4.2.0 first, and use the registry you specified in .spec.registries.imageRegistry.url for the REPO value, as illustrated below:

CHART_VERSION=86.2.3
REPO=oci://registry.mirantis.com/mke
WORK=$(mktemp -d)

Refer to Offline installation for information on air gap installations and how to specify registries in your MKE 4 configuration.

The CRDs live in the crds subchart at charts/crds/crds/. This move is additive -- same names, same v1/v1alpha1 storage version -- so a manual server-side apply is safe.

1. Ensure that your source cluster is backed up.

2. Fetch the target chart and extract the nested CRD to charts/crds/crds/.

   helm pull kube-prometheus-stack --repo "$REPO" --version "$CHART_VERSION" \
     --untar --untardir "$WORK"
   
   CRD_DIR="$WORK/kube-prometheus-stack/charts/crds/crds"
   ls -1 "$CRD_DIR"
   # crd-alertmanagerconfigs.yaml  crd-alertmanagers.yaml  crd-podmonitors.yaml
   # crd-probes.yaml  crd-prometheusagents.yaml  crd-prometheuses.yaml
   # crd-prometheusrules.yaml  crd-scrapeconfigs.yaml  crd-servicemonitors.yaml
   # crd-thanosrulers.yaml
   

3. Optional. Pre-check/backup:

   # Record current stored schema + which field manager owns them today
   for c in alertmanagerconfigs alertmanagers podmonitors probes prometheusagents \
           prometheuses prometheusrules scrapeconfigs servicemonitors thanosrulers; do
     kubectl get crd ${c}.monitoring.coreos.com -o yaml > "$WORK/backup-${c}.yaml" 2>/dev/null
   done
   

4. Apply the CRDs, using a whole-directory apply approach, wherein the kubectl command applies each file:

   kubectl apply --server-side --force-conflicts -f "$CRD_DIR"
   

   Essential flags

   It is critical that you retain the --server-side flag, as six of the files are 600–830 KB in size, and plain (client-side) apply fails with the metadata.annotations: Too long error due to the 262144-byte annotation cap. The --force-conflicts is required because the CRDs are currently field-owned by the existing chart/operator manager.

5. Verify the application of the CRDs.

   1. Confirm that all of the CRDs are established:

      kubectl get crd -l app.kubernetes.io/part-of=kube-prometheus-stack 2>/dev/null \
      || kubectl get crd | grep monitoring.coreos.com
      

   2. Confirm that the served/storage version is intact as v1 / v1alpha1 (v1alpha1 for alertmanagerconfigs, scrapeconfigs, and prometheusagents, and v1 for all others):

      for c in prometheuses alertmanagers thanosrulers prometheusagents \
              alertmanagerconfigs scrapeconfigs servicemonitors podmonitors \
              probes prometheusrules; do
          printf '%-20s stored=' "$c"
          kubectl get crd ${c}.monitoring.coreos.com \
          -o jsonpath='{range .spec.versions[?(@.storage==true)]}{.name}{end}{"\n"}'
      done
      

   3. Check that a new field landed, for example, a v0.91 addition on Prometheus:

   kubectl get crd prometheuses.monitoring.coreos.com -o yaml | grep -c 'description'
   

6. Clean up the operation.

   rm -rf "$WORK"