Warm up the MOSK management cache

TechPreview

This section describes how to speed up deployment and update process of MOSK clusters, which usually do not have access to the internet and consume artifacts from a management cluster using the mcc-cache service.

By default, after auto-update of a management cluster, before each MOSK cluster deployment or update, mcc-cache downloads the required list of images, thus slowing down the process.

Using the CacheWarmupRequest resource, you can predownload (warm up) a list of images included in a given set of Cluster releases into the mcc-cache service only once per release for further usage on all MOSK clusters.

The CacheWarmupRequest object is automatically deleted once the cache warm-up completes, whether it succeeds or fails. The warmed-up cache remains for deployment or update of MOSK clusters for a limited period. For details, see Validity of the warmed-up cache.

Caution

If the disk space for cache runs out, or the size of cached artifacts exceeds the configured limit, the cache for the oldest object is evicted. To avoid running out of space in the cache, verify and adjust its size before each cache warm-up.

Requirements

Cache warm-up requires a lot of disk storage, depending on the number of artifacts to be pre-cached for Cluster and OpenStack releases. Therefore, make sure to have enough space in the dedicated LocalVolumeProvisioner (lvp) partition for storing cached objects on each node of the management cluster before creating the CacheWarmupRequest resource. By default, 20 GiB of cache size is configured, but Mirantis highly recommends increasing it to the following values:

Recommended cache size

Cluster release

Recommended value in GiB

MOSK Cluster release with one OpenStack version

50

MOSK Cluster release with an OpenStack version upgrade from Epoxy to Gazpacho

120

Note

If openstackReleases is not set in the CacheWarmupRequest object:

  • Since MOSK management 2.32.1, all OpenStack releases supported by the specified Cluster releases are warmed up.

    Therefore, plan the cache size as for a Cluster release with an OpenStack version upgrade, including the intermediate versions, if any, or define openstackReleases explicitly. For details, see CacheWarmupRequest resource.

  • Before MOSK management 2.32.1, no OpenStack images are warmed up.

Caution

Since MOSK management 2.32.1, the registry of the mcc-cache service keeps the downloaded images in the same lvp volume as the NGINX cache. The registry storage is not limited by the cacheSize parameter and takes about the same space as the warmed-up images. Therefore, the lvp partition must fit both the configured cacheSize and the registry storage. For example, for a MOSK Cluster release with one OpenStack version, plan at least 50 GiB of cacheSize and at least 100 GiB for the lvp partition.

For details about the lvp partition requirements, see Default configuration of the host system storage.

Increase cache size for mcc-cache

After you calculate the disk size for warming up cache depending on your cluster settings and minimal cache warm-up requirements, configure the size of cache in the Cluster object of your cluster.

In the spec:providerSpec:value:kaas:regionalHelmReleases: section of the management Cluster object, add the following snippet to the mcc-cache entry with the required cacheSize value in GiB:

nginx:
  cacheSize: 100
kubectl --kubeconfig <pathToManagementClusterKubeconfig> edit cluster <clusterName>

Configuration example:

spec:
  providerSpec:
    value:
      kaas:
        regionalHelmReleases:
        - name: mcc-cache
          values:
            nginx:
              cacheSize: 100

Note

The cacheSize parameter only increases the NGINX cache size and does not change the size of the related Persistent Volume for the mcc-cache service. It also does not limit the registry storage that resides on the same volume. Make sure to have enough capacity in the related lvp partition on the management cluster nodes. For disk space requirements, see MOSK cluster hardware requirements.

Warm up cache using CLI

After you increase the size of cache on the cluster as described in Increase cache size for mcc-cache, create the CacheWarmupRequest object in the Kubernetes API.

Caution

Create CacheWarmupRequest objects only on the management cluster.

To warm up cache using CLI:

  1. Identify the latest available Cluster releases to use for deployment of new clusters and update of existing clusters:

    kubectl --kubeconfig <pathToManagementClusterKubeconfig> get kaasreleases -l=kaas.mirantis.com/active="true" -o=json | jq -r '.items[].spec.supportedClusterReleases[] | select(.availableUpgrades | length == 0) | .name'
    

    Example of system response:

    k0s-22-0-1-1-33-13
    mosk-21-2-1-26-2-1
    
  2. On the management cluster, create a .yaml file for the CacheWarmupRequest object using the following example:

    apiVersion: kaas.mirantis.com/v1alpha1
    kind: CacheWarmupRequest
    metadata:
      name: example-cluster-name
      namespace: default
    spec:
      clusterReleases:
      - k0s-22-0-1-1-33-13
      - mosk-21-2-1-26-2-1
      openstackReleases:
      - gazpacho
      fetchRequestTimeout: 30m
      clientsPerEndpoint: 2
      openstackOnly: false
    

    In this example:

    • The CacheWarmupRequest object is created for a management cluster named example-cluster-name.

    • The CacheWarmupRequest object is created in the only allowed default MOSK management project.

    • Two Cluster releases mosk-21-2-1-26-2-1 and k0s-22-0-1-1-33-13 will be predownloaded.

    • For mosk-21-2-1-26-2-1, only images related to the OpenStack version Gazpacho will be predownloaded.

    • Maximum time-out for a single request to download a single artifact is 30 minutes.

    • Two parallel workers will fetch artifacts per each mcc-cache service endpoint.

    • All artifacts will be fetched, not only those related to OpenStack.

    For details about the CacheWarmupRequest object, see CacheWarmupRequest resource.

  3. Apply the object to the cluster:

    kubectl --kubeconfig <pathToManagementKubeconfig> apply -f <pathToFile>
    

    Once done, during deployment and update of MOSK clusters, MOSK management uses cached artifacts from the mcc-cache service to facilitate and speed up the procedure.

When a new MOSK release becomes available and the management cluster auto-updates to a new release, repeat the above steps to predownload a new set of artifacts for MOSK clusters.

Validity of the warmed-up cache

The warmed-up artifacts remain available through the mcc-cache service for about one month after the cache warm-up, provided that the cache size is sufficient. After that, the cached artifacts expire, and the first request for each of them downloads it from the Mirantis repositories again. Therefore, if the deployment or update of MOSK clusters continues for longer than one month after the cache warm-up, repeat the cache warm-up before proceeding.

The registry of the mcc-cache service keeps its own copy of the downloaded artifacts for 7 days. This copy is used only if an artifact is missing in the NGINX cache, for example, if the artifact is evicted because of an insufficient cache size. To verify whether the cache was evicted, compare the used cache size on each mcc-cache pod with the configured cacheSize:

kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec <mccCachePodName> -c nginx -- du -sh /srv/nginx_cache

If the used cache size reaches the configured cacheSize value, increase it as described in Increase cache size for mcc-cache and repeat the cache warm-up.

Verify the cache warm-up result

The CacheWarmupRequest object is automatically deleted once the cache warm-up completes, whether it succeeds or fails. The warm-up summary is stored in the status:providerStatus:regionalStatus:cacheWarmup section of the management Cluster object and in the CacheWarmup condition of the cluster.

To verify the cache warm-up result:

kubectl --kubeconfig <pathToManagementClusterKubeconfig> get cluster <clusterName> \
-o jsonpath='{.status.providerStatus.regionalStatus.cacheWarmup}' | yq -P

Example of system response:

startedAt: "2026-10-06T19:15:27Z"
finishedAt: "2026-10-06T19:57:00Z"
success: false
errors: '["failed to load images list https://binary.mirantis.com/tungsten/bin/artifacts/tungstenfabric-operator-0.20.5.yaml: not found"]'
details:
  core/lcm: ok
  ceph: ok
  openstack: ok
  stacklight: ok
  tungsten: 'failed: failed to load images list https://binary.mirantis.com/tungsten/bin/artifacts/tungstenfabric-operator-0.20.5.yaml: not found'

The status contains the following fields:

  • startedAt and finishedAt

    Start and finish time of the cache warm-up.

  • success

    Result of the cache warm-up: true if all image lists are loaded and all images are warmed up, false otherwise. For the failure reasons, see errors and details.

  • errors

    Errors of the cache warm-up, if any: the image lists that failed to load and the images that failed to warm up. The list is truncated to the first 50 errors, each up to 200 characters long. For the full list of errors, see the logs of the mcc-cache-warmup-job-* job pod in the kaas namespace.

  • details

    Available since MOSK management 2.32.1. Result of loading the image lists by group: core/lcm for the images of KaaSRelease, and openstack, ceph, stacklight, tungsten, and so on for the images of the corresponding components of the Cluster releases. Each group has one of the following results:

    • ok - the image lists of the group are loaded

    • failed: <errors> - at least one image list of the group failed to load

Since MOSK management 2.32.1, a failed image list does not prevent warming up the images of other lists, but the cache warm-up fails in this case. The images of the failed list are not warmed up.

To repeat the cache warm-up, for example, after fixing the access to the artifacts, create a new CacheWarmupRequest object as described above.

If image pulls through mcc-cache fail with the digest mismatch error, see Image pull failure due to a corrupted cache.