Warm up the MOSK management cache
TechPreview
This section describes how to speed up deployment and update process of
MOSK clusters, which usually do not have access to the
internet and consume artifacts from a management cluster using the
mcc-cache service.
By default, after auto-update of a management cluster,
before each MOSK cluster deployment or update, mcc-cache
downloads the required list of images, thus slowing down the process.
Using the CacheWarmupRequest resource, you can predownload (warm up)
a list of images included in a given set of Cluster releases into the
mcc-cache service only once per release for further usage on all
MOSK clusters.
The CacheWarmupRequest object is automatically deleted once the cache
warm-up completes, whether it succeeds or fails. The warmed-up cache remains
for deployment or update of MOSK clusters for a limited
period. For details, see Validity of the warmed-up cache.
Caution
If the disk space for cache runs out, or the size of cached artifacts exceeds the configured limit, the cache for the oldest object is evicted. To avoid running out of space in the cache, verify and adjust its size before each cache warm-up.
Requirements
Cache warm-up requires a lot of disk storage, depending on the number of
artifacts to be pre-cached for Cluster and OpenStack releases.
Therefore, make sure to have enough space in the dedicated
LocalVolumeProvisioner (lvp) partition for storing cached objects on
each node of the management cluster before creating the CacheWarmupRequest
resource. By default, 20 GiB of cache size is configured, but Mirantis highly
recommends increasing it to the following values:
Cluster release |
Recommended value in GiB |
|---|---|
MOSK Cluster release with one OpenStack version |
50 |
MOSK Cluster release with an OpenStack version upgrade from Epoxy to Gazpacho |
120 |
Note
If openstackReleases is not set in the CacheWarmupRequest
object:
Since MOSK management 2.32.1, all OpenStack releases supported by the specified Cluster releases are warmed up.
Therefore, plan the cache size as for a Cluster release with an OpenStack version upgrade, including the intermediate versions, if any, or define
openstackReleasesexplicitly. For details, see CacheWarmupRequest resource.Before MOSK management 2.32.1, no OpenStack images are warmed up.
Caution
Since MOSK management 2.32.1, the registry of the mcc-cache
service keeps the downloaded images in the same lvp volume as the NGINX
cache. The registry storage is not limited by the cacheSize parameter
and takes about the same space as the warmed-up images. Therefore, the
lvp partition must fit both the configured cacheSize and the
registry storage. For example, for a MOSK Cluster release with one OpenStack
version, plan at least 50 GiB of cacheSize and at least 100 GiB for the
lvp partition.
For details about the lvp partition requirements, see
Default configuration of the host system storage.
Increase cache size for mcc-cache
After you calculate the disk size for warming up cache depending on your
cluster settings and minimal cache warm-up requirements, configure the size of
cache in the Cluster object of your cluster.
In the spec:providerSpec:value:kaas:regionalHelmReleases: section of the
management Cluster object, add the following snippet to the mcc-cache
entry with the required cacheSize value in GiB:
nginx:
cacheSize: 100
kubectl --kubeconfig <pathToManagementClusterKubeconfig> edit cluster <clusterName>
Configuration example:
spec:
providerSpec:
value:
kaas:
regionalHelmReleases:
- name: mcc-cache
values:
nginx:
cacheSize: 100
Note
The cacheSize parameter only increases the NGINX cache size and
does not change the size of the related Persistent Volume for the
mcc-cache service. It also does not limit the registry storage that
resides on the same volume. Make sure to have enough capacity in the related
lvp partition on the management cluster nodes. For disk space
requirements, see MOSK cluster hardware requirements.
Warm up cache using CLI
After you increase the size of cache on the cluster as described in
Increase cache size for mcc-cache, create the CacheWarmupRequest object in the
Kubernetes API.
Caution
Create CacheWarmupRequest objects only on the management
cluster.
To warm up cache using CLI:
Identify the latest available Cluster releases to use for deployment of new clusters and update of existing clusters:
kubectl --kubeconfig <pathToManagementClusterKubeconfig> get kaasreleases -l=kaas.mirantis.com/active="true" -o=json | jq -r '.items[].spec.supportedClusterReleases[] | select(.availableUpgrades | length == 0) | .name'
Example of system response:
k0s-22-0-1-1-33-13 mosk-21-2-1-26-2-1
On the management cluster, create a
.yamlfile for theCacheWarmupRequestobject using the following example:apiVersion: kaas.mirantis.com/v1alpha1 kind: CacheWarmupRequest metadata: name: example-cluster-name namespace: default spec: clusterReleases: - k0s-22-0-1-1-33-13 - mosk-21-2-1-26-2-1 openstackReleases: - gazpacho fetchRequestTimeout: 30m clientsPerEndpoint: 2 openstackOnly: false
In this example:
The
CacheWarmupRequestobject is created for a management cluster namedexample-cluster-name.The
CacheWarmupRequestobject is created in the only alloweddefaultMOSK management project.Two Cluster releases
mosk-21-2-1-26-2-1andk0s-22-0-1-1-33-13will be predownloaded.For
mosk-21-2-1-26-2-1, only images related to the OpenStack versionGazpachowill be predownloaded.Maximum time-out for a single request to download a single artifact is 30 minutes.
Two parallel workers will fetch artifacts per each
mcc-cacheservice endpoint.All artifacts will be fetched, not only those related to OpenStack.
For details about the
CacheWarmupRequestobject, see CacheWarmupRequest resource.Apply the object to the cluster:
kubectl --kubeconfig <pathToManagementKubeconfig> apply -f <pathToFile>
Once done, during deployment and update of MOSK clusters, MOSK management uses cached artifacts from the
mcc-cacheservice to facilitate and speed up the procedure.
When a new MOSK release becomes available and the management cluster auto-updates to a new release, repeat the above steps to predownload a new set of artifacts for MOSK clusters.
Validity of the warmed-up cache
The warmed-up artifacts remain available through the mcc-cache service for
about one month after the cache warm-up, provided that the cache size is
sufficient. After that, the cached artifacts expire, and the first request for
each of them downloads it from the Mirantis repositories again. Therefore, if
the deployment or update of MOSK clusters continues for
longer than one month after the cache warm-up, repeat the cache warm-up before
proceeding.
The registry of the mcc-cache service keeps its own copy of the downloaded
artifacts for 7 days. This copy is used only if an artifact is missing in the
NGINX cache, for example, if the artifact is evicted because of an
insufficient cache size. To verify whether the cache was evicted, compare the
used cache size on each mcc-cache pod with the configured cacheSize:
kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec <mccCachePodName> -c nginx -- du -sh /srv/nginx_cache
If the used cache size reaches the configured cacheSize value, increase it
as described in Increase cache size for mcc-cache and repeat the cache warm-up.
Verify the cache warm-up result
The CacheWarmupRequest object is automatically deleted once the cache
warm-up completes, whether it succeeds or fails. The warm-up summary is stored
in the status:providerStatus:regionalStatus:cacheWarmup section of the
management Cluster object and in the CacheWarmup condition of the
cluster.
To verify the cache warm-up result:
kubectl --kubeconfig <pathToManagementClusterKubeconfig> get cluster <clusterName> \
-o jsonpath='{.status.providerStatus.regionalStatus.cacheWarmup}' | yq -P
Example of system response:
startedAt: "2026-10-06T19:15:27Z"
finishedAt: "2026-10-06T19:57:00Z"
success: false
errors: '["failed to load images list https://binary.mirantis.com/tungsten/bin/artifacts/tungstenfabric-operator-0.20.5.yaml: not found"]'
details:
core/lcm: ok
ceph: ok
openstack: ok
stacklight: ok
tungsten: 'failed: failed to load images list https://binary.mirantis.com/tungsten/bin/artifacts/tungstenfabric-operator-0.20.5.yaml: not found'
The status contains the following fields:
startedAtandfinishedAtStart and finish time of the cache warm-up.
successResult of the cache warm-up:
trueif all image lists are loaded and all images are warmed up,falseotherwise. For the failure reasons, seeerrorsanddetails.
errorsErrors of the cache warm-up, if any: the image lists that failed to load and the images that failed to warm up. The list is truncated to the first 50 errors, each up to 200 characters long. For the full list of errors, see the logs of the
mcc-cache-warmup-job-*job pod in thekaasnamespace.
detailsAvailable since MOSK management 2.32.1. Result of loading the image lists by group:
core/lcmfor the images ofKaaSRelease, andopenstack,ceph,stacklight,tungsten, and so on for the images of the corresponding components of the Cluster releases. Each group has one of the following results:ok- the image lists of the group are loadedfailed: <errors>- at least one image list of the group failed to load
Since MOSK management 2.32.1, a failed image list does not prevent warming up the images of other lists, but the cache warm-up fails in this case. The images of the failed list are not warmed up.
To repeat the cache warm-up, for example, after fixing the access to the
artifacts, create a new CacheWarmupRequest object as described above.
If image pulls through mcc-cache fail with the digest mismatch error,
see Image pull failure due to a corrupted cache.