Image pull failure due to a corrupted cache
Applies since MOSK management 2.32.1
The mcc-cache service of a management cluster does not verify the cached
data when serving it, while MOSK clusters verify the digest
of every downloaded image layer. If the mcc-cache data is corrupted, for
example, after a disk failure on a management cluster node, image pulls
through mcc-cache fail with a digest mismatch error, such as
unexpected commit digest. The events of the affected pod contain an error
similar to the following:
Failed to pull image "<registry>/<repository>:<tag>": failed to pull and
unpack image "<registry>/<repository>:<tag>": failed to copy: failed commit
on ref "layer-sha256:<digest>": unexpected commit digest
sha256:<actualDigest>, expected sha256:<digest>: failed precondition
Each mcc-cache pod keeps its own copy of the data in the NGINX cache and
in the registry storage on the mcc-cache volume. Remove the corrupted
data as described below. The removed data is downloaded again on the next
request.
Clean up a single image layer
Use this procedure if the error refers to a specific image layer.
From the error message, obtain the image repository without the registry host name, for example,
openstack/heat, and the expected digest of the layer, for example,sha256:fb694a48a0afc48a91b49faf0b0ebb39c04670b2e7999cc0800714e0914e77d7:REPO=<repository> DIGEST=sha256:<digest>
Remove the layer from the NGINX cache and from the registry storage of every
mcc-cachepod:for pod in $(kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas get pods -l app.kubernetes.io/name=mcc-cache -o name); do kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec ${pod} -c nginx -- sh -c \ "find /srv/nginx_cache -type f | while read -r f; do head -c 2048 \"\$f\" | grep -aq '^KEY: .*/v2/${REPO}/blobs/${DIGEST}\$' && rm -fv \"\$f\"; done; true" kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec ${pod} -c registry -- \ rm -rfv /var/lib/registry/docker/registry/v2/blobs/sha256/${DIGEST:7:2}/${DIGEST#sha256:} done
The command removes the layer from all
mcc-cachepods because the error message does not indicate which pod served the corrupted data.Retry the image pull, for example, by deleting the affected pod. If the error refers to another layer, repeat the procedure for that layer.
Clean up all data of an mcc-cache pod
Use this procedure if the error refers to different layers or images, for example, after a disk failure on a management cluster node.
Caution
After the cleanup, the mcc-cache pod has no cached data, and
subsequent image pulls through this pod download the images from the
internet.
Clean up the data of one pod at a time and warm up the cache again before deploying or updating MOSK clusters. For details, see Warm up the MOSK management cache.
To clean up all data of an mcc-cache pod:
Remove the NGINX cache and the registry storage of the
mcc-cachepod:kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec <mccCachePodName> -c nginx -- sh -c 'rm -rf /srv/nginx_cache/*' kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec <mccCachePodName> -c registry -- sh -c 'rm -rf /var/lib/registry/docker'
Restart the pod to rebuild the in-memory index of the NGINX cache:
kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas delete pod <mccCachePodName>
Wait for the pod to become
Runningand repeat the procedure for other affectedmcc-cachepods, if any.