Image pull failure due to a corrupted cache

Applies since MOSK management 2.32.1

The mcc-cache service of a management cluster does not verify the cached data when serving it, while MOSK clusters verify the digest of every downloaded image layer. If the mcc-cache data is corrupted, for example, after a disk failure on a management cluster node, image pulls through mcc-cache fail with a digest mismatch error, such as unexpected commit digest. The events of the affected pod contain an error similar to the following:

Failed to pull image "<registry>/<repository>:<tag>": failed to pull and
unpack image "<registry>/<repository>:<tag>": failed to copy: failed commit
on ref "layer-sha256:<digest>": unexpected commit digest
sha256:<actualDigest>, expected sha256:<digest>: failed precondition

Each mcc-cache pod keeps its own copy of the data in the NGINX cache and in the registry storage on the mcc-cache volume. Remove the corrupted data as described below. The removed data is downloaded again on the next request.

Clean up a single image layer

Use this procedure if the error refers to a specific image layer.

  1. From the error message, obtain the image repository without the registry host name, for example, openstack/heat, and the expected digest of the layer, for example, sha256:fb694a48a0afc48a91b49faf0b0ebb39c04670b2e7999cc0800714e0914e77d7:

    REPO=<repository>
    DIGEST=sha256:<digest>
    
  2. Remove the layer from the NGINX cache and from the registry storage of every mcc-cache pod:

    for pod in $(kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas get pods -l app.kubernetes.io/name=mcc-cache -o name); do
      kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec ${pod} -c nginx -- sh -c \
        "find /srv/nginx_cache -type f | while read -r f; do head -c 2048 \"\$f\" | grep -aq '^KEY: .*/v2/${REPO}/blobs/${DIGEST}\$' && rm -fv \"\$f\"; done; true"
      kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec ${pod} -c registry -- \
        rm -rfv /var/lib/registry/docker/registry/v2/blobs/sha256/${DIGEST:7:2}/${DIGEST#sha256:}
    done
    

    The command removes the layer from all mcc-cache pods because the error message does not indicate which pod served the corrupted data.

  3. Retry the image pull, for example, by deleting the affected pod. If the error refers to another layer, repeat the procedure for that layer.

Clean up all data of an mcc-cache pod

Use this procedure if the error refers to different layers or images, for example, after a disk failure on a management cluster node.

Caution

After the cleanup, the mcc-cache pod has no cached data, and subsequent image pulls through this pod download the images from the internet.

Clean up the data of one pod at a time and warm up the cache again before deploying or updating MOSK clusters. For details, see Warm up the MOSK management cache.

To clean up all data of an mcc-cache pod:

  1. Remove the NGINX cache and the registry storage of the mcc-cache pod:

    kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec <mccCachePodName> -c nginx -- sh -c 'rm -rf /srv/nginx_cache/*'
    kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas exec <mccCachePodName> -c registry -- sh -c 'rm -rf /var/lib/registry/docker'
    
  2. Restart the pod to rebuild the in-memory index of the NGINX cache:

    kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n kaas delete pod <mccCachePodName>
    
  3. Wait for the pod to become Running and repeat the procedure for other affected mcc-cache pods, if any.