Manual migration of the AWS-based Docker data and LVP volumes to an NVMe disk

Although the Docker data and LVP volumes are automatically migrated to an NVMe disk during an AWS-based cluster upgrade, this process may require manual intervention. The data migration fails when a cluster node is stuck during upgrade and the corresponding LCMMachine contains Ansible-related errors with the failed Fail if data migration is incomplete task.

To verify the migration of Docker data and LVP volumes to an NVMe disk:

  1. Analyze Ansible logs to ensure that the data became consistent on the old and new disks.

  2. If you notice inconsistencies, verify that the following migration workflow is applied. If you notice that a step is not applied automatically, finish the migration manually.

    Warning

    Run the commands below only after making sure that a specific step was not applied automatically. Otherwise, the data may be lost.

    1. Stop the Docker Engine:

      systemctl stop docker-engine
      
    2. Unmount all mount points in the /mnt/local-volume directory:

      awk '/local-volume/ {print $2}' /etc/fstab | sort | uniq | xargs -I% umount -v %
      
    3. Verify whether /etc/fstab has legacy entries. For example, if it uses the /var/lib/local-volumes directory.

      If legacy entries exist:

      1. Copy data from /var/lib/local-volumes/ to /mnt/local-volumes/src:

        rsync --ignore-existing -aHAX /var/lib/local-volumes/ /mnt/local-volumes/src
        
      2. Remove legacy entries that contain /var/lib/local-volumes/ from /etc/fstab.

    4. Create temporary mount points for:

      • /dev/mapper/nvme_instance_storage-local_volumes for local volumes storage

      • /dev/mapper/nvme_instance_storage-docker for Docker data storage

    5. Copy data from /mnt/local-volumes/ to the local volumes storage:

      rsync -aHAX /mnt/local-volumes/ <tempMountPointPath>
      
    6. Copy data from /var/lib/docker/ to the Docker data storage:

      rsync -aHAX /var/lib/docker/ <tempMountPointPath>
      
    7. Remove or back up all data from /mnt/local-volumes/ and /var/lib/docker/ directories.

    8. Unmount temporary mount points for the Docker and local storage data:

      umount <tempMountPointPath>
      
    9. Mount local volumes and Docker storage to the permanent mount points:

      • With 755 permissions for /mnt/local-volumes, run:

        mount /dev/mapper/nvme_instance_storage-local_volumes /mnt/local-volumes
        
      • With 711 permissions for /var/lib/docker, run:

        mount /dev/mapper/nvme_instance_storage-docker /var/lib/docker
        
    10. Add the corresponding entries to /etc/fstab.

    11. Start the Docker Engine:

      systemctl start docker-engine
      
    12. Rename the /root/data_migration_in_progress file to /root/data_migration_completed.

      Ansible playbooks will be restarted automatically and the cluster upgrade will resume.