You may need to roll back Ceph components to a previous version if the upgrade procedure fails. To successfully return Ceph to the previous version, you must manually roll back all Ceph components, including Ceph Monitor nodes, Ceph RADOS Gateway nodes, and Ceph OSD nodes. If the Ceph storage upgrade fails, you can roll back the Ceph VMs (the Monitor and RADOS Gateway nodes) and Ceph OSDs.
During the upgrade, the backed up Ceph VMs are copied to the root directory of the KVM nodes on which the Ceph nodes reside.
To roll back the Ceph VMs:
Log in to the KVM node.
Copy the backed up disks to the directory with libvirt. For example,
to /var/lib/libvirt/images/cmn04.local/system.qcow2.
Start the instance:
virsh start <NODE_NAME>.<NODE_DOMAIN>
To roll back the Ceph OSDs:
Manually install Ceph packages of the previous version:
apt install <CEPH_PACKAGE_1>=<VERSION> <CEPH_PACKAGE_2>=<VERSION>
Restart services for all OSDs:
systemctl restart ceph-osd@<osd_num>
If the step 1 fails, perform the following steps:
Purge Ceph packages:
apt purge <CEPH_PACKAGE_1> <CEPH_PACKAGE_2>
Manually install Ceph packages of the previous version:
apt install <CEPH_PACKAGE_1>=<VERSION> <CEPH_PACKAGE_2>=<VERSION>
Follow the steps described in Restore the metadata of a Ceph OSD node. Alternatively, if restoring from a backup is not possible:
Purge Ceph packages:
apt purge <CEPH_PACKAGE_1> <CEPH_PACKAGE_2>
Verify that no Ceph devices are mounted:
umount <device_partition_path>
Run the following Salt states to redeploy the Ceph OSD node:
salt -C 'I@ceph:osd' state.sls ceph.osd
salt -C 'I@ceph:osd' saltutil.sync_grains
salt -C 'I@ceph:osd' state.sls ceph.osd.custom
salt -C 'I@ceph:osd' saltutil.sync_grains
salt -C 'I@ceph:osd' mine.update
salt -C 'I@ceph:setup' state.sls ceph.setup