1. Home
  2. MCP Operations Guide
  3. Upgrade and update an MCP cluster
  4. Upgrade an MCP cluster
  5. Upgrade DriveTrain to a newer release version
  6. Rollback MCP to a previous release version

Rollback MCP to a previous release version

Rollback MCP to a previous release version

You can rollback your MCP deployment to a previous MCP release version through DriveTrain using the Deploy - update cloud pipeline.

To rollback to the previous stable MCP release version:

  1. Verify that the correct previous Build ID release repositories are available. These include the local repositories present in the mirror image backup as well as the local aptly repositories.

  2. In the infra/init.yml file of the Reclass model, specify the previous Build ID in the mcp_version parameter.

  3. In the infra/init.yml file of the Reclass model, verify that the following pillar is present.

    parameters:
  linux:
    system:
      purge_repos: true

  4. Update the classes/system Git submodule of the Reclass model to the commit of the required Build ID by running the following command from the classes/system directory:

    git pull origin release/BUILD_ID

  5. Commit and push the changes to the Git repository where the cluster Reclass model is located.

  6. Select from the following options:

    • The ROLLBACK_BY_REDEPLOY was not selected for the update pipeline:

      1. Roll back the Salt Master node:

        1. On the KVM node hosting the Salt Master node, run the following commands:

          virsh destroy cfg01.domain
virsh define /var/lib/libvirt/images/cfg01.domain.xml
virsh start cfg01.domain; virsh snapshot-delete cfg0X.domain --metadata ${SNAPSHOT_NAME}
rm /var/lib/libvirt/images/cfg01.domain.${SNAPSHOT_NAME}.qcow2
rm /var/lib/libvirt/images/cfg01.domain.xml

        2. On the Salt Master node, apply the linux.system.repo Salt state.

      2. Roll back the CI/CD nodes:

        1. On the KVM nodes hosting the CI/CD nodes, run the following commands:

          virsh destroy cid0X.domain
virsh define /var/lib/libvirt/images/cid0X.domain.xml
virsh start cid0X.domain
virsh snapshot-delete cid0X.domain --metadata ${SNAPSHOT_NAME}
rm /var/lib/libvirt/images/cid0X.domain.${SNAPSHOT_NAME}.qcow2
rm /var/lib/libvirt/images/cid0X.domain.xml

        2. On all CI/CD nodes, restart the docker service and apply the linux.system.repo Salt state.

    • The ROLLBACK_BY_REDEPLOY parameter was selected for the update pipeline:

      1. Roll back the Salt Master node as described in Back up and restore the Salt Master node.

      2. Redeploy the CI/CD nodes:

        1. Virsh destroy and undefine the cid nodes:

          virsh destroy cid<NUM>.<DOMAIN_NAME>
virsh undefine cid<NUM>.<DOMAIN_NAME>

        2. Remove the cid salt-keys from the Salt Master node.

        3. Redeploy the CI/CD nodes using the Deploy CI/CD procedure.

  7. Run the Deploy - update cloud pipeline in the Jenkins web UI specifying the following parameters as required:

    Deploy - update cloud pipeline parameters
    Parameter Description
    CEPH_OSD_TARGET The Salt targeted physical Ceph OSD osd nodes.
    CID_TARGET The Salt targeted CI/CD cid nodes.
    CMN_TARGET The Salt targeted Ceph monitor cmn nodes.
    CMP_TARGET The Salt targeted physical compute cmp nodes.
    CTL_TARGET The Salt targeted controller ctl nodes.
    DBS_TARGET The Salt targeted database dbs nodes.
    GTW_TARGET The Salt targeted physical or virtual gateway gtw nodes.
    INTERACTIVE Ask interactive questions during the pipeline run. If not selected, the pipeline will either succeed or fail.
    KVM_TARGET The Salt targeted physical KVM kvm nodes.
    LOG_TARGET The Salt targeted log storage and visualization log nodes.
    MON_TARGET The Salt targeted StackLight LMA monitoring node mon nodes.
    MSG_TARGET The Salt targeted RabbitMQ server msg nodes.
    MTR_TARGET The Salt targeted StackLight LMA metering mtr nodes.
    NAL_TARGET The Salt targeted OpenContrail 3.2 analytics nal nodes.
    NTW_TARGET The Salt targeted OpenContrail 3.2 controller ntw nodes.
    PER_NODE Target nodes will be managed one by one. Recommended.
    PRX_TARGET The Salt targeted proxy prx nodes.
    RGW_TARGET The Salt targeted RADOS gateway rgw nodes.
    ROLLBACK_BY_REDEPLOY Select if live snapshots were taken during update.
    SALT_MASTER_URL URL of Salt Master node API.
    SALT_MASTER_CREDENTIALS ID of the Salt Master node API credentials stored in Jenkins.
    SNAPSHOT_NAME Live snapshot name.
    STOP_SERVICES Stop API services before the rollback.
    PURGE_PKGS The space-separated list of pkgs=versions to be purged on the physical targeted machines. For example, pkg_name1=pkg_version1 pkg_name2=pkg_version2.
    REMOVE_PKGS The space-separated list of pkgs=versions to be removed on the physical targeted machines. For example, pkg_name1=pkg_version1 pkg_name2=pkg_version2.
    RESTORE_CONTRAIL_DB

    Restore the Cassandra and ZooKeeper databases for OpenContrail 3.2. OpenContrail 4.x is not supported. Select only if rollback of the OpenContrail 3.2 controller nodes failed or a specific backup defined in the cluster model is required.

    If RESTORE_CONTRAIL_DB is selected, add the following configuration to the cluster/opencontrail/control.yml file of your Reclass model:

    • For ZooKeeper:

      parameters:
  zookeeper:
    backup:
      client:
        enabled: true
        restore_latest: 1
        restore_from: remote

    • For Cassandra:

      parameters:
  cassandra:
    backup:
      client:
        enabled: true
        restore_latest: 1
        restore_from: remote
    RESTORE_GALERA

    Restore the Galera database. Select only if the rollback of the database nodes failed or a specific backup defined in the cluster model is required.

    If RESTORE_GALERA is selected, add the xtrabackup restore lines to the cluster/openstack/database/init.yml of your Reclass model:

    parameters:
  xtrabackup:
    client:
      enabled: true
      restore_full_latest: 1
      restore_from: remote
    ROLLBACK_PKG_VERSIONS

    Copy back the list of package versions installed before the update acquired from the pipeline before pkgs were upgraded. The space-separated list of pkgs=versions to roll back to on the physical targeted machines. For example, pkg_name1=pkg_version1 pkg_name2=pkg_version2.

    If ROLLBACK_PKG_VERSIONS is empty, apt --allow-downgrades dist-upgrade will be run on the targeted physical machines.

    If ROLLBACK_PKG_VERSIONS contains the salt-minion package, you will have to rerun the pipeline for every targeted physical machine as it will be disconnected.
    TARGET_ROLLBACKS The comma-separated list of nodes to rollback. The valid values include ctl, prx, msg, dbs, log, mon, mtr, ntw, nal, gtw-virtual, cmn, rgw, cmp, kvm, osd, gtw-physical.
    TARGET_REBOOT

    The comma-separated list of physical nodes to reboot after a rollback. The valid values include cmp, kvm, osd, gtw-physical.

    Caution

    When the kvm node is defined, the pipeline can be interrupted due to the Jenkins slave reboot. If so, remove the already updated nodes from TARGET_UPDATES and rerun the pipeline.
    TARGET_HIGHSTATE The comma-separated list of physical nodes to run Salt highstate on after a rollback. The valid values include cmp, kvm, osd, gtw-physical.

    Common rollback workflow for different nodes types:

    1. Downtime for VCP occurs if a rollback for VCP VMs is required.
    2. If the ROLLBACK_BY_REDEPLOY parameter is selected, the VCP VMs will be destroyed, undefined, and their salt-key will be deleted from the Salt Master node.
    3. Verification of the service or API status is done.

    The procedure of the pipeline if ROLLBACK_BY_REDEPLOY is not selected:

    1. VCP VMs by their target type are destroyed.
    2. Live snapshot is deleted if any and its original base file is used to boot the VM.
    3. Repositories are updated.
    4. Physical machines are rolled back:
      1. Repositories are updated.
      2. Packages defined in PURGE_PKGS are purged.
      3. Packages defined in REMOVE_PKGS are removed.
      4. Package versions defined in ROLLBACK_PKG_VERSIONS are installed.
      5. The Salt highstate is applied.

  8. Verify that the following lines are not present in cluster/infra/backup/client_mysql.yml:

    parameters:
  xtrabackup:
    client:
      cron: false

  9. Verify that the following lines are not present in cluster/infra/backup/server.yml:

    parameters:
  xtrabackup:
    server:
      cron: false

  10. If OpenContrail 3.2 is used:

    1. Verify that the following lines are not present in cluster/infra/backup/client_zookeeper.yml and cluster/infra/backup/server.yml:

      parameters:
  zookeeper:
    backup:
      cron: false

    2. Verify that the following lines are not present in cluster/infra/backup/client_cassandra.yml and cluster/infra/backup/server.yml:

      parameters:
  cassandra:
    backup:
      cron: false

  11. If the ROLLBACK_BY_REDEPLOY parameter was selected and the Deploy - cloud update pipeline succeeds, continue the rollback:

    1. Roll back the Salt Master node as described in Back up and restore the Salt Master node procedure if necessary.

    2. If Ceph is enabled and you want to rollback Ceph monitoring nodes, proceed with Restore a Ceph Monitor node.

    3. Redeploy the nodes based on your model by running the Deploy - OpenStack pipeline. See the pipeline configuration details in Deploy an OpenStack environment, step 10.

      Note

      Specify k8s in the Install parameter field in case of an MCP Kubernetes deployment.

    4. Rerun the Deploy - cloud update pipeline with RESTORE_GALERA and RESTORE_CONTRAIL_DB selected.

updated: 2024-01-04 11:47