OpenContrail 4.x: Restore the Cassandra database¶

You may need to restore the Cassandra database after a hardware or software failure.

To restore the Cassandra database:

1. Log in to the Salt Master node.

2. Open your project Git repository with the Reclass model on the cluster level.

3. Add the following snippet to cluster/<cluster_name>/infra/backup/client_cassandra.yml:

   cassandra:
     backup:
       client:
         enabled: true
         restore_latest: 1
         restore_from: remote
   

   where:

   • restore_latest can have, for example, the following values:

     • 1, which means restoring the database from the last complete backup.

     • 2, which means restoring the database from the second latest complete backup.

   • restore_from can have the local or remote values. The remote value uses scp to get the files from the cassandra server.

4. Proceed either with automatic restore steps using the Jenkins web UI pipeline or with manual restore steps:

   • Automatic restore steps:

     1. Verify that the following class is present in cluster/cicd/control/leader.yml:

        classes:
        - system.jenkins.client.job.deploy.update.restore_cassandra
        

        If you manually add this class, apply the changes:

        salt -C 'I@jenkins:client' state.sls jenkins.client
        

     2. Log in to the Jenkins web UI.

     3. Open the cassandra - restore pipeline.

     4. Specify the following parameters:

        Parameter	Description and values
        SALT_MASTER_CREDENTIALS	The Salt Master credentials to use for connection, defaults to salt.
        SALT_MASTER_URL	The Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.

     5. Click Deploy.

   • Manual restore steps:

     1. Stop the supervisor-database service on the OpenContrail control nodes:

        salt -C 'I@opencontrail:control' cmd.run 'doctrail controller systemctl stop contrail-database'
        

     2. Remove the Cassandra files on OpenContrail control nodes:

        salt -C 'I@opencontrail:control' cmd.run 'rm -rf /var/lib/configdb/*'
        

     3. Start the supervisor-database service on the Cassandra client backup node:

        salt -C 'I@cassandra:backup:client' cmd.run 'doctrail controller systemctl start contrail-database'
        

     4. Apply the cassandra state.

        Mirantis recommends running the following command using Linux GNU Screen or alternatives.

        salt -C 'I@cassandra:backup:client' cmd.run "su root -c 'salt-call state.sls cassandra'"
        

        This state restores the databases and creates a file in /var/backups/cassandra/dbrestored.

        Caution

        If you rerun the state, it will not restore the database again. To repeat the restore procedure, first delete the /var/backups/cassandra/dbrestored file and then rerun the cassandra state again.

     5. Reboot the Cassandra backup client role node first:

        salt -C 'I@cassandra:backup:client' system.reboot
        

     6. Reboot the other OpenContrail control nodes:

        salt -C 'I@opencontrail:control and not I@cassandra:backup:client' system.reboot
        

     7. Wait for 60 seconds and restart the supervisor-database service on the OpenContrail control nodes:

        salt -C 'I@opencontrail:control' cmd.run 'doctrail controller systemctl restart contrail-database'
        

     8. Wait for 60 seconds and verify that OpenContrail is in correct state on control nodes:

        salt -C 'I@opencontrail:control' cmd.run 'doctrail controller contrail-status'