Restore the Salt Master node

Restore the Salt Master node

You may need to restore the Salt Master node after a hardware or software failure.

To restore the Salt Master node from a Backupninja rsync backup:

  1. Redeploy the Salt Master node using the day01 image with the configuration ISO drive for the Salt Master VM as described in Deploy the Salt Master node.

    Caution

    Make sure to securely back up the configuration ISO drive image. This image contains critical information required to re-install your cfg01 node in case of storage failure, including master key for all encrypted secrets in the cluster metadata model.

    Failure to back up the configuration ISO image may result in loss of ability to manage MCP in certain hardware failure scenarios.

  2. Log in to the Salt Master node.

  3. Configure your deployment model by including the following pillar in cluster/infra/config/init.yml:

    parameters:
      salt:
        master:
          initial_data:
            engine: backupninja
            source: kvm03       # the backupninja server that stores Salt Master backups
            host: cfg01.<domain_name>  # for example: cfg01.deploy-name.local
        minion:
          initial_data:
            engine: backupninja
            source: kvm03       # the backupninja server that stores Salt Master backups
            host: cfg01.<domain_name>  # for example: cfg01.deploy-name.local
    
  4. Verify that the pillar for Backupninja is present:

    salt-call pillar.data backupninja
    

    If the pillar is not present, configure it as described in Enable a backup schedule for the Salt Master node using Backupninja.

  5. Verify that the pillar for master and minion is present:

    salt-call pillar.data salt:minion:initial_data
    salt-call pillar.data salt:master:initial_data
    

    If the pillar is not present, verify the pillar configuration in cluster/infra/config/init.yml described above.

  6. Apply the salt.master.restore and salt.minion.restore states:

    salt-call state.sls salt.master.restore,salt.minion.restore
    

    Running the states above restores the Salt Master node`s PKI and CA certificates and creates files as a flag in the /srv/salt/ directory that indicates the Salt Master node restore is completed.

    Caution

    If you rerun the state, it will not restore the Salt Master node again. To repeat the restore procedure, first delete the master-restored and minion-restored files from the /srv/salt directory and rerun the above states.

  7. Verify that the Salt Master node is restored:

    salt-key
    salt -t2 '*' saltutil.refresh_pillar
    ls -la /etc/pki/ca/salt_master_ca/