Replace the Salt Master and Salt minions SSH RSA keys

Replace the Salt Master and Salt minions SSH RSA keys

This section provides the instruction of how to replace the Salt Master and the Salt minions SSH RSA keys.

To replace Salt Master and Salt minions SSH RSA keys:

  1. Log in to the Salt Master node.

  2. Verify that all nodes are available:

    salt \* test.ping
    
  3. Create classes/cluster/<cluster-name>/infra/minions-maintenance.yml with the following content:

    parameters:
      _param:
        char_number_sign: "#"
      linux:
        system:
          file:
            restart-minion.sh:
              name: /usr/local/bin/restart-minion.sh
              user: root
              group: root
              mode: 750
              contents: |
                ${_param:char_number_sign}!/bin/bash
                /usr/sbin/service salt-minion stop
                rm -f /etc/salt/pki/minion/minion*;
                /usr/sbin/service salt-minion start
          job:
            restart-minion:
              enabled: True
              command: /usr/local/bin/restart-minion.sh
              user: root
              minute: '*/5'
    
  4. Include the minions-maintenance class in the infra/init.yml file:

    classes:
    ...
    - cluster.<cluster-name>.infra.minions-maintenance
    
  5. Put all Salt minions into the maintenance mode:

    salt \* state.sls linux.system.file,linux.system.job
    

    The command above will cause all Salt minions to remove their keys and restart each 5 minutes.

  6. Count your minions:

    MINIONS_NUMBER=$(ls /etc/salt/pki/master/minions/ -1 | wc -l)
    
  7. Verify that all minions are put into the maintenance mode by checking the diff between /master/minions/ and master/minions_denied/:

    diff <(ls /etc/salt/pki/master/minions/ -1 | wc -l) \
    <(ls /etc/salt/pki/master/minions_denied/ -1 | wc -l)
    

    Start the verification at the beginning of the zero or fifth minute to have enough time to purge old minions keys. Proceed only if the diff is empty. If you see the diff for more than 10 minutes, some minions are rejected to execute the cron job. Identify the root cause of the issue and resolve it before proceeding.

  8. Stop the Salt Master node:

    service salt-master stop
    
  9. Change directory to the Salt Master key:

    cd /etc/salt/pki/master
    
  10. Remove the Salt Master key:

    rm -f master.p*
    
  11. Generate a new key without a password:

    ssh-keygen -t rsa -b 4096 -f master.pem
    
  12. Remove the RSA public key for the new key as Salt Master does not require it:

    rm -f master.pem.pub
    
  13. Generate the .pem public key for the Salt Master node:

    openssl rsa -in master.pem -pubout -out master.pub
    

    Note

    Press Enter for the empty password.

  14. Remove the minions list on the Salt Master node:

    salt-key -y -d '*'
    
  15. Start the Salt Master node:

    service salt-master start
    
  16. Verify that the minions are present:

    Note

    The minions should register on the first or sixth minute.

    salt-key -L
    
  17. Verify that the current minions count is the same as in the step 6:

    ls /etc/salt/pki/master/minions/ -1 | wc -l
    echo $MINIONS_NUMBER
    
  18. Disable the maintenance mode for minions by disabling the cron job in classes/cluster/<cluster-name>/infra/minions-maintenance.yml:

    job:
      restart-minion:
        enabled: False
    
  19. Update your minions:

    salt \* state.sls linux.system.job
    
  20. Remove the minions-maintenance class from the infra/init.yml file:

    classes:
    ...
    # Remove the following line
    - cluster.<cluster-name>.minions-maintenance
    
  21. Remove the minions-maintenance pillar definition from the Reclass model:

    rm -f classes/cluster/<cluster-name>/infra/minions-maintenance.yml