This section provides the instruction of how to replace the Salt Master and the Salt minions SSH RSA keys.
To replace Salt Master and Salt minions SSH RSA keys:
Log in to the Salt Master node.
Verify that all nodes are available:
salt \* test.ping
Create classes/cluster/<cluster-name>/infra/minions-maintenance.yml
with the following content:
parameters:
_param:
char_number_sign: "#"
linux:
system:
file:
restart-minion.sh:
name: /usr/local/bin/restart-minion.sh
user: root
group: root
mode: 750
contents: |
${_param:char_number_sign}!/bin/bash
/usr/sbin/service salt-minion stop
rm -f /etc/salt/pki/minion/minion*;
/usr/sbin/service salt-minion start
job:
restart-minion:
enabled: True
command: /usr/local/bin/restart-minion.sh
user: root
minute: '*/5'
Include the minions-maintenance
class in the infra/init.yml
file:
classes:
...
- cluster.<cluster-name>.infra.minions-maintenance
Put all Salt minions into the maintenance mode:
salt \* state.sls linux.system.file,linux.system.job
The command above will cause all Salt minions to remove their keys and restart each 5 minutes.
Count your minions:
MINIONS_NUMBER=$(ls /etc/salt/pki/master/minions/ -1 | wc -l)
Verify that all minions are put into the maintenance mode by checking the
diff between /master/minions/
and master/minions_denied/
:
diff <(ls /etc/salt/pki/master/minions/ -1 | wc -l) \
<(ls /etc/salt/pki/master/minions_denied/ -1 | wc -l)
Start the verification at the beginning of the zero or fifth minute to have enough time to purge old minions keys. Proceed only if the diff is empty. If you see the diff for more than 10 minutes, some minions are rejected to execute the cron job. Identify the root cause of the issue and resolve it before proceeding.
Stop the Salt Master node:
service salt-master stop
Change directory to the Salt Master key:
cd /etc/salt/pki/master
Remove the Salt Master key:
rm -f master.p*
Generate a new key without a password:
ssh-keygen -t rsa -b 4096 -f master.pem
Remove the RSA public key for the new key as Salt Master does not require it:
rm -f master.pem.pub
Generate the .pem
public key for the Salt Master node:
openssl rsa -in master.pem -pubout -out master.pub
Note
Press Enter for the empty password.
Remove the minions list on the Salt Master node:
salt-key -y -d '*'
Start the Salt Master node:
service salt-master start
Verify that the minions are present:
Note
The minions should register on the first or sixth minute.
salt-key -L
Verify that the current minions count is the same as in the step 6:
ls /etc/salt/pki/master/minions/ -1 | wc -l
echo $MINIONS_NUMBER
Disable the maintenance mode for minions by disabling the cron job
in classes/cluster/<cluster-name>/infra/minions-maintenance.yml
:
job:
restart-minion:
enabled: False
Update your minions:
salt \* state.sls linux.system.job
Remove the minions-maintenance
class from the infra/init.yml
file:
classes:
...
# Remove the following line
- cluster.<cluster-name>.minions-maintenance
Remove the minions-maintenance
pillar definition from the Reclass
model:
rm -f classes/cluster/<cluster-name>/infra/minions-maintenance.yml