This section describes how to verify the status of the GlusterFS volumes and troubleshoot issues if any.
To verify the GlusterFS volumes status:
Log in to the Salt Master node.
Verify the GlusterFS volumes status:
salt -C 'I@glusterfs:server' cmd.run "gluster volume status all"
If the system output contains issues, such as in the example below, or/and the volume status cannot be retrieved, refer to the GlusterFS official documentation to resolve the issues.
Example of system response:
kvm01.cookied-cicd-bm-os-contrail40-maas.local:
Another transaction is in progress for aptly. Please try again after sometime.
Another transaction is in progress for gerrit. Please try again after sometime.
Another transaction is in progress for keystone-credential-keys. Please try again after sometime.
Another transaction is in progress for mysql. Please try again after sometime.
Another transaction is in progress for registry. Please try again after sometime.
Locking failed on 10.167.8.243. Please check log file for details.
Inspect the GlusterFS server logs for volumes at
/var/log/glusterfs/bricks/srv-glusterfs-<volume name>.log on the
kvm nodes.
In case of any issues with the replication status of GlusterFS, stop all the GlusterFS volume-related services to prevent data corruption and immediately proceed with a troubleshooting to restore the volume in question.
If you need to reboot a kvm that hosts GlusterFS, verify that all
GlusterFS clients have volumes mounted after a node reboot:
Log in to the Salt Master node.
Identify the GlusterFS VIP address:
salt-call pillar.get
Example of system response:
_param:infra_kvm_address
local:
10.167.8.240
Verify that all GlusterFS clients have volumes mounted. For example:
salt -C 'I@glusterfs:client and not I@glusterfs:server' cmd.run "mount | grep 10.167.8.240"
Example of system response:
prx02.cookied-cicd-bm-os-contrail40-maas.local:
10.167.8.240:/salt_pki on /srv/salt/pki type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
ctl01.cookied-cicd-bm-os-contrail40-maas.local:
10.167.8.240:/keystone-credential-keys on /var/lib/keystone/credential-keys type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
10.167.8.240:/keystone-keys on /var/lib/keystone/fernet-keys type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
cid03.cookied-cicd-bm-os-contrail40-maas.local:
prx01.cookied-cicd-bm-os-contrail40-maas.local:
10.167.8.240:/salt_pki on /srv/salt/pki type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
cid01.cookied-cicd-bm-os-contrail40-maas.local:
cid02.cookied-cicd-bm-os-contrail40-maas.local:
ctl03.cookied-cicd-bm-os-contrail40-maas.local:
cfg01.cookied-cicd-bm-os-contrail40-maas.local:
10.167.8.240:/salt_pki on /srv/salt/pki type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
ctl02.cookied-cicd-bm-os-contrail40-maas.local:
In the example above, several VMs do not have the GlusterFS volumes
mounted, for example, cid03, cid01, cid02. In such case,
reboot the corresponding VMs and verify the status again.
Verify that all mounted volumes identified in the previous step match the pillar information on the corresponding VM. For example:
salt prx02.cookied-cicd-bm-os-contrail40-maas.local pillar.get glusterfs:client:volumes
Example of system response:
----------
salt_pki:
----------
opts:
defaults,backup-volfile-servers=10.167.8.241:10.167.8.242:10.167.8.243
path:
/srv/salt/pki
server:
10.167.8.240
In the output above, the server IP address and the only mounted
salt_pki volume information match the output for the prx02 VM
shown in the previous step.
Caution
Do not proceed to reboot the next kvm node that hosts
the GlusterFS cluster before all volumes are mounted on
VMs of the first kvm node.