Add a Ceph OSD node¶

This section describes how to add a Ceph OSD node to an existing Ceph cluster.

To add a Ceph OSD node:

1. Connect the Ceph OSD salt-minion node to salt-master.

2. In your project repository, if the nodes are not generated dynamically, add the following lines to cluster/ceph/init.yml and modify according to your environment:

   _param:
      ceph_osd_node05_hostname: osd005
      ceph_osd_node05_address: 172.16.47.72
      ceph_osd_node05_backend_address: 10.12.100.72
      ceph_osd_node05_public_address: 10.13.100.72
      ceph_osd_node05_deploy_address: 192.168.0.72
      ceph_osd_system_codename: xenial
   linux:
     network:
       host:
         osd005:
           address: ${_param:ceph_osd_node05_address}
           names:
           - ${_param:ceph_osd_node05_hostname}
           - ${_param:ceph_osd_node05_hostname}.${_param:cluster_domain}
   

   Note

   Skip the ceph_osd_node05_deploy_address parameter if you have DHCP enabled on a PXE network.

3. If the nodes are not generated dynamically, add the following lines to the cluster/infra/config/nodes.yml and modify according to your environment. Otherwise, increase the number of generated OSDs.

   parameters:
     reclass:
       storage:
         node:
           ceph_osd_node05:
             name: ${_param:ceph_osd_node05_hostname}
             domain: ${_param:cluster_domain}
             classes:
             - cluster.${_param:cluster_name}.ceph.osd
             params:
               salt_master_host: ${_param:reclass_config_master}
               linux_system_codename:  ${_param:ceph_osd_system_codename}
               single_address: ${_param:ceph_osd_node05_address}
               deploy_address: ${_param:ceph_osd_node05_deploy_address}
               backend_address: ${_param:ceph_osd_node05_backend_address}
               ceph_public_address: ${_param:ceph_osd_node05_public_address}
               ceph_crush_parent: rack02
   

   Note

   Skip the deploy_address parameter if you have DHCP enabled on a PXE network.

4. ^{Since 2019.2.3, skip this step} Verify that the cluster/ceph/osd.yml file and the pillar of the new Ceph OSD do not contain the following lines:

   parameters:
     ceph:
       osd:
         crush_update: false
   

5. Log in to the Jenkins web UI.

6. Select from the following options:

   • For MCP versions starting from the 2019.2.10 maintenance update, open the Ceph - add osd (upmap) pipeline.
   • For MCP versions prior to the 2019.2.10 maintenance update, open the Ceph - add node pipeline.

   Note

   Prior to the 2019.2.10 maintenance update, the Ceph - add node and Ceph - add osd (upmap) Jenkins pipeline jobs are available as technical preview only.

   Caution

   A large change in the crush weights distribution after the addition of Ceph OSDs can cause massive unexpected rebalancing, affect performance, and in some cases can cause data corruption. Therefore, if you are using Ceph - add node, Mirantis recommends that you add all disks with zero weight and reweight them gradually.

7. Specify the following parameters:

   Parameter	Description and values
   SALT_MASTER_CREDENTIALS	The Salt Master credentials to use for connection, defaults to salt.
   SALT_MASTER_URL	The Salt Master node host URL with the salt-api port, defaults to the jenkins_salt_api_url parameter. For example, http://172.18.170.27:6969.
   HOST	Add the Salt target name of the new Ceph OSD. For example, osd005*.
   HOST_TYPE Removed since 2019.2.3 update	Add osd as the type of Ceph node that is going to be added.
   CLUSTER_FLAGS Added since 2019.2.7 update	Add a comma-separated list of flags to check after the pipeline execution.
   USE_UPMAP Added since 2019.2.13 update	Use to facilitate the upmap module during rebalancing to minimize impact on cluster performance.

8. Click Deploy.

   The Ceph - add node pipeline workflow prior to the 2019.2.3 maintenance update:

   1. Apply the reclass state.
   2. Apply the linux, openssh, salt, ntp, rsyslog, ceph.osd states.

   The Ceph - add node pipeline workflow starting from 2019.2.3 maintenance update:

   1. Apply the reclass state.
   2. Verify that all installed Ceph clients have the Luminous version.
   3. Apply the linux, openssh, salt, ntp, rsyslog, states.
   4. Set the Ceph cluster compatibility to Luminous.
   5. Switch the balancer module to the upmap mode.
   6. Set the norebalance flag before adding a Ceph OSD.
   7. Apply the ceph.osd state on the selected Ceph OSD node.
   8. Update the mappings for the remapped placement group (PG) using upmap back to the old Ceph OSDs.
   9. Unset the norebalance flag and verify that the cluster is healthy.

9. If you use a custom CRUSH map, update the CRUSH map:

   1. Verify the updated /etc/ceph/crushmap file on cmn01. If correct, apply the CRUSH map using the following commands:

      crushtool -c /etc/ceph/crushmap -o /etc/ceph/crushmap.compiled
      ceph osd setcrushmap -i /etc/ceph/crushmap.compiled
      

   2. Add the following lines to the cluster/ceph/osd.yml file:

      parameters:
        ceph:
          osd:
            crush_update: false
      

   3. Apply the ceph.osd state to persist the CRUSH map:

      salt -C 'I@ceph:osd' state.sls ceph.osd
      

10. Integrate the Ceph OSD nodes with StackLight:

    1. Update the Salt mine:

       salt -C 'I@ceph:osd or I@telegraf:remote_agent' state.sls salt.minion.grains
       salt -C 'I@ceph:osd or I@telegraf:remote_agent' saltutil.refresh_modules
       salt -C 'I@ceph:osd or I@telegraf:remote_agent' mine.update
       

       Wait for one minute.

    2. Apply the following states:

       salt -C 'I@ceph:osd or I@telegraf:remote_agent' state.sls telegraf
       salt -C 'I@ceph:osd' state.sls fluentd
       salt 'mon*' state.sls prometheus