1. Home
  2. MCP Operations Guide
  3. Upgrade and update an MCP cluster
  4. Update an MCP cluster
  5. Update Ceph

Update Ceph

Update Ceph

Starting from the Build ID 2019.2.0, you can update Ceph packages to the latest minor versions on the Ceph OSD, Monitor, and RADOS Gateway nodes using the Update Ceph packages pipeline job. During the update, all Ceph services restart one by one and the pipeline does not proceed until the cluster is healthy.

Warning

If you are updating Ceph from a version prior 14.2.20, verify that the ceph:common:config:mon:auth_allow_insecure_global_id_reclaim pillar is unset or set to true.

Note

This procedure does not include the backup of Ceph VMs. To back up the Ceph VMs, perform the steps described in Back up and restore Ceph.

To update Ceph packages to the latest minor versions:

  1. Verify that you have upgraded your MCP cluster to the latest Build ID as described in Upgrade DriveTrain to a newer release version.

  2. Log in to the Jenkins web UI.

  3. Open the Update Ceph packages pipeline job.

  4. Specify the following parameters:

    Parameter Description and values
    SALT_MASTER_CREDENTIALS Credentials for Salt API stored in Jenkins.
    SALT_MASTER_URL The URL of Salt API. For example, https://10.10.10.1:8000.
    TARGET_SERVERS Removed since 2019.2.5 update Salt compound target to match the nodes to be updated. For example, G@osfamily:debian or * to update the packages on all Ceph nodes.
    CLUSTER_FLAGS Added since 2019.2.7 update Add a comma-separated list of flags to apply before and after the pipeline execution.

  5. Click Build.

  6. Click Full stage view to track the update process.

Caution

The Jenkins pipeline job changes repositories, runs upgrade packages, and restarts the service on each node of selected groups. As the pipeline installs packages from configured repositories and does not verify the version to install, for some environments the version of Ceph packages can differ between nodes after upgrade. It can affect one node due to manual configuration without a model or an entire group, like Ceph OSD nodes, due to a misconfiguration in Reclass. It is possible to run a cluster with mismatching versions. However, such configuration is not supported and, with a specific version, may cause cluster outage.

The Jenkins pipeline job provides information about packages versions change in the console output for each node. Consider checking them before proceeding to the next node, especially on the first node of each component.

The following table contains the details of the update stages:

The Update Ceph packages pipeline job workflow
# Stage Details
1 List the target servers
  1. Selects the Ceph nodes from the available targets.
  2. Obtains the minions from TARGET_SERVERS to list all available targets and then selects the ones that include rgw, cmn, or osd in the name.
2 Apply package upgrades on all nodes Updates and installs new Ceph packages on the selected nodes.
3 Restart the services
  1. Restarts Ceph Monitor on all cmn nodes one by one.
  2. Starting from the 2019.2.8 update, restarts Ceph Manager on all mgr nodes one by one.
  3. Restarts Ceph OSDs on all osd nodes one by one.

After a service restart on each node, the Jenkins pipeline job waits for the cluster to become healthy.
updated: 2023-01-30 16:05