Upgrade OpenStack¶

This section provides instructions on how to upgrade OpenStack to a major version on a MOSK cluster.

Note

The update of the OpenStack components within the same major OpenStack version is performed seamlessly as part of the MOSK cluster update.

Prerequisites¶

Verify that your OpenStack cloud is running on the latest MOSK release. See Release Compatibility Matrix for the release matrix and supported upgrade paths.
Just before the upgrade, back up your OpenStack databases. See the following documentation for details:
- Reference Architecture: OpenStack database
- Operations Guide: Back up and restore an OpenStack database
Verify that OpenStack is healthy and operational. All OpenStack components in the health group in the OpenStackDeploymentStatus CR should be in the Ready state. See OpenStackDeploymentStatus custom resource for details.
Verify the workability of your OpenStack deployment by running Tempest against the OpenStack cluster as described in Run Tempest tests. Verification of the testing pass rate before upgrading will help you measure your cloud quality before and after upgrade.
Read carefully through the Release Notes of your MOSK version paying attention to the Known issues section and the OpenStack upstream release notes for the target OpenStack version.
Calculate the maintenance window using Plan the cluster update and Calculate a maintenance window duration for update Deprecated and notify users.
When upgrading to OpenStack Yoga, remove the Panko service from the cloud by removing the event entry from the spec:features:services structure in the OpenStackDeployment resource as described in Remove an OpenStack service.

Note

The OpenStack Panko service has been removed from the product and is no longer maintained in the upstream OpenStack. See the project repository page for details.

Perform the upgrade¶

To start the OpenStack upgrade, change the value of the spec:openstack_version parameter in the OpenStackDeployment object to the target OpenStack release.

After you change the value of the spec:openstack_version parameter, the OpenStack Controller initializes the upgrade process.

To verify the upgrade status, use:

Logs from the osdpl container in the OpenStack Controller rockoon pod.

The OpenStackDeploymentStatus object.

When upgrade starts, the OPENSTACK VERSION field content changes to the target OpenStack version, and STATE displays APPLYING:

kubectl -n openstack get osdplst

Example of system output:

NAME      OPENSTACK VERSION   CONTROLLER VERSION   STATE
osh-dev   antelope            0.15.6               APPLYING

When upgrade finishes, the STATE field should display APPLIED:

kubectl -n openstack get osdplst

Example of system output:

NAME      OPENSTACK VERSION   CONTROLLER VERSION   STATE
osh-dev   antelope            0.15.6               APPLIED

The maintenance window for the OpenStack upgrade usually takes from two to four hours, depending on the cloud size.

Verify the upgrade¶

Verify that OpenStack is healthy and operational. All OpenStack components in the health group in the OpenStackDeploymentStatus CR should be in the Ready state. See OpenStackDeploymentStatus custom resource for details.
Verify the workability of your OpenStack deployment by running Tempest against the OpenStack cluster as described in Run Tempest tests.

Upgrade from Antelope to Caracal¶

Before upgrading, verify that you have completed the Prerequisites and removed the domains from federation mappings as described below.

Warning

If your MOSK cluster is running version 24.3 and includes the Instance High Availability service (OpenStack Masakari), the OpenStack upgrade will fail due to an incorrect migration of the Masakari database from legacy SQLAlchemy Migrate to Alembic caused by a misconfigured alembic_table. To avoid this issue, follow the workaround steps outlined in [47603] Masakari fails during the OpenStack upgrade to Caracal before proceeding with the upgrade.

Warning

If you initially deployed your MOSK cluster with OpenStack Victoria or earlier releases and gradually upgraded it to Antelope, and you do not perform periodic cleanups of OpenStack databases from soft-deleted rows, the upgrade will stuck due to the failing cinder-db-sync job.

To prevent, diagnose, and fix this issue, perform the workaround steps outlined in [47695] Cinder database sync job fails during upgrade from Antelope to Caracal before proceeding with the upgrade.

MOSK enables you to upgrade directly from Antelope to Caracal without the need to upgrade to the intermediate Bobcat release. To upgrade the cloud, complete the upgrade steps instruction changing the value of the spec:openstack_version parameter in the OpenStackDeployment object from antelope to caracal.

Remove domains from federation mappings¶

Important

Perform the domains removal from the federation mappings if your MOSK cluster configuration includes federated identity management system, such as IAM or any other supported identity provider.

Before Caracal, Keystone does not properly handle domain specifications for users in mappings. Even though domains are specified for users, Keystone always creates users in the domain associated with the identity provider the user logs in from.

Starting with Caracal, Keystone honors the domains specified for users in mappings. Many example mappings, including the previous default mapping in MOSK, use domain specifications. After upgrading to Caracal, the new users logging in through federation may be assigned to a different Keystone domain, while existing users will retain their current domain. This behaviour may negatively impact monitoring, compliance, and overall cluster operations.

To maintain the same functionality after the upgrade, remove the domain element from both the local.user element and local element, which sets default domain values for user and group elements, from the previous default mappings.

You can use the openstack mapping commands to manage mappings:

To list available mappings: openstack mapping list
To display the mapping rules: openstack mapping show <name>
To modify the mapping rules: openstack mapping set <name> --rules <rules>

Example mapping rules in Antelope:

[
  {
    "local": [
      {
        "user": {
          "name": "{0}",
          "email": "{1}",
          "domain": {
            "name": "Default"
          }
        }
      },
      {
        "groups": "{2}",
        "domain": {
          "name": "Default"
        }
      },
      {
        "domain": {
          "name": "Default"
        }
      }
    ],
    "remote": [
      {
        "type": "OIDC-iam_username"
      },
      {
        "type": "OIDC-email"
      },
      {
        "type": "OIDC-iam_roles"
      }
    ]
  }
]

Example mapping rules in Caracal:

[
  {
    "local": [
      {
        "user": {
          "name": "{0}",
          "email": "{1}"
        }
      },
      {
        "groups": "{2}",
        "domain": {
          "name": "Default"
        }
      }
    ],
    "remote": [
      {
        "type": "OIDC-iam_username"
      },
      {
        "type": "OIDC-email"
      },
      {
        "type": "OIDC-iam_roles"
      }
    ]
  }
]

Upgrade from Yoga to Antelope¶

MOSK enables you to upgrade directly from Yoga to Antelope without the need to upgrade to the intermediate Zed release.

Before upgrading, verify that you have completed the Prerequisites.

Important

There are several known issue affecting MOSK clusters running OpenStack Antelope that can disrupt the network connectivity of the cloud workloads.

If your cluster is still running OpenStack Yoga, update to the MOSK 24.2.1 patch release first and only then upgrade to OpenStack Antelope. If you have not been applying patch releases previously and would prefer to switch back to major releases-only mode, you will be able to do this when MOSK 24.3 is released.

If you have updated your cluster to OpenStack Antelope, apply the workarounds described in Release notes: OpenStack known issues for the following issues:

[45879] [Antelope] Incorrect packet handling between instance and its gateway
[44813] Traffic disruption observed on trunk ports

To upgrade the cloud, complete the upgrade steps instruction changing the value of the spec:openstack_version parameter in the OpenStackDeployment object from yoga to antelope.

Upgrade from Victoria to Yoga¶

Caution

If your cluster is running on top of the MOSK 23.1.2 patch version, the OpenStack upgrade to Yoga may fail due to the delay in the Cinder start. For the workaround, see 23.1.2 known issues: OpenStack upgrade failure.

Before upgrading, verify that you have completed the Prerequisites.

If your cloud runs on top of the OpenStack Victoria release, you must first upgrade to the technical OpenStack releases Wallaby and Xena before upgrading to Yoga.

Caution

The Wallaby and Xena releases are not recommended for a long-run production usage. These versions are transitional, so-called technical releases with limited testing scopes. For the OpenStack versions support cycle, refer to OpenStack support cycle.

To upgrade the cloud, complete the upgrade steps for each release version in line in the following strict order:

Upgrade the cloud from victoria to wallaby
Upgrade the cloud from wallaby to xena
Upgrade the cloud from xena to yoga