Upgrade OpenStack

This section provides instructions on how to upgrade OpenStack control plane on a MOSK cluster to a new major release version.

Note

The update of the OpenStack components within the same major OpenStack version is performed seamlessly as part of the MOSK cluster update.

Maintenance window

In terms of impact, the OpenStack upgrade resembles the OpenStack and OpenSDN components update phase of a cluster update. During this phase, the system pulls new container images for all OpenStack components and restarts all OpenStack services. Use the impact and time estimates of this phase as a baseline when planning the OpenStack upgrade.

Unlike an OpenStack component update, an OpenStack major release upgrade introduces the following operational behaviors:

  • The upgrade updates OpenStack services one by one in a strictly defined sequence to maintain cross-service compatibility throughout the process.

  • The upgrade runs offline schema updates and online data migrations for select services. Offline schema updates briefly disrupt service API availability, and their execution time scales with the target release and database size.

Typically, expect the maintenance window to take two to four hours, depending on the cloud size.

Prerequisites

  1. Verify that your OpenStack cloud is running on the latest MOSK release. See Compatibility Matrix for the release matrix and supported upgrade paths.

  2. Just before the upgrade, back up your OpenStack databases. See the following documentation for details:

  3. Verify that OpenStack is healthy and operational. All OpenStack components in the health group in the OpenStackDeploymentStatus CR should be in the Ready state. See OpenStackDeploymentStatus custom resource for details.

  4. Verify the workability of your OpenStack deployment by running Tempest against the OpenStack cluster as described in Run Tempest tests. Verification of the testing pass rate before upgrading will help you measure your cloud quality before and after upgrade.

  5. Read carefully through the Release Notes of your MOSK version paying attention to the Known issues section and the OpenStack upstream release notes for the target OpenStack version.

  6. Plan the maintenance window and notify users accordingly.

Migrate Tenant Telemetry from Gnocchi

If the metric service (Gnocchi) is enabled in your cloud, remove metric and add metric-storage in the spec:features:services field of the OpenStackDeployment object in the same change that switches spec:openstack_version from epoxy to gazpacho.

Data impact

The upgrade to Gazpacho removes Redis, the key-value store that Gnocchi used to hold metrics, regardless of whether the metric service is enabled. As a result, all metrics currently stored in Gnocchi are lost. In autoscaling mode, this amounts to the last hour of metrics.

User impact

Users of the OpenStack Alarming (Aodh) service must delete and re-create their alarms against the new metrics and queries.

Users of the OpenStack Heat autoscaling feature must additionally update their Heat templates and Heat stacks:

  • Gnocchi-specific alarm resources no longer work and must be replaced with the OS::Aodh::PrometheusAlarm resource. The list of the affected resources includes:

    • OS::Aodh::GnocchiAggregationByMetricsAlarm

    • OS::Aodh::GnocchiAggregationByResourcesAlarm

    • OS::Aodh::GnocchiResourcesAlarm

    • OS::Aodh::CompositeAlarm

  • Gnocchi-specific alarm options, such as metric, query, threshold, granularity, and aggregation method, must be replaced with an equivalent PromQL query parameter of the OS::Aodh::PrometheusAlarm stack resource that reproduces the intended triggering logic. Refer to Access Tenant Telemetry services on how to access the metric-storage service to find available metric names.

See also

Tenant Telemetry service

Perform the upgrade from Epoxy to Gazpacho

To start the OpenStack upgrade, change the value of the spec:openstack_version parameter in the OpenStackDeployment object from epoxy to gazpacho.

After you change the value of the spec:openstack_version parameter, the OpenStack Controller initializes the upgrade process.

To verify the upgrade status, use:

  • Logs from the osdpl container in the OpenStack Controller rockoon pod.

  • The OpenStackDeploymentStatus object.

    When upgrade starts, the OPENSTACK VERSION field content changes to the target OpenStack version, and STATE displays APPLYING:

    kubectl -n openstack get osdplst

    Example of system output:

    NAME             OPENSTACK VERSION   CONTROLLER VERSION   STATE      LCM PROGRESS   HEALTH   MOSK RELEASE
mosk-openstack   gazpacho            1.3.8                APPLYING   2/19           24/24    21.2.0+26.2

    When upgrade finishes, the STATE field should display APPLIED:

    kubectl -n openstack get osdplst

    Example of system output:

    NAME             OPENSTACK VERSION   CONTROLLER VERSION   STATE     LCM PROGRESS   HEALTH   MOSK RELEASE
mosk-openstack   gazpacho            1.3.8                APPLIED   19/19          24/24    21.2.0+26.2

Verify the upgrade

  1. Verify that OpenStack is healthy and operational. All OpenStack components in the health group in the OpenStackDeploymentStatus CR should be in the Ready state. See OpenStackDeploymentStatus custom resource for details.

  2. Verify the workability of your OpenStack deployment by running Tempest against the OpenStack cluster as described in Run Tempest tests.