Mirantis Cloud Platform Release Notes¶

Encryption of sensitive data in the Reclass model¶

^SECURITY

Implemented the GPG encryption to protect sensitive data in the Git repositories of the Reclass model as well as the key management mechanism for secrets encryption and decryption.

Galera verification and restoration pipeline¶

Implemented the automatic way to verify and restore the Galera cluster in the MCP deployment. In case of a cluster outage, the number of manual steps to start the cluster, as well as ensuring the necessary access can significantly delay the restoration of services and is prone to operator errors. Therefore, to reduce the complexity of the procedure and support greater scalability, the Verify and Restore Galera cluster pipeline has been created.

Jenkins version upgrade¶

Upgraded the Jenkins version in DriveTrain to the latest LTS v2.138.3.

Partitioning table for the VCP images¶

Implemented the dynamical strategy to prevent uploads from filling up the disk on the VCP nodes.

Rate limiting for the NGINX proxy service¶

^SECURITY

Implemented the possibility to limit the number of HTTP requests that a user can make in a given period of time for an OpenStack environment. The rate-limiting with NGINX can be used to protect an OpenStack environment against DDoS attacks as well as to protect the community application servers from being overwhelmed by too many user requests at the same time.

TCP-only support for Memcached¶

^SECURITY

Disabled the Memcached listener on the UDP port by default. To reduce the attack surface and improve the product security, Memcached on the controller nodes listens on TCP only. For the existing OpenStack environments deployed on top of the earlier MCP versions, implemented the possibility to manually disable the Memcached listener on the UDP port.

Encryption of the Keystone tokens stored within Memcached¶

^SECURITY

Implemented the protection of the Keystone tokens stored within Memcached.

MCP OpenStack supports the Memcached protection since the Pike release. By default, this functionality is disabled in the Pike deployments. For Queens, the Memcached protection is enabled by default with the ENCRYPT security strategy.

Octavia enhancements¶

Hardened the OpenStack Octavia LBaaS components and introduced the following enhancements:

• Added the OpenStack Queens support.
• Added the Transport Layer Security (TLS) support with Barbican.
• Changed location of the certificates used for connection to amphora. Now, they are created on the Salt Master node and then loaded on the gtw nodes.
• ^{TECHNICAL PREVIEW} Implemented clusterization for the Octavia Manager services.
• Added the Octavia artifacts to the MCP offline image.

Ironic deployment¶

^{DOCUMENTATION}, ^{TECHNICAL PREVIEW}

Added the list of the MCP Ironic supported features and known limitations. The new section in the MCP Reference Architecture Guide includes the Ironic drivers and features with known limitations that MCP DriveTrain supports. Since the Ironic service is available in MCP only as a Technical Preview feature, the driver or feature support status in that section stands for the ability of MCP DriveTrain to deploy and configure the features by means of the Ironic Salt formula through the cluster model.

Horizon load balancing¶

Enabled the load balancing mode for Horizon by default for the new MCP OpenStack deployments. The new approach allows for load reduction on one proxy node and spreading the load among all proxy nodes.

For the existing MCP OpenStack environments, implemented the flow to manually configure Horizon load balancing.

Partitioning table for the VCP images¶

Implemented the strategy to prevent uploads from filling up the disk on the Horizon proxy nodes.

Pike to Queens upgrade¶

^{TECHNICAL PREVIEW}

Implemented the upgrade of OpenStack Pike deployments to Queens.

The official MCP documentation includes the reference information to consider when creating a detailed maintenance plan for the upgrade. We recommend using the descriptive analysis of the techniques and tools, as well as the high-level upgrade flow included in the documentation to create a cloud-specific detailed upgrade procedure, assess the risks, estimate possible downtimes, plan the rollback, backup, and testing activities.

OpenStack packages update¶

^{TECHNICAL PREVIEW}

Implemented the flow to provide minor updates for the OpenStack packages without changing the major versions of the packages. In other words, the update between the package versions within a single major OpenStack release.

Kubernetes 1.12.4 support¶

^{Updated to 1.13.5 in 2019.2.3}

Added support for the community Kubernetes version 1.12.4. For the list of enhancements and bug fixes, see: Kubernetes release notes.

Docker replaced by containerd¶

Completed development and added full support for containerd runtime to execute containers and manage container images on a node instead of Docker in an MCP Calico-based Kubernetes cluster. As compared to Docker, containerd introduces lower memory footprint, faster container start, easier upgrades and updates.

The upgrade procedure of a Docker-based Kubernetes cluster to the containerd-based one comprises a use case when third-party workloads run under Docker along with the MCP Kubernetes-based ones. Therefore, Docker is not stopped and removed during the upgrade to prevent these third-party workloads from being corrupted. However, you can disable Docker after the upgrade if required.

Migration of kube-addon-manager to a Kubernetes pod¶

Migrated the kube-addon-manager service to a separate pod controlled by Kubernetes to fit the community implementation. Previously, kube-addon-manager was running as a systemd service and was using the default system authorization that could not be handled by Kubernetes.

The main changes made during the kube-addon-manager migration are as follows:

• kube-addon-manager uses its own service account for authorization controlled by Kubernetes
• kube-addon-manager is created as a manifest before all other addons
• kube-addon-manager is handled by kubelet

Automatic Calico upgrade procedure¶

^{TECHNICAL PREVIEW}

Implemented the automatic upgrade procedure for Calico from version 2.6 to 3.3 by adjusting the existing Kubernetes upgrade pipeline job.

Additionally, you can use the same pipeline job to update Calico to a minor version, for example, from 3.1 to 3.3.

The Calico upgrade process implies the Kubernetes services downtime for workloads operations, for example, workloads spawning and removing. The downtime is caused by the necessity of the etcd schema migration where the Calico endpoints data and other Calico configuration data is stored.

Horizontal pod autoscaling¶

Introduced the capability to adjust the number of a Kubernetes pod replicas without using an external orchestrator by enabling the horizontal pod autoscaling feature. The feature is based on observed CPU and/or memory utilization and can be enabled using the metrics-server add-on. You can enable horizontal pod autoscaling either on a new or existing MCP Kubernetes cluster.

OpenStack cloud provider¶

^{TECHNICAL PREVIEW in 2019.2.0, GA in 2019.2.2}

Implemented the capability to use the OpenStack cloud provider functionality on new Kubernetes clusters that are deployed on VMs on top of OpenStack.

The OpenStack cloud provider extends the basic functionality of Kubernetes by fulfilling the provider requirement for several resources. This is achieved through communication with several OpenStack APIs.

The two main functions provided by the OpenStack cloud provider are PersistentVolume for pods and LoadBalancer for services.

Virtlet 1.4.4 support¶

Updated Virtlet to version 1.4.4 that contains the following improvements:

• Added support for Kubernetes 1.12.x
• Added support for containerd
• Added support for cpusets
• Switched to the Mirantis hardened version of libvirt
• Improved the Virtlet examples
• Added injecting of ConfigMaps or Secrets into rootfs
• Improved the Virtlet user documentation and made it available from https://docs.virtlet.cloud to introduce a more user-friendly format
• Fixed a number of bugs to harden the product robustness

OpenContrail 4.1 support for OpenStack¶

Added support for the community OpenContrail version 4.1 integrated with the following OpenStack releases: Ocata, Pike, and Queens.

Upgrade path from OpenContrail 3.2 to 4.1¶

^{TECHNICAL PREVIEW}

Implemented the automatic upgrade procedure for OpenContrail from version 3.2 to 4.x that allows upgrading the OpenContrail nodes in an Ocata- or Pike-based MCP cluster to version 4.1 using the Deploy - upgrade Opencontrail to 4.x Jenkins pipeline job.

Update path from OpenContrail 4.0 to 4.1¶

^{TECHNICAL PREVIEW}

Implemented the automatic update procedure for OpenContrail 4.x that covers the update of OpenContrail nodes from version 4.0 to 4.1. The update is performed using the Deploy - update Opencontrail to 4.x Jenkins pipeline job.

StackLight components versions update¶

Updated the versions of the following StackLight LMA components:

• Prometheus from version 2.2.1 to 2.5.0
• Alerta from version 5.6.10 to 6.5.0
• Alertmanager from version 0.14.0 to 0.15.3 [1]
• Pushgateway from version 0.4.0 to 0.6.0
• Grafana from version 5.2.4 to 5.3.4
• Telegraf from version 1.5.3 to 1.9.1
• td-agent from version 3.1.1-0 to 3.2.1
• Fluentd from version 1.0.2 to 1.2.6
• Elasticsearch from version 5.6.12 to 6.5.2 [0]
• Kibana from version 5.6.12 to 6.5.2 [0]

Salesforce notifier service¶

Implemented the capability to configure Alertmanager to create Salesforce cases from Alertmanager notifications through the Salesforce notifier service. If you have already enabled Salesforce or email notifications through the Push Notification service, follow the procedure described in MCP Operations Guide: Switch to Alertmanager-based notifications.

Retention policy for logs and audit indices¶

Added the capability to manage the retention policy for logs and audit indices in an Elasticsearch cluster.

Ceph update to a minor version¶

^{TECHNICAL PREVIEW}

Implemented the capability to update Ceph packages to the latest minor versions on the Ceph OSD, Monitor, and RADOS Gateway nodes using the Update Ceph packages pipeline job.

Native Prometheus support¶

Improved Ceph monitoring by adding support for the Ceph Prometheus plugin that is based on the native Prometheus exporter introduced in Ceph Luminous. The Ceph Prometheus plugin collects a wider set of Ceph metrics as opposed to Telegraf and provides for better monitoring capabilities for large clusters. Updated Ceph-related Grafana dashboards to display new metrics.

For new deployments, the Ceph Prometheus plugin is enabled by default. For existing deployments, you can enable the Ceph Prometheus plugin manually or during the upgrade of StackLight LMA.

15644¶

The network driver may fail to allocate kernel memory. You may also detect the following symptoms of the issue:

• Traces in kern.log related to the BNX driver
• Ceph OSD flapping in the Ceph cluster during a rebalance

To prevent the issue, calculate the sysctl minimum reserved memory and set it using the vm.min_free_kbytes parameter for each type of node depending on your cluster model.

Workaround:

1. Open your Git project repository with the Reclass model on the cluster level.

2. In /etc/sysctl.conf specify the following pillar:

   linux:
     system:
       kernel:
         sysctl:
           vm.min_free_kbytes: <calculated_value>
   

3. Choose from the following options:

   • If you are making changes before the deployment, proceed with further configuration as required.

   • If you are making changes to an existing environment, apply the changes:

     1. Log in to the Salt Master node.

     2. Apply the following state:

        salt '*' state.apply linux.system.kernel
        

21033¶

The Salt Master CA does not provide the Certificate Revocation List (CRL) and index files to identify the revoked or expired certificates.

Workaround:

To list all currently issued certificates, follow the step 3 of the Replace the Salt Master CA certificates procedure.

24868¶

During the upgrade of an MCP cluster, after the installation of the salt-master, salt-common, salt-api, and salt-minion packages, the Deploy - update cloud pipeline may hang up with the Connection refused error message and trying to connect to salt-api.

Workaround:

1. Log in to the Salt Master node.

2. Restart the salt-api service:

   systemctl restart salt-api.service
   

3. Rerun the Deploy - update cloud pipeline.

25172¶

When changing any network settings (routes, up_cmds commands, MTU), the linux.network formula restarts the target interface and all related interfaces. For example, when changes are related to a bridge interface, all its interfaces will be restarted what leads to VMs failures. Therefore, Mirantis recommends configuring all required bridge interfaces on KVMs before a cluster deployment.

The workaround is to apply all required settings manually without a bridge restart. If a bridge restart on a KVM node is crucial:

1. Plan a maintenance window for your MCP cluster.
2. Stop all VMs of a node that requires a bridge restart.
3. Apply the required settings changes.
4. Restart the bridge interface.
5. Start all VMs.

26113¶

^{Fixed in 2019.2.3}

Occasionally, the deployment of OpenContrail v4.x with OpenStack Pike may fail due to the duplication of the salt-minion services.

Workaround:

1. Log in to the Salt Master node.

2. Apply the following state:

   salt -t 10 "rgw*" cmd.run 'pkill -9 salt-minion'
   

The service restarts automatically in a few minutes.

26330¶

The CVP - Sanity checks Jenkins pipeline may fail if the TEST_REPO parameter is not empty.

Workaround:

Leave the TEST_REPO parameter empty. This option is deprecated starting MCP Build ID 2019.2.0.

26417¶

When commissioning nodes with Intel X520-2 10 GB Ethernet Network Interface Cards (NICs), such cards may not be discovered.

Workaround:

Do not use Intel X520-2 10GB NICs with firmware version 0x30030001.

27010¶

When upgrading from the MCP Build ID 2018.11.0 to 2019.2.0, the Deploy - upgrade MCP DriveTrain Jenkins pipeline job fails due to the mirror jobs failing to trigger the newest version.

Workaround:

1. Log in to the Jenkins web UI.
2. Run the git-mirror-downstream-mk-pipelines and git-mirror-downstream-pipeline-library Jenkins pipeline jobs with BRANCHES set to release/2019.2.0.
3. Rerun the Deploy - upgrade MCP DriveTrain Jenkins pipeline job with UPDATE_PIPELINES set to false.

27135¶

^{Fixed in 2019.2.3}

Creating instant backups using Backupninja, Xtrabackup, ZooKeeper, or Cassandra may fail due to an issue with permissions.

Workaround:

1. Log in to the Salt Master node.

2. Obtain the SSH RSA key specified in /root/.ssh/id_rsa.pub.

3. On the system level of the Reclass model, add the obtained SSH RSA key to system/<service_name>/server.yml for Backupninja or Xtrabackup or to system/<service_name>/backup/server.yml for Cassandra or ZooKeeper. For example, for Backupninja add the following pillar to system/backupninja/server.yml.

   parameters:
     backupninja:
       server:
         key:
           backupninja_pub_key:
             enabled: true
             key: <key_from_/root/id_rsa.pub>
   

4. Apply the corresponding service state. For example, for Backupninja apply the following state on the nodes with the Backupninja pillar defined:

   salt -C 'I@backupninja:client or I@backupninja:server' state.sls backupninja
   

27638¶

When performing operations through Jenkins that require the Salt Minion package update and restart, for example, MCP DriveTrain upgrade, a cloud environment update, packages update, and so on, Jenkins pipeline jobs may fail due to the known community dbus-daemon issue.

Workaround:

1. On the Salt Master node, run:

   systemctl daemon-reexec
   systemctl restart salt-minion
   

2. Log in to the Jenkins web UI.

3. Re-run the failed Jenkins pipeline job.

32633¶

Occasionally, application of the Salt states across all nodes during the deployment pipelines execution fails with Pepper error: Server error. The issue affects large deployments with a big number of Salt Minions and may affect the services deployment during the later deployment steps.

To workaround the issue, select from the following options:

• Enable the Salt batching for the affected Salt states. For example, if the linux.system state fails, apply the following patch to the pipeline-library repository:

  diff --git a/src/com/mirantis/mk/Orchestrate.groovy b/src/com/mirantis/mk/Orchestrate.groovy
  index 509fe87..575d6ca 100644
  --- a/src/com/mirantis/mk/Orchestrate.groovy
  +++ b/src/com/mirantis/mk/Orchestrate.groovy
  @@ -44,7 +44,7 @@ def installFoundationInfra(master, staticMgmtNet=false, extra_tgt = '') {
       } catch (Throwable e) {
           common.warningMsg('Salt state salt.minion.base is not present in the Salt-formula yet.')
       }
  -    salt.enforceState([saltId: master, target: "* ${extra_tgt}", state: ['linux.system'], retries: 2])
  +    salt.enforceState([saltId: master, target: "* ${extra_tgt}", state: ['linux.system'], batch: '15', retries: 2])
       if (staticMgmtNet) {
           salt.runSaltProcessStep(master, "* ${extra_tgt}", 'cmd.shell', ["salt-call state.sls linux.network; salt-call service.restart salt-minion"], null, true, 60)
       }
  

  The patch sets the batch size to 15% of the target nodes that include the "* ${extra_tgt}" nodes. In the absence of additional conditions, the state will be applied to the 15% of the total number of these nodes.

• Manually re-run the failed state. For example, if the salt.minion state fails, perform the following steps:

  1. Log in to the Salt Master node.

  2. Re-apply the failed state on the affected nodes manually:

     salt '*' state.sls salt.minion
     

  3. Restart the salt-minion service manually:

     salt '*' cmd.run 'salt-call service.restart salt-minion'
     salt '*' saltutil.clear_cache
     salt '*' saltutil.refresh_pillar
     salt '*' saltutil.sync_all
     

     During the restart of the salt-minion service, verify that the Salt Master node does not catch the exception with getting the lost minion.

  4. Restart the failed pipeline to proceed with update, deployment, or another required operation.

32079¶

The values of the net.ipv4.neigh.default.gc_thresh1, net.ipv4.neigh.default.gc_thresh2, and net.ipv4.neigh.default.gc_thresh3 kernel parameters in pillars may differ from the ones in the output of the sysctl command on the mon* and ctl* nodes because of the specific values hardcoded in Docker.

Workaround:

1. Open your Git project repository with the Reclass model on the cluster level.

2. In classes/cluster/<cluster_name>/cicd/control/init.yml and classes/cluster/<cluster_name>/infra/config/docker.yml, add the following pillar:

   linux:
     system:
       kernel:
         # hardcoded in overlay network driver https://github.com/docker/libnetwork/pull/1789/files
         sysctl:
           net.ipv4.neigh.default.gc_thresh1: 8192
           net.ipv4.neigh.default.gc_thresh2: 49152
           net.ipv4.neigh.default.gc_thresh3: 65536
   

3. If you have StackLight enabled, also add the same pillar to classes/cluster/<cluster_name>/stacklight/server.yml.

4. Apply the changes:

   salt '*' saltutil.sync_all
   

28046¶

When the Open vSwitch (OVS) network interfaces have the same MAC address, for example, when a bond interface is split into several vLANs with tags, OVS prior to version 2.10 may not add flow rules to some OVS bridges.

Workaround:

Choose from the following options:

• Add a unique MAC address to the ports description. For example:

  bond1.${_param:aint_public_vlan}:
  name: bond1.${_param:aint_public_vlan}
  enabled: true
  proto: manual
  type: ovs_port
  bridge: br-aint_public
  ovs_bridge: br-aint_public
  hwaddress: <unique_mac>
  ovs_port_type: OVSPort
  use_interfaces:
  
  bond1
  

• Use the following configuration order:

  1. Plug the tagged interfaces into the Linux bridges.
  2. Connect the Linux bridges into the OVS bridges.

• Use external networks:

  1. Pass the entire interface to the OVS bridge and map it to a single physical network.
  2. Split the interface on vLANs by setting provider:segmentation_id for each Neutron network.

34308¶

The Deploy - upgrade control VMs Jenkins pipeline job may fail with the HTTP Error 504: Gateway Time-out error message. The workaround is to increase the timeout for NGINX.

Workaround:

1. Open your Git project repository with the Reclass model on the cluster level.

2. In classes/cluster/<cluster_name>/infra/config/init.yml, increase the timeout for NGINX:

   nginx:
     server:
       site:
         nginx_proxy_salt_api:
           proxy:
             timeout: 1000
   

3. Apply the following state:

   salt -C 'I@salt:master' saltutil.sync_all
   salt -C 'I@salt:master' state.sls nginx.server
   

35060¶

The service.enable Salt module does not enable the nova-novncproxy service if it was disabled using systemd.

Workaround:

• Enable the service using systemd
• Disable the service using Salt and then enable it also using Salt

34708¶

Due to the specifics of SaltStack version 2017.7, when using the x509.py module, Salt ignores the test=True option and applies the changes.

22489¶

^{Pike, Queens}

In the OpenStack environments with OpenContrail and Barbican, if you use a non-default Keystone domain, the LBaaS VIP cannot be created. LBaaS cannot download a secret created by the Barbican user in any project other than the project where opencontrail_barbican_user has admin privileges.

Workaround:

1. On every OpenStack controller node where Barbican API is installed, add the following configuration to /etc/barbican/policy.json:

   barbican:
     server:
       policy:
         all_domains_reader: 'user:<user_ID> and project:<project_ID>'
         secret_acl_read: "'read':%(target.secret.read)s or rule:all_domains_reader"
         container_acl_read: "'read':%(target.container.read)s or rule:all_domains_reader"
   

   By default, LBaaS uses the admin user to obtain secrets from Barbican. Replace <user_ID> and <project_ID> with a corresponding OpenStack ID of this user and the project where this user has an admin role.

2. Log in to the Salt Master node.

3. Apply the following state:

   salt -C 'I@barbican:server' state.apply barbican
   

This configuration adds appropriate rights to read the secrets and containers from Barbican.

25742¶

^Queens

The Reclass model for OpenStack Queens includes the deprecated Heat CloudWatch API, which may cause false positive alerts for the Heat CloudWatch service in StackLight LMA. The issue affects only the existing deployments with OpenStack Queens.

Workaround:

1. Upgrade your MCP deployment to the Build ID 2019.2.0 as described in MCP Operations Guide: Upgrade MCP to a newer release version.

2. Open your Git project repository with the Reclass model on the cluster level.

3. In openstack/init.yml, specify the following class:

   openstack_heat_cloudwatch_api_enabled: False
   

4. Log in to the Salt Master node.

5. Apply the haproxy state on all OpenStack controller nodes:

   salt ctl* state.apply haproxy
   

6. Apply the nginx state on all proxy nodes:

   salt prx* state.apply nginx
   

26149¶

^Queens

When resetting the OpenStack administrator password, the state.sls keystone state does not apply the changes. The issue affects only the OpenStack Queens release.

Workaround:

1. Log in to an OpenStack controller node.

2. Source the keystonercv3 file:

   source /root/keystonercv3
   

3. Set a new password:

   openstack user set admin --password <new_password>
   

   Once done, the services that use the administrator password will fail to authenticate.

4. From the Salt Master node, open the /srv/salt/reclass/classes/<cluster_name>/infra/secrets.yaml file and specify the new password using the keystone_admin_password parameter.

5. Re-run the Deploy - OpenStack Jenkins pipeline job.

26269¶

^{Queens. Fixed in 2019.2.3}

Changing the logging level for the OpenStack services may fail.

Workaround:

1. Apply the Salt formula patch to your Oslo templates Salt formula.
2. Apply the OpenStack states depending on your deployment. For example, if on Nova compute you have Nova, Neutron, and Cinder, apply salt cmp* state.apply nova,neutron,cinder. Alternatively, re-run the Deploy - OpenStack Jenkins pipeline job.

27071¶

^{Pike, Queens}

On the OpenStack Pike or Queens environments with Octavia, if during creation, updating, or deleting of a load balancer or other resources a gtw node is rebooted or the octavia-worker service is restarted, the stale load balancer stucks in the PENDING_UPDATE or PENDING_DELETE state.

Workaround:

1. Log in to any OpenStack controller node.

2. Obtain the target load balancer ID:

   openstack loadbalancer list | awk '/ PENDING_CREATE / {print $2}
   

3. Choose from the following options:

   • For the MCP version 2019.2.4 and later, run the following command:

     openstack loadbalancer delete --force <load_balancer_id>
     

     Note

     The --force flag requires admin rights and works only if a load balancer was not updated during the last hour.

   • For the MCP versions older than 2019.2.4:

     1. Log in to any dbs node.

     2. Log in to the MySQL database:

        mysql -uoctavia -p
        

     3. Run the following command with the load balancer ID obtained in the step 2. For example:

        update load_balancer set provisioning_status='ERROR' \
        where id='0fc571fe-6ad1-4311-ab13-765b5526cd30';
        

27403¶

^{Pike, Queens}

On the OpenStack Pike or Queens environments with Octavia, if a gtw node hosting the Octavia services has issues with tenant network causing the Octavia management network lb-mgmt-net to become unreachable from this gtw node, the Octavia controller services stop working properly without connection to the amphora instances.

Workaround:

• If you run the Octavia services on all gtw nodes using octavia_manager_cluster and only one gtw node has tenant network issues, manually stop the Octavia controller services (octavia-health-manager, octavia-housekeeping, octavia-worker) on the affected node until the network issue on this node is resolved. In this case, the Octavia controller services will continue working properly.

• If you run the Octavia services only on the gtw01 node, manually stop the Octavia controller services and choose from the following options:

  • Start the Octavia controller services on another gtw0x node:

    1. Open your Git project repository with the Reclass model on the cluster level.

    2. In cluster/<cluster_name>/infra/config/nodes.yml, change the node for the Octavia services, for example, to gtw02:

       parameters:
         reclass:
           storage:
             node:
               openstack_gateway_node02:
                 classes:
                 - cluster.${_param:cluster_name}.openstack.octavia_manager
                 params:
                   octavia_hm_bind_ip: ${_param:octavia_health_manager_node01_address}
       

    3. Log in to the Salt Master node.

    4. Apply the following states:

       salt-call state.sls reclass.storage
       salt '*' saltutil.refresh_pillar
       salt -C 'I@neutron:client' state.sls neutron.client
       salt '*' mine.update
       

    5. For the gtw node where you moved the Octavia services, apply the Octavia states. For example:

       salt 'gtw02*' state.sls octavia
       

  • ^{TECHNICAL PREVIEW} Enable octavia_manager_cluster:

    1. Open your Git project repository with the Reclass model on the cluster level.

    2. In infra/<cluster_name>/infra/config/init.yml, change the following class

       - system.reclass.storage.system.openstack_gateway_single_octavia
       

       to

       - system.reclass.storage.system.openstack_gateway_cluster_octavia
       

    3. Log in to the Salt Master node.

    4. Apply the following states:

       salt-call state.sls reclass.storage
       salt '*' saltutil.refresh_pillar
       salt -C 'I@neutron:client' state.sls neutron.client
       salt '*' mine.update
       salt -C "I@octavia:manager and not *01*" state.sls octavia
       

33365¶

^{Pike, Queens}

The Nova scheduler counts the disk space of the volume-backed instances and causes NoValidHostFound errors from Nova when booting an instance. The reason is that Nova considers the size of the root volume specified in the instance flavor to be consumed by that instance on the compute host even if the instance is booted from the Cinder volume and does not consume any disk resources on the compute host.

Workarounds:

• If your cloud uses instances booted only or mostly from Cinder volumes, increase the disk overcommit ratio:

  1. Open your Git project repository with the Reclass model on the cluster level.

  2. In cluster/<cluster_name>/openstack/control.yml, increase the disk allocation ratio as required using the disk_allocation_ratio parameter:

     nova:
       controller:
         disk_allocation_ratio: <integer>
     

  3. From the Salt Master node, apply the nova state:

     salt 'ctl*' state.apply nova
     

• If only some instances boot from Cinder volumes, create a separate flavor of zero size for the root volume to be used by such instances. Use these flavors when creating instances booted from Cinder volumes.

  1. Open your Git project repository with the Reclass model on the cluster level.

  2. In cluster/<cluster_name>/openstack/control.yml, define a new flavor and set disk to 0. Set other parameters as required. For example:

     nova:
        client:
          enabled: true
          server:
            identity:
             flavor:
               flavor1:
                 flavor_id: 10
                 ram: 4096
                 disk: 0
                 vcpus: 1
     

  3. From the Salt Master node, apply the novaclient state:

     salt 'ctl*' state.apply novaclient
     

33576¶

^{Pike, Queens}

A Neutron port on a private network may receive traffic from other networks or VLANs during wiring. The workaround is to use iptables instead of the Open vSwitch security groups.

Workaround:

1. Open your Git project repository with the Reclass model on the cluster level.

2. In cluster/<cluster_name>/openstack/compute.yaml, set the firewall_driver to iptables_hybrid:

   neutron:
     compute:
       firewall_driver: iptables_hybrid
   

3. Apply the neutron state from the Salt Master node:

   salt -C 'I@neutron:server' state.sls neutron
   

34028¶

^{Pike, Queens}

The Keepalived service may fail during the upgrade from MCP versions lower than 2018.11.0 to 2019.2.0. The workaround is to disable Keepalived monitoring and enable it once you complete the upgrade.

Workaround:

1. Open your Git project repository with the Reclass model on the cluster level.

2. In classes/cluster/<cluster_name>/infra/init.yml, disable Keepalived monitoring:

   keepalived:
     _support:
       telegraf:
         enabled: false
   

3. Verify that all nodes have the Telegraf Keepalived plugin disabled:

   salt -C "*" saltutil.refresh_pillar
   

4. Verify that no nodes respond against test.ping:

   salt -C "I@keepalived:_support:telegraf:enabled:True" test.ping
   

5. Apply the change:

   salt -C 'I@telegraf:agent' state.sls telegraf
   

6. Once you complete the upgrade, revert the step 2.

7. Apply the change:

   salt -C 'I@telegraf:agent' state.sls telegraf
   

34455¶

^{Pike, Queens}

After deployment of an OpenStack environment, the VCP nodes may have an incorrect DNS server if MAAS is used. The reason is that during a VCP node boot, it obtains a DNS server from MAAS, which may differ from the DNS server specified in the deployment model.

To resolve the issue for the existing VCP nodes, remove the wrong DNS server address from the /etc/resolv.conf configuration file. To resolve the issue before deploying a new environment or adding new VCP nodes to an existing environment, specify the network data in cloud-init.

To apply a workaround for existing VCP nodes:

1. Log in to the Salt Master node.

2. Obtain the MAAS DNS server address:

   salt-call pillar.get maas:region:bind:host
   

3. Remove the MAAS DNS server address from the affected nodes:

   salt -C '<target_compound>' cmd.run 'sed -i /<maas_server>/d /etc/resolv.conf
   

To apply a permanent solution for every new VCP node:

1. Log in to the Salt Master node.

2. Obtain the list of VCP nodes defined in the model:

   salt '<any_kvm_node>' --out json pillar.get salt:control:cluster:internal:node | jq -r '.[] | keys[]'
   

   Example of system response:

   bmk01
   cid01
   cid02
   cid03
   ...
   prx01
   prx02
   

3. Determine the VCP nodes pillars that contain the cloud-init data:

   salt '<any_kvm_node>' --out yaml pillar.items | grep 'salt_control_cluster_node_cloud_init_'
   

   Example of system response:

   salt_control_cluster_node_cloud_init_openstack_control:
   salt_control_cluster_node_cloud_init_openstack_dns:
   salt_control_cluster_node_cloud_init_openstack_proxy:
   salt_control_cluster_node_cloud_init_infra_storage:
   salt_control_cluster_node_cloud_init_cicd_control:
   salt_control_cluster_node_cloud_init_stacklight_telemetry:
   

4. Open your Git project repository with the Reclass model on the cluster level.

5. In classes/cluster/<cluster_name>/infra/kvm.yml, specify the network data for the required nodes. For example, for the cid and prx nodes:

   parameters:
     _param:
       salt_control_vcp_deploy_interface: 'ens2'
       salt_control_vcp_deploy_interface_netmask: ${_param:deploy_network_netmask}
       salt_control_vcp_deploy_interface_gateway: ${_param:deploy_network_gateway}
       salt_control_vcp_dns_server_1: ${_param:dns_server01}
       salt_control_vcp_dns_server_2: ${_param:dns_server02}
       salt_control_cluster_node_cloud_init_network_data:
         network_data:
           links:
             - type: 'phy'
               id: ${_param:salt_control_vcp_deploy_interface}
               name: ${_param:salt_control_vcp_deploy_interface}
           services:
             - type: "dns"
               address: ${_param:salt_control_vcp_dns_server_1}
             - type: "dns"
               address: ${_param:salt_control_vcp_dns_server_2}
       salt_control_common_network_data_networks_deploy_interface_no_dhcp_common: &common_no_dhcp_data
         link: ${_param:salt_control_vcp_deploy_interface}
         type: 'ipv4'
         id: 'private-ipv4'
         netmask: ${_param:salt_control_vcp_deploy_interface_netmask}
         routes:
           - gateway: ${_param:salt_control_vcp_deploy_interface_gateway}
             network: '0.0.0.0'
             netmask: '0.0.0.0'
       salt_control_cluster_node_cloud_init_cicd_control:
         network_data: ${_param:salt_control_cluster_node_cloud_init_network_data}
       salt_control_cluster_node_cloud_init_openstack_proxy:
         network_data: ${_param:salt_control_cluster_node_cloud_init_network_data}
     salt:
       control:
         cluster:
           internal:
             node:
               cid01:
                 cloud_init:
                   network_data:
                     networks:
                       - <<: *common_no_dhcp_data
                         ip_address: ${_param:cicd_control_node01_deploy_address}
               cid02:
                 cloud_init:
                   network_data:
                     networks:
                       - <<: *common_no_dhcp_data
                         ip_address: ${_param:cicd_control_node02_deploy_address}
               cid03:
                 cloud_init:
                   network_data:
                     networks:
                       - <<: *common_no_dhcp_data
                         ip_address: ${_param:cicd_control_node03_deploy_address}
               prx01:
                 cloud_init:
                   network_data:
                     networks:
                       - <<: *common_no_dhcp_data
                         ip_address: ${_param:openstack_proxy_node01_deploy_address}
               prx02:
                 cloud_init:
                   network_data:
                     networks:
                       - <<: *common_no_dhcp_data
                         ip_address: ${_param:openstack_proxy_node02_deploy_address}
   

6. Synchronize the Salt resources:

   salt -C 'I@salt:control' saltutil.sync_all
   

7. Proceed with OpenStack environment deployment:

   • Automatically, as described in MCP Deployment Guide: Deploy an OpenStack environment.

   • Manually, by booting the VCP nodes:

     salt -C 'I@salt:control' salt.control
     

25969¶

The OpenStack cloud provider redefines the internal IP of the Kubernetes nodes with an IP of every NIC and can assign a wrong IP address as a primary address of a node. This can lead to failures in the output of the kubectl exec and kubectl logs commands.

Workaround:

1. Log in to any Kubernetes node.

2. Choose from the following options:

   • If the Kubernetes VMs have two network interfaces:

     1. In /etc/kubernetes/cloud-config, set the public-network-name cfg option for OpenStack cloud provider to the name of the OpenStack environment public network:

        [Networking]
        public-network-name=public
        

     2. Apply the changes:

        • On the Kubernetes Master node, run:

          systemctl restart openstack-cloud-controller-manager
          

        • On the Kubernetes Node, run:

          systemctl restart kubelet
          

     3. Repeat the steps 1-2 on the remaining Kubernetes Master nodes and Kubernetes nodes.

   • If the Kubernetes VMs have more than two network interfaces:

     1. In /etc/default/kubelet, set the kubelet --address parameter to 0.0.0.0 for kubelet to listen to all interfaces.

        Warning

        This setting may have a security impact on a Kubernetes cloud.

     2. Apply the changes:

        systemctl restart kubelet
        

     3. Repeat the steps 1-2 on the remaining Kubernetes Master nodes and Kubernetes nodes.

23177¶

Dynamic Kernel Module Support fails to build DPDK kernel modules for OpenContrail v3.2.3 on kernels newer than v4.8. The workaround is to use DPDK libraries v17.02 instead of v2.1.

24943¶

If the OpenContrail cluster has ports with the allowed address pair (AAP) prefix length less than /24 for IPv4 and /120 for IPv6, such AAPs may not work after the upgrade of OpenContrail v3.2 to v4.0. The workaround is to modify all AAPs on all virtual interfaces through the OpenContrail web UI. For example, change 1.2.3.4/16 to 1.2.3.4/24.

25264¶

^{Fixed in 2019.2.3} In the OpenContrail 4.x deployments, after restoring the ZooKeeper database, contrail-control may be inactive on all ntw nodes due to an issue with permissions for certificates.

Workaround:

1. Log in to the Salt Master node.

2. Change permissions:

   salt -C 'I@opencontrail:control' cmd.run 'chown -R contrail:contrail /etc/contrail'
   

3. Verify that the files are owned by the OpenContrail user:

   salt -C 'I@opencontrail:control' cmd.run 'ls -la /etc/contrail'
   

25629¶

^{Fixed in 2019.2.3} In the OpenContrail 4.x deployments, some web UI tabs fail to open. For example, opening of Setting -> Config Editor raises [SyntaxError: Failed to parse JSON body: Unexpected end of input] in logs. Opening of Monitor -> Infrastructure -> Virtual Router restarts the web UI with The worker has disconnected error in logs.

Workaround:

1. Log in to any ntw or nal node.
2. In /etc/haproxy/haproxy.cfg, remove the option nolinger parameter from the contrail-api and contrail-analytics sections of the file.
3. Repeat the step 2 on the remaining ntw and nal nodes.

25857¶

The OpenContrail web UI may display the Instance data is available in config but not available in analytics false error message for some properly operating SNAT instances in Monitoring > Virtual Routers > Instances. Do not remove such instances.

26133¶

Tempest tests may cause contrail-api fail to start. The workaround depends on the workloads put on the cloud after performing the tempest test, contact Mirantis support to resolve the issue.

26673¶

^{Fixed in 2019.2.8} Updating the name of a shared network in the Horizon web UI fails with the Failed to update network <network_name> error message. As a workaround, update the network through CLI or the OpenContrail web UI.

29253¶

^{Fixed in 2019.2.4}

The Kafka service may fail to start on the MCP deployments with OpenContrail 4.1.

The Kafka service has the timeout option for connection to the ZooKeeper cluster. Sometimes, the specified timeout value is less than the time needed for ZooKeeper to perform election and start the service requests. The Kafka service stops working if connection to the ZooKeeper cluster is not established during the specified amount of time (timeout).

Workaround:

1. Log in to the Salt Master node.

2. Start the failed Kafka service on the affected node(s):

   salt -C "<affected_node_name>" cmd.run "doctrail analyticsdb service confluent-kafka start"
   

29091¶

Opening or refreshing the OpenContrail 4.1 web UI in the Google Chrome browser causes the SSH handshake failure.

Workaround:

Select from the following options:

• Use a different browser, for example, Firefox or Safari
• Access the OpenContrail web UI through the prx nodes

34807¶

The OpenContrail 4.1 vRouter may crash when applying the contrail-vrouter-agent configuration. No workaround is required, the vRouter automatically restarts after the crash and correctly applies the new configuration.

35484¶

After upgrading an MCP cluster, some VMs on the OpenStack compute nodes can be unreachable through a floating IP due to an incorrect checksum. In such case, the InterfaceConfiguration: Virtual-network UUID mismatch for interface error entry appears in the vrouter-agent log for the interface of the affected VM.

The workaround is to manually disable tx-checksumming for the bond interface. For details, see Juniper documentation: Troubleshooting: SSH/TCP traffic fails in Contrail due to checksum errors.

Workaround:

1. Log in to the OpenStack compute node that runs the affected VM.

2. Disable tx-checksumming for the bond interface:

   ethtool -K bond0 tx off
   

28119¶

^{Fixed in 2019.2.4}

CADF notifications are unavailable with Elasticsearch and Kibana v6. The workaround is to deploy Elasticsearch and Kibana v5 instead during the deployment of a new MCP cluster.

Workaround:

1. Open your Git project repository with Reclass model on the cluster level.

2. In classes/cluster/<cluster_name>/stacklight/log.yml, specify the elasticsearch_version: 5 and kibana_version: 5 parameters:

   parameters:
     _param:
       ...
       elasticsearch_version: 5
       kibana_version: 5
   

3. Log in to the Salt Master node.

4. Apply the following states:

   salt '*' saltutil.refresh_pillar
   salt '*' state.sls linux.system.repo
   

5. Proceed with further configuration as required.

19913¶

Restarting the RADOS Gateway service using systemctl may fail. The workaround is to restart the service manually.

Workaround:

1. Log in to an rgw node.

2. Obtain the process ID of the RADOS Gateway service:

   ps uax | grep radosgw
   

   Example of system response:

   root     17526  0.0  0.0  13232   976 pts/0    S+   10:30   \
   0:00 grep --color=auto radosgw
   ceph     20728  0.1  1.4 1306844 58204 ?       Ssl  Jan28   \
   2:51 /usr/bin/radosgw -f --cluster ceph --name client.rgw.rgw01 --setuser ceph --setgroup ceph
   

   Where the process ID is 20728.

3. Stop the process using the obtained process ID. For example:

   kill -9 $20728
   

4. Start the RADOS Gateway service specifying the node name, for example, client.rgw.rgw01:

   /usr/bin/radosgw --cluster ceph --name client.rgw.rgw01 --setuser ceph --setgroup ceph
   

5. Perform the steps 1 - 4 from the remaining rgw nodes one by one.

23318¶

^{Fixed in 2019.2.3}

The upgrade of a Ceph cluster from Jewel to Luminous using the Ceph - upgrade Jenkins pipeline job does not include an automatic check if other components were upgraded before upgrading the rgw nodes. As a result, uploading a file to object storage may fail. The workaround is to upgrade the rgw nodes only after you have successfully upgraded the mon, mgr, and osd nodes.

24197¶

The tempest.api.object_storage.test_account_quotas.AccountQuotasTest.test_admin_modify_quota Tempest test fails because modifying the account quota is not possible even if the OpenStack user has the ResellerAdmin role. Setting a quota using the Swift CLI and API served by RADOS Gateway is also not possible. As a workaround set the quotas using the radosgw-admin utility (requires an SSH access to an OpenStack environment) as described in Quota management or using the RADOS Gateway Admin Operations API as described in Quotas.

24205¶

Creating Swift containers with custom headers using the Heat stack or the tempest.api.orchestration.stacks.test_swift_resources.SwiftResourcesTestJSON.test_acl Tempest test fails. As a workaround, first create a container without additional parameters and then set the metadata variables as required.

29811¶

^{Fixed in 2019.2.4}

The mon_max_pg_per_osd variable is set in a wrong section and does not apply on the Ceph OSDs. The workaround is to manually apply the necessary changes to the cluster model.

Workaround:

1. In classes/cluster/<cluster_name>/ceph/common.yml, define the additional parameters in the ceph:common pillar as follows:

   parameters:
     ceph:
       common:
         config:
           global:
             mon_max_pg_per_osd: 600
   

2. In /classes/service/ceph/mon/cluster.yml and /classes/service/ceph/mon/single.yml, remove the configuration for mon_max_pg_per_osd:

   common:
    #  config:
    #    mon:
    #      mon_max_pg_per_osd: 600
   

3. Apply the ceph.common state on the Ceph nodes:

   salt -C "I@ceph:common" state.sls ceph.common
   

4. Set the noout and norebalance flags:

   ceph osd set noout
   ceph osd set norebalance
   

5. Restart the Ceph Monitor services on the cmn nodes one by one. Verify that the nodes are in the HEALTH_OK status after each Ceph Monitor restart.

   salt -C <HOST_NAME> cmd.run 'systemctl restart ceph-mon.target'
   salt -C <HOST_NAME> cmd.run 'systemctl restart ceph-mgr.target'
   salt -C <HOST_NAME> cmd.run 'ceph -s'
   

6. Restart the Ceph OSD services on the osd nodes one by one:

   1. On each Ceph OSD node verify the OSDs running:

      ceph001# ceph osd status 2>&1 | grep $(hostname)
      

   2. For each Ceph OSD number:

      ceph001# service ceph-osd@OSD_NR_FROM_LIST status
      ceph001# service ceph-osd@OSD_NR_FROM_LIST restart
      ceph001# service ceph-osd@OSD_NR_FROM_LIST status
      

   3. Verify that the cluster is in the HEALTH_OK status before restarting the next Ceph OSD.

7. When the last Ceph OSD restarts, unset the noout and norebalance flags:

   ceph osd unset noout
   ceph osd unset norebalance