OpenStack

OpenStack


22489

Pike, Queens

In the OpenStack environments with OpenContrail and Barbican, if you use a non-default Keystone domain, the LBaaS VIP cannot be created. LBaaS cannot download a secret created by the Barbican user in any project other than the project where opencontrail_barbican_user has admin privileges.

Workaround:

  1. On every OpenStack controller node where Barbican API is installed, add the following configuration to /etc/barbican/policy.json:

    barbican:
      server:
        policy:
          all_domains_reader: 'user:<user_ID> and project:<project_ID>'
          secret_acl_read: "'read':%(target.secret.read)s or rule:all_domains_reader"
          container_acl_read: "'read':%(target.container.read)s or rule:all_domains_reader"
    

    By default, LBaaS uses the admin user to obtain secrets from Barbican. Replace <user_ID> and <project_ID> with a corresponding OpenStack ID of this user and the project where this user has an admin role.

  2. Log in to the Salt Master node.

  3. Apply the following state:

    salt -C 'I@barbican:server' state.apply barbican
    

This configuration adds appropriate rights to read the secrets and containers from Barbican.


25742

Queens

The Reclass model for OpenStack Queens includes the deprecated Heat CloudWatch API, which may cause false positive alerts for the Heat CloudWatch service in StackLight LMA. The issue affects only the existing deployments with OpenStack Queens.

Workaround:

  1. Upgrade your MCP deployment to the Build ID 2019.2.0 as described in MCP Operations Guide: Upgrade MCP to a newer release version.

  2. Open your Git project repository with the Reclass model on the cluster level.

  3. In openstack/init.yml, specify the following class:

    openstack_heat_cloudwatch_api_enabled: False
    
  4. Log in to the Salt Master node.

  5. Apply the haproxy state on all OpenStack controller nodes:

    salt ctl* state.apply haproxy
    
  6. Apply the nginx state on all proxy nodes:

    salt prx* state.apply nginx
    

26149

Queens

When resetting the OpenStack administrator password, the state.sls keystone state does not apply the changes. The issue affects only the OpenStack Queens release.

Workaround:

  1. Log in to an OpenStack controller node.

  2. Source the keystonercv3 file:

    source /root/keystonercv3
    
  3. Set a new password:

    openstack user set admin --password <new_password>
    

    Once done, the services that use the administrator password will fail to authenticate.

  4. From the Salt Master node, open the /srv/salt/reclass/classes/<cluster_name>/infra/secrets.yaml file and specify the new password using the keystone_admin_password parameter.

  5. Re-run the Deploy - OpenStack Jenkins pipeline job.


26269

Queens. Fixed in 2019.2.3

Changing the logging level for the OpenStack services may fail.

Workaround:

  1. Apply the Salt formula patch to your Oslo templates Salt formula.

  2. Apply the OpenStack states depending on your deployment. For example, if on Nova compute you have Nova, Neutron, and Cinder, apply salt cmp* state.apply nova,neutron,cinder. Alternatively, re-run the Deploy - OpenStack Jenkins pipeline job.


27071

Pike, Queens

On the OpenStack Pike or Queens environments with Octavia, if during creation, updating, or deleting of a load balancer or other resources a gtw node is rebooted or the octavia-worker service is restarted, the stale load balancer stucks in the PENDING_UPDATE or PENDING_DELETE state.

Workaround:

  1. Log in to any OpenStack controller node.

  2. Obtain the target load balancer ID:

    openstack loadbalancer list | awk '/ PENDING_CREATE / {print $2}
    
  3. Choose from the following options:

    • For the MCP version 2019.2.4 and later, run the following command:

      openstack loadbalancer delete --force <load_balancer_id>
      

      Note

      The --force flag requires admin rights and works only if a load balancer was not updated during the last hour.

    • For the MCP versions older than 2019.2.4:

      1. Log in to any dbs node.

      2. Log in to the MySQL database:

        mysql -uoctavia -p
        
      3. Run the following command with the load balancer ID obtained in the step 2. For example:

        update load_balancer set provisioning_status='ERROR' \
        where id='0fc571fe-6ad1-4311-ab13-765b5526cd30';
        

27403

Pike, Queens

On the OpenStack Pike or Queens environments with Octavia, if a gtw node hosting the Octavia services has issues with tenant network causing the Octavia management network lb-mgmt-net to become unreachable from this gtw node, the Octavia controller services stop working properly without connection to the amphora instances.

Workaround:

  • If you run the Octavia services on all gtw nodes using octavia_manager_cluster and only one gtw node has tenant network issues, manually stop the Octavia controller services (octavia-health-manager, octavia-housekeeping, octavia-worker) on the affected node until the network issue on this node is resolved. In this case, the Octavia controller services will continue working properly.

  • If you run the Octavia services only on the gtw01 node, manually stop the Octavia controller services and choose from the following options:

    • Start the Octavia controller services on another gtw0x node:

      1. Open your Git project repository with the Reclass model on the cluster level.

      2. In cluster/<cluster_name>/infra/config/nodes.yml, change the node for the Octavia services, for example, to gtw02:

        parameters:
          reclass:
            storage:
              node:
                openstack_gateway_node02:
                  classes:
                  - cluster.${_param:cluster_name}.openstack.octavia_manager
                  params:
                    octavia_hm_bind_ip: ${_param:octavia_health_manager_node01_address}
        
      3. Log in to the Salt Master node.

      4. Apply the following states:

        salt-call state.sls reclass.storage
        salt '*' saltutil.refresh_pillar
        salt -C 'I@neutron:client' state.sls neutron.client
        salt '*' mine.update
        
      5. For the gtw node where you moved the Octavia services, apply the Octavia states. For example:

        salt 'gtw02*' state.sls octavia
        
    • TECHNICAL PREVIEW Enable octavia_manager_cluster:

      1. Open your Git project repository with the Reclass model on the cluster level.

      2. In infra/<cluster_name>/infra/config/init.yml, change the following class

        - system.reclass.storage.system.openstack_gateway_single_octavia
        

        to

        - system.reclass.storage.system.openstack_gateway_cluster_octavia
        
      3. Log in to the Salt Master node.

      4. Apply the following states:

        salt-call state.sls reclass.storage
        salt '*' saltutil.refresh_pillar
        salt -C 'I@neutron:client' state.sls neutron.client
        salt '*' mine.update
        salt -C "I@octavia:manager and not *01*" state.sls octavia
        

33365

Pike, Queens

The Nova scheduler counts the disk space of the volume-backed instances and causes NoValidHostFound errors from Nova when booting an instance. The reason is that Nova considers the size of the root volume specified in the instance flavor to be consumed by that instance on the compute host even if the instance is booted from the Cinder volume and does not consume any disk resources on the compute host.

Workarounds:

  • If your cloud uses instances booted only or mostly from Cinder volumes, increase the disk overcommit ratio:

    1. Open your Git project repository with the Reclass model on the cluster level.

    2. In cluster/<cluster_name>/openstack/control.yml, increase the disk allocation ratio as required using the disk_allocation_ratio parameter:

      nova:
        controller:
          disk_allocation_ratio: <integer>
      
    3. From the Salt Master node, apply the nova state:

      salt 'ctl*' state.apply nova
      
  • If only some instances boot from Cinder volumes, create a separate flavor of zero size for the root volume to be used by such instances. Use these flavors when creating instances booted from Cinder volumes.

    1. Open your Git project repository with the Reclass model on the cluster level.

    2. In cluster/<cluster_name>/openstack/control.yml, define a new flavor and set disk to 0. Set other parameters as required. For example:

      nova:
         client:
           enabled: true
           server:
             identity:
              flavor:
                flavor1:
                  flavor_id: 10
                  ram: 4096
                  disk: 0
                  vcpus: 1
      
    3. From the Salt Master node, apply the novaclient state:

      salt 'ctl*' state.apply novaclient
      

33576

Pike, Queens

A Neutron port on a private network may receive traffic from other networks or VLANs during wiring. The workaround is to use iptables instead of the Open vSwitch security groups.

Workaround:

  1. Open your Git project repository with the Reclass model on the cluster level.

  2. In cluster/<cluster_name>/openstack/compute.yaml, set the firewall_driver to iptables_hybrid:

    neutron:
      compute:
        firewall_driver: iptables_hybrid
    
  3. Apply the neutron state from the Salt Master node:

    salt -C 'I@neutron:server' state.sls neutron
    

34028

Pike, Queens

The Keepalived service may fail during the upgrade from MCP versions lower than 2018.11.0 to 2019.2.0. The workaround is to disable Keepalived monitoring and enable it once you complete the upgrade.

Workaround:

  1. Open your Git project repository with the Reclass model on the cluster level.

  2. In classes/cluster/<cluster_name>/infra/init.yml, disable Keepalived monitoring:

    keepalived:
      _support:
        telegraf:
          enabled: false
    
  3. Verify that all nodes have the Telegraf Keepalived plugin disabled:

    salt -C "*" saltutil.refresh_pillar
    
  4. Verify that no nodes respond against test.ping:

    salt -C "I@keepalived:_support:telegraf:enabled:True" test.ping
    
  5. Apply the change:

    salt -C 'I@telegraf:agent' state.sls telegraf
    
  6. Once you complete the upgrade, revert the step 2.

  7. Apply the change:

    salt -C 'I@telegraf:agent' state.sls telegraf
    

34455

Pike, Queens

After deployment of an OpenStack environment, the VCP nodes may have an incorrect DNS server if MAAS is used. The reason is that during a VCP node boot, it obtains a DNS server from MAAS, which may differ from the DNS server specified in the deployment model.

To resolve the issue for the existing VCP nodes, remove the wrong DNS server address from the /etc/resolv.conf configuration file. To resolve the issue before deploying a new environment or adding new VCP nodes to an existing environment, specify the network data in cloud-init.

To apply a workaround for existing VCP nodes:

  1. Log in to the Salt Master node.

  2. Obtain the MAAS DNS server address:

    salt-call pillar.get maas:region:bind:host
    
  3. Remove the MAAS DNS server address from the affected nodes:

    salt -C '<target_compound>' cmd.run 'sed -i /<maas_server>/d /etc/resolv.conf
    

To apply a permanent solution for every new VCP node:

  1. Log in to the Salt Master node.

  2. Obtain the list of VCP nodes defined in the model:

    salt '<any_kvm_node>' --out json pillar.get salt:control:cluster:internal:node | jq -r '.[] | keys[]'
    

    Example of system response:

    bmk01
    cid01
    cid02
    cid03
    ...
    prx01
    prx02
    
  3. Determine the VCP nodes pillars that contain the cloud-init data:

    salt '<any_kvm_node>' --out yaml pillar.items | grep 'salt_control_cluster_node_cloud_init_'
    

    Example of system response:

    salt_control_cluster_node_cloud_init_openstack_control:
    salt_control_cluster_node_cloud_init_openstack_dns:
    salt_control_cluster_node_cloud_init_openstack_proxy:
    salt_control_cluster_node_cloud_init_infra_storage:
    salt_control_cluster_node_cloud_init_cicd_control:
    salt_control_cluster_node_cloud_init_stacklight_telemetry:
    
  4. Open your Git project repository with the Reclass model on the cluster level.

  5. In classes/cluster/<cluster_name>/infra/kvm.yml, specify the network data for the required nodes. For example, for the cid and prx nodes:

    parameters:
      _param:
        salt_control_vcp_deploy_interface: 'ens2'
        salt_control_vcp_deploy_interface_netmask: ${_param:deploy_network_netmask}
        salt_control_vcp_deploy_interface_gateway: ${_param:deploy_network_gateway}
        salt_control_vcp_dns_server_1: ${_param:dns_server01}
        salt_control_vcp_dns_server_2: ${_param:dns_server02}
        salt_control_cluster_node_cloud_init_network_data:
          network_data:
            links:
              - type: 'phy'
                id: ${_param:salt_control_vcp_deploy_interface}
                name: ${_param:salt_control_vcp_deploy_interface}
            services:
              - type: "dns"
                address: ${_param:salt_control_vcp_dns_server_1}
              - type: "dns"
                address: ${_param:salt_control_vcp_dns_server_2}
        salt_control_common_network_data_networks_deploy_interface_no_dhcp_common: &common_no_dhcp_data
          link: ${_param:salt_control_vcp_deploy_interface}
          type: 'ipv4'
          id: 'private-ipv4'
          netmask: ${_param:salt_control_vcp_deploy_interface_netmask}
          routes:
            - gateway: ${_param:salt_control_vcp_deploy_interface_gateway}
              network: '0.0.0.0'
              netmask: '0.0.0.0'
        salt_control_cluster_node_cloud_init_cicd_control:
          network_data: ${_param:salt_control_cluster_node_cloud_init_network_data}
        salt_control_cluster_node_cloud_init_openstack_proxy:
          network_data: ${_param:salt_control_cluster_node_cloud_init_network_data}
      salt:
        control:
          cluster:
            internal:
              node:
                cid01:
                  cloud_init:
                    network_data:
                      networks:
                        - <<: *common_no_dhcp_data
                          ip_address: ${_param:cicd_control_node01_deploy_address}
                cid02:
                  cloud_init:
                    network_data:
                      networks:
                        - <<: *common_no_dhcp_data
                          ip_address: ${_param:cicd_control_node02_deploy_address}
                cid03:
                  cloud_init:
                    network_data:
                      networks:
                        - <<: *common_no_dhcp_data
                          ip_address: ${_param:cicd_control_node03_deploy_address}
                prx01:
                  cloud_init:
                    network_data:
                      networks:
                        - <<: *common_no_dhcp_data
                          ip_address: ${_param:openstack_proxy_node01_deploy_address}
                prx02:
                  cloud_init:
                    network_data:
                      networks:
                        - <<: *common_no_dhcp_data
                          ip_address: ${_param:openstack_proxy_node02_deploy_address}
    
  6. Synchronize the Salt resources:

    salt -C 'I@salt:control' saltutil.sync_all
    
  7. Proceed with OpenStack environment deployment: