Mirantis OpenStack for Kubernetes Deployment Guide¶

Configure networking on a server with three NICs¶

This section provides recommended configuration for networking on servers with three NICs that include:

• eno1 that is used as a PXE interface

• ens3f1 and ens3f2 that are used for bond0

To configure networking:

1. Create subnets. For example:

   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-backend
     namespace: managed-ns
   spec:
     cidr: 10.12.0.0/24
   
   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-frontend
     namespace: managed-ns
   spec:
     cidr: 10.12.1.0/24
   

2. Create the openstack-example-3nic L2 template.

   Caution

   The bootstrapping engine automatically assigns an IP address to the PXE nic 0 NIC. To prevent the IP duplication during updates, do not assign the IP address manually.

   Example template:

   apiVersion: ipam.mirantis.com/v1alpha1
   kind: L2Template
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: openstack-example-3nic
     namespace: managed-ns
   spec:
     clusterRef: managed-cluster
     autoIfMappingPrio:
     - provision
     - eno
     - ens
     - enp
     npTemplate: |-
       version: 2
       ethernets:
         {{nic 0}}:
           # IMPORTANT: Do not assign an IP address here explicitly
           # to prevent IP duplication issues. The IP will be assigned
           # automatically by the bootstrapping engine.
           # addresses: []
           match:
             macaddress: {{mac 0}}
           set-name: {{nic 0}}
           mtu: 1500
         {{nic 1}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 1}}
           set-name: {{nic 1}}
           mtu: 9000
         {{nic 2}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 2}}
           set-name: {{nic 2}}
           mtu: 9000
       bonds:
         bond0:
           interfaces:
             - {{nic 1}}
             - {{nic 2}}
       vlans:
         pr-floating:
           id: 403
           link: bond0
         stor-frontend:
           id: 404
           link: bond0
           addresses:
             - {{ip "stor-frontend:storage-frontend"}}
           mtu: 9000
         stor-backend:
           id: 405
           link: bond0
           addresses:
             - {{ip "stor-backend:storage-backend"}}
           mtu: 9000
   

Configure networking on a server with five NICs¶

This section provides recommended configuration for networking on servers with five NICs that include:

• eno1 that is used as a PXE interface

• ens3f1 and ens3f2 that are used for bond0

• ens5f1 and ens5f2 that are used for bond1

To configure networking:

1. Create subnets. For example:

   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-backend
     namespace: managed-ns
   spec:
     cidr: 10.12.0.0/24
   
   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-frontend
     namespace: managed-ns
   spec:
     cidr: 10.12.1.0/24
   
   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: tenant-tunnel
     namespace: managed-ns
   spec:
     cidr: 10.12.3.0/24
   

2. Create the openstack-example-5nic L2 template.

   Caution

   The bootstrapping engine automatically assigns an IP address to the PXE nic 0 NIC. To prevent the IP duplication during updates, do not assign the IP address manually.

   Example template:

   apiVersion: ipam.mirantis.com/v1alpha1
   kind: L2Template
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: openstack-example-5nic
     namespace: managed-ns
   spec:
     clusterRef: managed-cluster
     autoIfMappingPrio:
     - provision
     - eno
     - ens
     - enp
     npTemplate: |-
       version: 2
       ethernets:
         {{nic 0}}:
           # IMPORTANT: Do not assign an IP address here explicitly
           # to prevent IP duplication issues. The IP will be assigned
           # automatically by the bootstrapping engine.
           # addresses: []
           match:
             macaddress: {{mac 0}}
           set-name: {{nic 0}}
           mtu: 1500
         {{nic 1}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 1}}
           set-name: {{nic 1}}
           mtu: 9000
         {{nic 2}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 2}}
           set-name: {{nic 2}}
           mtu: 9000
         {{nic 3}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 3}}
           set-name: {{nic 3}}
           mtu: 9000
         {{nic 4}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 4}}
           set-name: {{nic 4}}
           mtu: 9000
       bonds:
         bond0:
           interfaces:
             - {{nic 1}}
             - {{nic 2}}
         bond1:
           interfaces:
             - {{nic 3}}
             - {{nic 4}}
       vlans:
         pr-floating:
           id: 403
           link: bond1
         stor-frontend:
           id: 404
           link: bond0
           addresses:
             - {{ip "stor-frontend:storage-frontend"}}
           mtu: 9000
         stor-backend:
           id: 405
           link: bond1
           addresses:
             - {{ip "stor-backend:storage-backend"}}
           mtu: 9000
         tenant-tunnel:
           id: 406
           link: bond1
           addresses:
             - {{ip "tenant-tunnel:tenant-tunnel"}}
           mtu: 9000
   

Enable LVM ephemeral storage¶

This section instructs you on how to configure LVM as back end for the VM disks and ephemeral storage.

To enable LVM ephemeral storage:

1. In BareMetalHostProfile in the spec:volumeGroups section, specify the following pillar for the OpenStack compute nodes:

   spec:
     devices:
       - device:
           byName: /dev/nvme0n1
           minSizeGiB: 30
           wipe: true
         partitions:
           - name: lvm_nova_vol
             sizeGiB: 0
             wipe: true
     volumeGroups:
       - devices:
         - partition: lvm_nova_vol
         name: nova-vol
     logicalVolumes:
       - name: fake
         vg: nova-vol
         sizeGiB: 0.1
     fileSystems:
       - fileSystem: ext4
         logicalVolume: fake
         mountPoint: /fake
   

   Note

   Due to a limitation, it is not possible to create volume groups without logical volumes and formatted partitions. Therefore, set the logicalVolumes:name, fileSystem:logicalVolume, and file:System:mountPoint parameters to fake.

   For details about BareMetalHostProfile, see Mirantis Container Cloud Operations Guide: Create a custom bare metal host profile.

2. Configure the OpenStackDeployment CR to deploy OpenStack with LVM ephemeral storage. For example:

   spec:
     features:
       nova:
         images:
           backend: lvm
           lvm:
             volume_group: "nova-vol"
   

3. Optional. Enable encryption for the LVM ephemeral storage by adding the following metadata in the OpenStackDeployment CR:

   spec:
     features:
       nova:
         images:
           encryption:
             enabled: true
             cipher: "aes-xts-plain64"
             key_size: 256
   

   Caution

   Live migration is not supported for such instances.

Enable DPDK with OVS¶

This section instructs you on how to enable DPDK with the Neutron OVS back end.

To enable DPDK with OVS:

1. Verify that your deployment meets the following requirements:

   • The required drivers have been installed on the host operating system.

     Different Poll Mode Driver (PMD) types may require different kernel drivers to properly work with NIC. For more information about the DPDK drivers, read DPDK official documentation: Linux Drivers and Overview of Networking Drivers.

   • The DPDK NICs are not used on the host operating system.

   • The huge pages feature is enabled on the host. See Enable huge pages for OpenStack for details.

2. Enable DPDK in the OsDpl custom resource through the node specific overrides settings. For example:

   spec:
     nodes:
       <NODE-LABEL>::<NODE-LABEL-VALUE>:
         features:
           neutron:
             dpdk:
               bridges:
               - ip_address: 10.12.2.80/24
                 name: br-phy
               driver: igb_uio
               enabled: true
               nics:
               - bridge: br-phy
                 name: nic01
                 pci_id: "0000:05:00.0"
             tunnel_interface: br-phy
   

Enable SR-IOV with OVS¶

This section instructs you on how to enable SR-IOV with the Neutron OVS back end.

To enable SR-IOV with OVS:

1. Verify that your deployment meets the following requirements:

   • NICs with the SR-IOV support are installed

   • SR-IOV and VT-d are enabled in BIOS

2. Enable IOMMU in the kernel by configuring intel_iommu=on in the GRUB configuration file. Specify the parameter for compute nodes in BareMetalHostProfile in the grubConfig section:

   spec:
     grubConfig:
         defaultGrubOptions:
           - 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX intel_iommu=on"'
   

3. Configure the OpenStackDeployment CR to deploy OpenStack with the VLAN tenant network encapsulation.

   Caution

   To ensure correct appliance of the configuration changes, configure VLAN segmentation during the initial OpenStack deployment.

   Configuration example:

   spec:
     features:
       neutron:
         external_networks:
         - bridge: br-ex
           interface: pr-floating
           mtu: null
           network_types:
           - flat
           physnet: physnet1
           vlan_ranges: null
         - bridge: br-tenant
           interface: bond0
           network_types:
             - vlan
           physnet: tenant
           vlan_ranges: 490:499,1420:1459
         tenant_network_types:
           - vlan
   

4. Enable SR-IOV in the OpenStackDeployment CR through the node-specific overrides settings. For example:

   spec:
     nodes:
       <NODE-LABEL>::<NODE-LABEL-VALUE>:
         features:
           neutron:
             sriov:
               enabled: true
               nics:
               - device: enp10s0f1
                 num_vfs: 7
                 physnet: tenant
   

Advanced configuration for OpenStack compute nodes¶

The section describes how to perform advanced configuration for the OpenStack compute nodes. Such configuration can be required in some specific use cases, such as DPDK, SR-IOV, or huge pages features usage.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=8-19"

spec:
  nodes:
    <NODE-LABEL>::<NODE-LABEL-VALUE>:
      services:
        compute:
          nova_compute:
            values:
              conf:
                nova:
                  compute:
                    cpu_dedicated_set: 2-17
                    cpu_shared_set: 18-47

Configure DNS to access OpenStack¶

The OpenStack services are exposed through the Ingress NGINX controller.

To configure DNS to access your OpenStack environment:

1. Obtain the external IP address of the Ingress service:

   kubectl -n openstack get services ingress
   

   Example of system response:

   NAME      TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                                      AGE
   ingress   LoadBalancer   10.96.32.97   10.172.1.101   80:34234/TCP,443:34927/TCP,10246:33658/TCP   4h56m
   

2. Select from the following options:

   • If you have a corporate DNS server, update your corporate DNS service and create appropriate DNS records for all OpenStack public endpoints.

     To obtain the full list of public endpoints:

     kubectl -n openstack get ingress -ocustom-columns=NAME:.metadata.name,HOSTS:spec.rules[*].host | awk '/cluster-fqdn/ {print $2}'
     

     Example of system response:

     barbican.it.just.works
     cinder.it.just.works
     cloudformation.it.just.works
     designate.it.just.works
     glance.it.just.works
     heat.it.just.works
     horizon.it.just.works
     keystone.it.just.works
     neutron.it.just.works
     nova.it.just.works
     novncproxy.it.just.works
     octavia.it.just.works
     placement.it.just.works
     

   • If you do not have a corporate DNS server, perform one of the following steps:

     • Add the appropriate records to /etc/hosts locally. For example:

       10.172.1.101 barbican.it.just.works
       10.172.1.101 cinder.it.just.works
       10.172.1.101 cloudformation.it.just.works
       10.172.1.101 designate.it.just.works
       10.172.1.101 glance.it.just.works
       10.172.1.101 heat.it.just.works
       10.172.1.101 horizon.it.just.works
       10.172.1.101 keystone.it.just.works
       10.172.1.101 neutron.it.just.works
       10.172.1.101 nova.it.just.works
       10.172.1.101 novncproxy.it.just.works
       10.172.1.101 octavia.it.just.works
       10.172.1.101 placement.it.just.works
       

     • Deploy your DNS server on top of Kubernetes:

       1. Deploy a standalone CoreDNS server by including the following configuration into coredns.yaml:

          apiVersion: lcm.mirantis.com/v1alpha1
          kind: HelmBundle
          metadata:
            name: coredns
            namespace: osh-system
          spec:
            repositories:
            - name: hub_stable
              url: https://kubernetes-charts.storage.googleapis.com
            releases:
            - name: coredns
              chart: hub_stable/coredns
              version: 1.8.1
              namespace: coredns
              values:
                image:
                  repository: mirantis.azurecr.io/openstack/extra/coredns
                  tag: "1.6.9"
                isClusterService: false
                servers:
                - zones:
                  - zone: .
                    scheme: dns://
                    use_tcp: false
                  port: 53
                  plugins:
                  - name: cache
                    parameters: 30
                  - name: errors
                  # Serves a /health endpoint on :8080, required for livenessProbe
                  - name: health
                  # Serves a /ready endpoint on :8181, required for readinessProbe
                  - name: ready
                  # Required to query kubernetes API for data
                  - name: kubernetes
                    parameters: cluster.local
                  - name: loadbalance
                    parameters: round_robin
                  # Serves a /metrics endpoint on :9153, required for serviceMonitor
                  - name: prometheus
                    parameters: 0.0.0.0:9153
                  - name: forward
                    parameters: . /etc/resolv.conf
                  - name: file
                    parameters: /etc/coredns/it.just.works.db it.just.works
                serviceType: LoadBalancer
                zoneFiles:
                - filename: it.just.works.db
                  domain: it.just.works
                  contents: |
                    it.just.works.            IN      SOA     sns.dns.icann.org. noc.dns.icann.org. 2015082541 7200 3600 1209600 3600
                    it.just.works.            IN      NS      b.iana-servers.net.
                    it.just.works.            IN      NS      a.iana-servers.net.
                    it.just.works.            IN      A       1.2.3.4
                    *.it.just.works.           IN      A      1.2.3.4
          

       2. Update the public IP address of the Ingress service:

          sed -i 's/1.2.3.4/10.172.1.101/' release/ci/30-coredns.yaml
          kubectl apply -f release/ci/30-coredns.yaml
          

       3. Verify that the DNS resolution works properly:

          1. Assign an external IP to the service:

             kubectl -n coredns patch service coredns-coredns --type='json' -p='[{"op": "replace", "path": "/spec/ports", "value": [{"name": "udp-53", "port": 53, "protocol": "UDP", "targetPort": 53}]}]'
             kubectl -n coredns patch service coredns-coredns --type='json' -p='[{"op": "replace", "path": "/spec/type", "value":"LoadBalancer"}]'
             

          2. Obtain the external IP address of CoreDNS:

             kubectl -n coredns get service coredns-coredns
             

             Example of system response:

             NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
             coredns-coredns   ClusterIP   10.96.178.21   10.172.1.102      53/UDP,53/TCP   25h
             

       4. Point your machine to use the correct DNS. It is 10.172.1.102 in the example system response above.

       5. If you plan to launch Tempest tests or use the OpenStack client from a keystone-client-XXX pod, verify that the Kubernetes built-in DNS service is configured to resolve your public FQDN records by adding your public domain to Corefile. For example, to add the it.just.works domain:

          kubectl -n kube-system get configmap coredns -oyaml
          

          Example of system response:

          apiVersion: v1
          data:
            Corefile: |
              .:53 {
                  errors
                  health
                  ready
                  kubernetes cluster.local in-addr.arpa ip6.arpa {
                    pods insecure
                    fallthrough in-addr.arpa ip6.arpa
                  }
                  prometheus :9153
                  forward . /etc/resolv.conf
                  cache 30
                  loop
                  reload
                  loadbalance
              }
              it.just.works:53 {
                  errors
                  cache 30
                  forward . 10.96.178.21
              }
          

Access your OpenStack environment¶

This section explains how to access your OpenStack environment as the Admin user.

Before you proceed, verify that you can access the Kubernetes API and have privileges to read secrets from the openstack namespace in Kubernetes or you are able to exec to the pods in this namespace.

kubectl -n openstack exec \
  $(kubectl -n openstack get pod -l application=keystone,component=client -ojsonpath='{.items[*].metadata.name}') \
  -ti -- bash

Debugging the HelmBundle controller¶

The HelmBundle controller is running in the following containers:

• tiller - runs Tiller, the server portion of Helm

• controller - contains processes that handle the HelmBundle object and deploy it though Tiller

kubectl -n osh-system get helmbundles openstack-operator -o yaml

status:
  releaseStatuses:
    openstack-operator:
      attempt: 1
      chart: openstack-controller/openstack-operator
      finishedAt: "2020-04-21T11:24:08Z"
      hash: caa6a811e5540333b4a02d74f7ec855460d97f81efdd69ed13d516256cbf9f81
      notes: ""
      revision: 1
      status: DEPLOYED
      success: true
      version: 0.1.9-55

Debugging the OpenStack Controller¶

The OpenStack Controller is running in several containers in the openstack-operator-xxxx pod in the osh-system namespace. For the full list of containers and their roles, refer to MOS Reference Architecture: Openstack controller containers.

To verify the status of the OpenStack Controller, run:

kubectl -n osh-system get pods

Example of a system response:

NAME                                  READY   STATUS    RESTARTS   AGE
helm-controller-0                     2/2     Running   0          70m
local-volume-provisioner-2xsl5        1/1     Running   0          70m
local-volume-provisioner-fqdf9        1/1     Running   0          70m
local-volume-provisioner-ncclp        1/1     Running   0          70m
openstack-operator-7f4b5cf88c-4tj26   5/5     Running   0          70m

To verify the logs for the osdpl container, run:

kubectl -n osh-system logs -f <openstack-operator-xxxx> -c osdpl

Debugging the OsDpl CR¶

This section includes the ways to mitigate the most common issues with the OsDpl CR. We assume that you have already debugged the HelmBundle and OpenStack Controllers to rule out possible failures with these components as described in Debugging the HelmBundle controller and Debugging the OpenStack Controller.

kubectl -n openstack get osdpl osh-dev

NAME      AGE   DEPLOYED   DRAFT
osh-dev   22h   false      false

kubectl -n openstack logs -f placement-api-84669d79b5-49drw -c init

Entrypoint WARNING: 2020/04/21 11:52:50 entrypoint.go:72: Resolving dependency Job placement-ks-user in namespace openstack failed: Job Job placement-ks-user in namespace openstack is not completed yet .
Entrypoint WARNING: 2020/04/21 11:52:52 entrypoint.go:72: Resolving dependency Job placement-ks-endpoints in namespace openstack failed: Job Job placement-ks-endpoints in namespace openstack is not completed yet .

kubectl -n openstack-ceph-shared get secrets openstack-ceph-keys

NAME                  TYPE     DATA   AGE
openstack-ceph-keys   Opaque   7      23h

Enable DPDK for Tungsten Fabric¶

This section describes how to enable DPDK mode for the Tungsten Fabric (TF) vRouter.

To enable DPDK for TF:

1. Install the required drivers on the host operating system. The vfio-pci, uio_pci_generic, or mlnx drivers can be used with the TF vRouter agent in DPDK mode. For details about DPDK drivers, see Linux Drivers.

2. Enable huge pages on the host as described in Mirantis Container Cloud Operations Guide: Enable huge pages in a host profile.

3. Mark the hosts for deployment with DPDK with the tfvrouter-dpdk=enabled label.

4. Open the TF Operator custom resource for editing:

   kubectl -n tf edit tfoperators.operator.tf.mirantis.com openstack-tf
   

5. Enable DPDK:

   spec:
     controllers:
       tf-vrouter:
         agent-dpdk:
           enabled: true
   

Enable SR-IOV for Tungsten Fabric¶

This section instructs you on how to enable SR-IOV with the Neutron Tungsten Fabric (TF) back end.

To enable SR-IOV for TF:

1. Verify that your deployment meets the following requirements:

   • NICs with the SR-IOV support are installed

   • SR-IOV and VT-d are enabled in BIOS

2. Enable IOMMU in the kernel by configuring intel_iommu=on in the GRUB configuration file. Specify the parameter for compute nodes in BareMetalHostProfile in the grubConfig section:

   spec:
     grubConfig:
       defaultGrubOptions:
         - 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX intel_iommu=on"'
   

3. Enable SR-IOV in the OpenStackDeployment CR through the node-specific overrides settings. For example:

   spec:
     nodes:
       <NODE-LABEL>::<NODE-LABEL-VALUE>:
         features:
           neutron:
             sriov:
               enabled: true
               nics:
               - device: enp10s0f1
                 num_vfs: 7
                 physnet: tenant
   

4. Enable SR-IOV in the TFOperator CR through the custom vRouter settings. For example:

   spec:
     controllers:
       tf-vrouter:
         agent:
           customSpecs:
           - name: sriov
             label:
               name: <NODE-LABEL>
               value: <NODE-LABEL-VALUE>
             containers:
             - name: agent
               env:
               - name: SRIOV_PHYSICAL_INTERFACE
                 value: enp10s0f1
               - name: SRIOV_VF
                 value: "7"
   

Specify custom vRouter settings¶

You may need to specify custom settings for the Tungsten Fabric (TF) vRouter nodes, for example, to change the name of the tunnel network interface or enable debug level logging. To do so, use the customSpecs parameter in the TFOperator CR.

For example, to enable debug level logging on a specific node or multiple nodes:

spec:
  controllers:
    tf-vrouter:
      agent:
        customSpecs:
        - name: debug
          label:
            name: <NODE-LABEL>
            value: <NODE-LABEL-VALUE>
          containers:
          - name: agent
            env:
            - name: LOG_LEVEL
              value: SYS_DEBUG

The customSpecs parameter inherits all settings for the tf-vrouter containers that are set on the spec:controllers:agent level and overrides or adds additional parameters. The example configuration above overrides the logging level from SYS_INFO, which is the default logging level, to SYS_DEBUG.

Enable debug logs for the Tungsten Fabric services¶

To enable debug logging for the Tungsten Fabric (TF) services:

1. Open the TF custom resource for modification:

   kubectl -n tf edit tfoperators.operator.tf.mirantis.com openstack-tf
   

2. Specify the LOG_LEVEL variable with the SYS_DEBUG value for the required TF service. For example, for the config-api service:

   spec:
     controllers:
       tf-config:
         api:
           containers:
           - name: api
             env:
             - name: LOG_LEVEL
               value: SYS_DEBUG
   

Troubleshoot access to the Tungsten Fabric web UI¶

If you cannot access the Tungsten Fabric (TF) web UI service, verify that the FQDN of the TF web UI is resolvable on your PC by running one of the following commands:

host tf-webui.it.just.works
# or
ping tf-webui.it.just.works
# or
dig host tf-webui.it.just.works

All commands above should resolve the web UI domain name to the IP address that should match the EXTERNAL-IPs subnet dedicated to Kubernetes.

If the TF web UI domain name has not been resolved to the IP address, your PC is using a different DNS or the DNS does not contain the record for the TF web UI service. To resolve the issue, define the IP address of the Ingress service from the openstack namespace of Kubernetes in the hosts file of your machine. To obtain the Ingress IP address:

kubectl -n openstack get svc ingress -o custom-columns=HOSTS:.status.loadBalancer.ingress[*].ip

If the web UI domain name is resolvable but you still cannot access the service, verify the connectivity to the cluster.