Mirantis OpenStack for Kubernetes Deployment Guide¶

Configure networking on a server with three NICs¶

This section provides recommended configuration for networking on servers with three NICs that include:

• eno1 that is used as a PXE interface

• ens3f1 and ens3f2 that are used for bond0

To configure networking:

1. Create subnets. For example:

   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-backend
     namespace: managed-ns
   spec:
     cidr: 10.12.0.0/24
   
   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-frontend
     namespace: managed-ns
   spec:
     cidr: 10.12.1.0/24
   

2. Create the openstack-example-3nic L2 template.

   Caution

   The bootstrapping engine automatically assigns an IP address to the PXE nic 0 NIC. To prevent the IP duplication during updates, do not assign the IP address manually.

   Example template:

   apiVersion: ipam.mirantis.com/v1alpha1
   kind: L2Template
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: openstack-example-3nic
     namespace: managed-ns
   spec:
     clusterRef: managed-cluster
     autoIfMappingPrio:
     - provision
     - eno
     - ens
     - enp
     npTemplate: |-
       version: 2
       ethernets:
         {{nic 0}}:
           # IMPORTANT: Do not assign an IP address here explicitly
           # to prevent IP duplication issues. The IP will be assigned
           # automatically by the bootstrapping engine.
           # addresses: []
           match:
             macaddress: {{mac 0}}
           set-name: {{nic 0}}
           mtu: 1500
         {{nic 1}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 1}}
           set-name: {{nic 1}}
           mtu: 9000
         {{nic 2}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 2}}
           set-name: {{nic 2}}
           mtu: 9000
       bonds:
         bond0:
           interfaces:
             - {{nic 1}}
             - {{nic 2}}
       vlans:
         pr-floating:
           id: 403
           link: bond0
         stor-frontend:
           id: 404
           link: bond0
           addresses:
             - {{ip "stor-frontend:storage-frontend"}}
           mtu: 9000
         stor-backend:
           id: 405
           link: bond0
           addresses:
             - {{ip "stor-backend:storage-backend"}}
           mtu: 9000
   

Configure networking on a server with five NICs¶

This section provides recommended configuration for networking on servers with five NICs that include:

• eno1 that is used as a PXE interface

• ens3f1 and ens3f2 that are used for bond0

• ens5f1 and ens5f2 that are used for bond1

To configure networking:

1. Create subnets. For example:

   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-backend
     namespace: managed-ns
   spec:
     cidr: 10.12.0.0/24
   
   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-frontend
     namespace: managed-ns
   spec:
     cidr: 10.12.1.0/24
   
   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: tenant-tunnel
     namespace: managed-ns
   spec:
     cidr: 10.12.3.0/24
   

2. Create the openstack-example-5nic L2 template.

   Caution

   The bootstrapping engine automatically assigns an IP address to the PXE nic 0 NIC. To prevent the IP duplication during updates, do not assign the IP address manually.

   Example template:

   apiVersion: ipam.mirantis.com/v1alpha1
   kind: L2Template
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: openstack-example-5nic
     namespace: managed-ns
   spec:
     clusterRef: managed-cluster
     autoIfMappingPrio:
     - provision
     - eno
     - ens
     - enp
     npTemplate: |-
       version: 2
       ethernets:
         {{nic 0}}:
           # IMPORTANT: Do not assign an IP address here explicitly
           # to prevent IP duplication issues. The IP will be assigned
           # automatically by the bootstrapping engine.
           # addresses: []
           match:
             macaddress: {{mac 0}}
           set-name: {{nic 0}}
           mtu: 1500
         {{nic 1}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 1}}
           set-name: {{nic 1}}
           mtu: 9000
         {{nic 2}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 2}}
           set-name: {{nic 2}}
           mtu: 9000
         {{nic 3}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 3}}
           set-name: {{nic 3}}
           mtu: 9000
         {{nic 4}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 4}}
           set-name: {{nic 4}}
           mtu: 9000
       bonds:
         bond0:
           interfaces:
             - {{nic 1}}
             - {{nic 2}}
         bond1:
           interfaces:
             - {{nic 3}}
             - {{nic 4}}
       vlans:
         pr-floating:
           id: 403
           link: bond1
         stor-frontend:
           id: 404
           link: bond0
           addresses:
             - {{ip "stor-frontend:storage-frontend"}}
           mtu: 9000
         stor-backend:
           id: 405
           link: bond1
           addresses:
             - {{ip "stor-backend:storage-backend"}}
           mtu: 9000
         tenant-tunnel:
           id: 406
           link: bond1
           addresses:
             - {{ip "tenant-tunnel:tenant-tunnel"}}
           mtu: 9000
   

Enable DPDK with OVS¶

This section instructs you on how to enable DPDK with the Neutron OVS back end.

To enable DPDK with OVS:

1. Verify that your deployment meets the following requirements:

   • The required drivers have been installed on the host operating system.

     Different Poll Mode Driver (PMD) types may require different kernel drivers to properly work with NIC. For more information about the DPDK drivers, read DPDK official documentation: Linux Drivers and Overview of Networking Drivers.

   • The DPDK NICs are not used on the host operating system.

   • The huge pages feature is enabled on the host. See Enable huge pages for OpenStack for details.

2. Enable DPDK in the OsDpl custom resource through the node specific overrides settings. For example:

   spec:
     nodes:
       <NODE-LABEL>::<NODE-LABEL-VALUE>:
         features:
           neutron:
             dpdk:
               bridges:
               - ip_address: 10.12.2.80/24
                 name: br-phy
               driver: igb_uio
               enabled: true
               nics:
               - bridge: br-phy
                 name: nic01
                 pci_id: "0000:05:00.0"
             tunnel_interface: br-phy
   

Enable SR-IOV with OVS¶

This section instructs you on how to enable SR-IOV with the Neutron OVS back end.

To enable SR-IOV with OVS:

1. Verify that your deployment meets the following requirements:

   • NICs with the SR-IOV support are installed

   • SR-IOV and VT-d are enabled in BIOS

2. Enable IOMMU in the kernel by configuring intel_iommu=on in the GRUB configuration file. Specify the parameter for compute nodes in BareMetalHostProfile in the grubConfig section:

   spec:
     grubConfig:
         defaultGrubOptions:
           - 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX intel_iommu=on"'
   

3. Configure the OpenStackDeployment CR to deploy OpenStack with the VLAN tenant network encapsulation.

   Caution

   To ensure correct appliance of the configuration changes, configure VLAN segmentation during the initial OpenStack deployment.

   Configuration example:

   spec:
     features:
       neutron:
         external_networks:
         - bridge: br-ex
           interface: pr-floating
           mtu: null
           network_types:
           - flat
           physnet: physnet1
           vlan_ranges: null
         - bridge: br-tenant
           interface: bond0
           network_types:
             - vlan
           physnet: tenant
           vlan_ranges: 490:499,1420:1459
         tenant_network_types:
           - vlan
   

4. Enable SR-IOV in the OpenStackDeployment CR through the node-specific overrides settings. For example:

   spec:
     nodes:
       <NODE-LABEL>::<NODE-LABEL-VALUE>:
         features:
           neutron:
             sriov:
               enabled: true
               nics:
               - device: enp10s0f1
                 num_vfs: 7
                 physnet: tenant
   

Advanced configuration for OpenStack computes nodes¶

The section describes how to perform advanced configuration for the OpenStack compute nodes. Such configuration can be required in some specific use cases, such as DPDK, SR-IOV, or huge pages features usage.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=8-19"

spec:
  nodes:
    <NODE-LABEL>::<NODE-LABEL-VALUE>:
      services:
        compute:
          nova_compute:
            values:
              conf:
                nova:
                  compute:
                    cpu_dedicated_set: 2-17
                    cpu_shared_set: 18-47

Configure DNS to access OpenStack¶

The OpenStack services are exposed through the Ingress NGINX controller.

To configure DNS to access your OpenStack environment:

1. Obtain the external IP address of the Ingress service:

   kubectl -n openstack get services ingress
   

   Example of system response:

   NAME      TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                                      AGE
   ingress   LoadBalancer   10.96.32.97   10.172.1.101   80:34234/TCP,443:34927/TCP,10246:33658/TCP   4h56m
   

2. Select from the following options:

   • If you have a corporate DNS server, update your corporate DNS service and create appropriate DNS records for all OpenStack public endpoints.

     To obtain the full list of public endpoints:

     kubectl -n openstack get ingress -ocustom-columns=NAME:.metadata.name,HOSTS:spec.rules[*].host | awk '/cluster-fqdn/ {print $2}'
     

     Example of system response:

     barbican.it.just.works
     cinder.it.just.works
     cloudformation.it.just.works
     designate.it.just.works
     glance.it.just.works
     heat.it.just.works
     horizon.it.just.works
     keystone.it.just.works
     neutron.it.just.works
     nova.it.just.works
     novncproxy.it.just.works
     octavia.it.just.works
     placement.it.just.works
     

   • If you do not have a corporate DNS server, perform one of the following steps:

     • Add the appropriate records to /etc/hosts locally. For example:

       10.172.1.101 barbican.it.just.works
       10.172.1.101 cinder.it.just.works
       10.172.1.101 cloudformation.it.just.works
       10.172.1.101 designate.it.just.works
       10.172.1.101 glance.it.just.works
       10.172.1.101 heat.it.just.works
       10.172.1.101 horizon.it.just.works
       10.172.1.101 keystone.it.just.works
       10.172.1.101 neutron.it.just.works
       10.172.1.101 nova.it.just.works
       10.172.1.101 novncproxy.it.just.works
       10.172.1.101 octavia.it.just.works
       10.172.1.101 placement.it.just.works
       

     • Deploy your DNS server on top of Kubernetes:

       1. Deploy a standalone CoreDNS server by including the following configuration into coredns.yaml:

          apiVersion: lcm.mirantis.com/v1alpha1
          kind: HelmBundle
          metadata:
            name: coredns
            namespace: osh-system
          spec:
            repositories:
            - name: hub_stable
              url: https://kubernetes-charts.storage.googleapis.com
            releases:
            - name: coredns
              chart: hub_stable/coredns
              version: 1.8.1
              namespace: coredns
              values:
                image:
                  repository: mirantis.azurecr.io/openstack/extra/coredns
                  tag: "1.6.9"
                isClusterService: false
                servers:
                - zones:
                  - zone: .
                    scheme: dns://
                    use_tcp: false
                  port: 53
                  plugins:
                  - name: cache
                    parameters: 30
                  - name: errors
                  # Serves a /health endpoint on :8080, required for livenessProbe
                  - name: health
                  # Serves a /ready endpoint on :8181, required for readinessProbe
                  - name: ready
                  # Required to query kubernetes API for data
                  - name: kubernetes
                    parameters: cluster.local
                  - name: loadbalance
                    parameters: round_robin
                  # Serves a /metrics endpoint on :9153, required for serviceMonitor
                  - name: prometheus
                    parameters: 0.0.0.0:9153
                  - name: forward
                    parameters: . /etc/resolv.conf
                  - name: file
                    parameters: /etc/coredns/it.just.works.db it.just.works
                serviceType: LoadBalancer
                zoneFiles:
                - filename: it.just.works.db
                  domain: it.just.works
                  contents: |
                    it.just.works.            IN      SOA     sns.dns.icann.org. noc.dns.icann.org. 2015082541 7200 3600 1209600 3600
                    it.just.works.            IN      NS      b.iana-servers.net.
                    it.just.works.            IN      NS      a.iana-servers.net.
                    it.just.works.            IN      A       1.2.3.4
                    *.it.just.works.           IN      A      1.2.3.4
          

       2. Update the public IP address of the Ingress service:

          sed -i 's/1.2.3.4/10.172.1.101/' release/ci/30-coredns.yaml
          kubectl apply -f release/ci/30-coredns.yaml
          

       3. Verify that the DNS resolution works properly:

          1. Assign an external IP to the service:

             kubectl -n coredns patch service coredns-coredns --type='json' -p='[{"op": "replace", "path": "/spec/ports", "value": [{"name": "udp-53", "port": 53, "protocol": "UDP", "targetPort": 53}]}]'
             kubectl -n coredns patch service coredns-coredns --type='json' -p='[{"op": "replace", "path": "/spec/type", "value":"LoadBalancer"}]'
             

          2. Obtain the external IP address of CoreDNS:

             kubectl -n coredns get service coredns-coredns
             

             Example of system response:

             NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
             coredns-coredns   ClusterIP   10.96.178.21   10.172.1.102      53/UDP,53/TCP   25h
             

       4. Point your machine to use the correct DNS. It is 10.172.1.102 in the example system response above.

       5. If you plan to launch Tempest tests or use the OpenStack client from a keystone-client-XXX pod, verify that the Kubernetes built-in DNS service is configured to resolve your public FQDN records by adding your public domain to Corefile. For example, to add the it.just.works domain:

          kubectl -n kube-system get configmap coredns -oyaml
          

          Example of system response:

          apiVersion: v1
          data:
            Corefile: |
              .:53 {
                  errors
                  health
                  ready
                  kubernetes cluster.local in-addr.arpa ip6.arpa {
                    pods insecure
                    fallthrough in-addr.arpa ip6.arpa
                  }
                  prometheus :9153
                  forward . /etc/resolv.conf
                  cache 30
                  loop
                  reload
                  loadbalance
              }
              it.just.works:53 {
                  errors
                  cache 30
                  forward . 10.96.178.21
              }
          

Access your OpenStack environment¶

This section explains how to access your OpenStack environment as the Admin user.

Before you proceed, verify that you can access the Kubernetes API and have privileges to read secrets from the openstack namespace in Kubernetes or you are able to exec to the pods in this namespace.

kubectl -n openstack exec \
  $(kubectl -n openstack get pod -l application=keystone,component=client -ojsonpath='{.items[*].metadata.name}') \
  -ti -- bash

Debugging the HelmBundle controller¶

The HelmBundle controller is running in the following containers:

• tiller - runs Tiller, the server portion of Helm

• controller - contains processes that handle the HelmBundle object and deploy it though Tiller

kubectl -n osh-system get helmbundles openstack-operator -o yaml

status:
  releaseStatuses:
    openstack-operator:
      attempt: 1
      chart: openstack-controller/openstack-operator
      finishedAt: "2020-04-21T11:24:08Z"
      hash: caa6a811e5540333b4a02d74f7ec855460d97f81efdd69ed13d516256cbf9f81
      notes: ""
      revision: 1
      status: DEPLOYED
      success: true
      version: 0.1.9-55

Debugging the OpenStack Controller¶

The OpenStack Controller is running in several containers in the openstack-operator-xxxx pod in the osh-system namespace. For the full list of containers and their roles, refer to MOS Reference Architecture: Openstack controller containers.

To verify the status of the OpenStack Controller, run:

kubectl -n osh-system get pods

Example of a system response:

NAME                                  READY   STATUS    RESTARTS   AGE
helm-controller-0                     2/2     Running   0          70m
local-volume-provisioner-2xsl5        1/1     Running   0          70m
local-volume-provisioner-fqdf9        1/1     Running   0          70m
local-volume-provisioner-ncclp        1/1     Running   0          70m
openstack-operator-7f4b5cf88c-4tj26   5/5     Running   0          70m

To verify the logs for the osdpl container, run:

kubectl -n osh-system logs -f <openstack-operator-xxxx> -c osdpl

Debugging the OsDpl CR¶

This section includes the ways to mitigate the most common issues with the OsDpl CR. We assume that you have already debugged the HelmBundle and OpenStack Controllers to rule out possible failures with these components as described in Debugging the HelmBundle controller and Debugging the OpenStack Controller.

kubectl -n openstack get osdpl osh-dev

NAME      AGE   DEPLOYED   DRAFT
osh-dev   22h   false      false

kubectl -n openstack logs -f placement-api-84669d79b5-49drw -c init

Entrypoint WARNING: 2020/04/21 11:52:50 entrypoint.go:72: Resolving dependency Job placement-ks-user in namespace openstack failed: Job Job placement-ks-user in namespace openstack is not completed yet .
Entrypoint WARNING: 2020/04/21 11:52:52 entrypoint.go:72: Resolving dependency Job placement-ks-endpoints in namespace openstack failed: Job Job placement-ks-endpoints in namespace openstack is not completed yet .

kubectl -n openstack-ceph-shared get secrets openstack-ceph-keys

NAME                  TYPE     DATA   AGE
openstack-ceph-keys   Opaque   7      23h

Enable debug logs for the Tungsten Fabric services¶

To enable debug logging for the Tungsten Fabric (TF) services:

1. Open the TF custom resource for modification:

   kubectl -n tf edit tfoperators.operator.tf.mirantis.com openstack-tf
   

2. Specify the LOG_LEVEL variable with the SYS_DEBUG value for the required TF service. For example, for the config-api service:

   spec:
     controllers:
       tf-config:
         api:
           containers:
           - name: api
             env:
             - name: LOG_LEVEL
               value: SYS_DEBUG
   

Troubleshoot access to the Tungsten Fabric web UI¶

If you cannot access the Tungsten Fabric (TF) web UI service, verify that the FQDN of the TF web UI is resolvable on your PC by running one of the following commands:

host tf-webui.it.just.works
# or
ping tf-webui.it.just.works
# or
dig host tf-webui.it.just.works

All commands above should resolve the web UI domain name to the IP address that should match the EXTERNAL-IPs subnet dedicated to Kubernetes.

If the TF web UI domain name has not been resolved to the IP address, your PC is using a different DNS or the DNS does not contain the record for the TF web UI service. To resolve the issue, define the IP address of the Ingress service from the openstack namespace of Kubernetes in the hosts file of your machine. To obtain the Ingress IP address:

kubectl -n openstack get svc ingress -o custom-columns=HOSTS:.status.loadBalancer.ingress[*].ip

If the web UI domain name is resolvable but you still cannot access the service, verify the connectivity to the cluster.