Mirantis OpenStack for Kubernetes Deployment Guide¶

Configure kernel modules¶

The OpenStack Nova and Neutron services may require additional kernel modules to be loaded for normal operation. This section instructs you on how to enable the required kernel modules.

To enable KVM

A kernel-based Virtual Machine (KVM) is a virtualization module in the Linux kernel that enables the kernel to function as a hypervisor. The KVM-based virtualization is recommended on compute hosts to provide high performance VMs. See OpenStack Train official documentation: Enable KVM for details.

Configure networking¶

Before you proceed with the networking configuration, read MOS Reference Architecture: Networking.

This section describes how to configure persistent networking on a host with 3 NICs using the Mirantis Container Cloud L2 Templates:

• eno1 is used as a PXE interface

• ens3f1 and ens3f2 are used for bond0

To configure networking:

1. Create subnets:

   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-backend
     namespace: child-ns
   spec:
     cidr: 10.12.0.0/24
   
   ---
   apiVersion: ipam.mirantis.com/v1alpha1
   kind: Subnet
   metadata:
     labels:
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: storage-frontend
     namespace: child-ns
   spec:
     cidr: 10.12.1.0/24
   

2. Create the openstack-example-3nic L2 template:

   Caution

   The bootstrapping engine automatically assigns an IP address to the PXE nic 0 NIC. To prevent the IP duplication during updates, do not assign the IP address manually.

   apiVersion: ipam.mirantis.com/v1alpha1
   kind: L2Template
   metadata:
     labels:
       ipam/Cluster: child-cluster
       kaas.mirantis.com/provider: baremetal
       kaas.mirantis.com/region: region-one
     name: openstack-example-3nic
     namespace: child-ns
   spec:
     autoIfMappingPrio:
     - provision
     - eno
     - ens
     - enp
     npTemplate: |-
       version: 2
       ethernets:
         {{nic 0}}:
           # IMPORTANT: Do not assign an IP address here explicitly
           # to prevent IP duplication issues. The IP will be assigned
           # automatically by the bootstrapping engine.
           # addresses: []
           match:
             macaddress: {{mac 0}}
           set-name: {{nic 0}}
           mtu: 1500
         {{nic 1}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 1}}
           set-name: {{nic 1}}
           mtu: 1500
         {{nic 2}}:
           dhcp4: false
           dhcp6: false
           match:
             macaddress: {{mac 2}}
           set-name: {{nic 2}}
           mtu: 1500
       bonds:
         bond0:
           interfaces:
             - {{nic 1}}
             - {{nic 2}}
       vlans:
         pr-floating:
           id: 403
           link: bond0
         stor-frontend:
           id: 404
           link: bond0
           addresses:
             - {{ip "stor-frontend:storage-frontend"}}
         stor-backend:
           id: 405
           link: bond0
           addresses:
             - {{ip "stor-backend:storage-backend"}}
   

Configure hosts files for a deployment with TF¶

TungstenFabric services use the /etc/hosts file as a source of information for services configuration and registration in the database.

To configure the hosts file:

On each node that will be used for running TungstenFabric services, verify that the /etc/hosts file has an entry with a local hostname and appropriate IP address from the Management network according to MOS Reference Architecture: Networking. For example:

127.0.0.1 localhost
<management-ip> <node-hostname>
....

Deploy an OpenStack cluster¶

This section instructs you on how to deploy OpenStack on top of Kubernetes using the OpenStack Controller and openstackdeployments.lcm.mirantis.com (OsDpl) CR.

To deploy an OpenStack cluster:

1. Verify that you have pre-configured the networking according to MOS Reference Architecture: Networking.

2. Verify that the TLS certificates that will be required for the OpenStack cluster deployment have been pre-generated.

   Note

   The Transport Layer Security (TLS) protocol is mandatory on public endpoints.

   Caution

   To avoid certificates renewal with subsequent OpenStack updates during which additional services with new public endpoints may appear, we recommend using wildcard SSL certificates for public endpoints. For example, *.it.just.works, where it.just.works is a cluster public domain.

   The sample code block below illustrates how to generate a self-signed certificate for the it.just.works domain. The procedure presumes the cfssl and cfssljson tools are installed on the machine.

   mkdir cert && cd cert
   
   tee ca-config.json << EOF
   {
     "signing": {
       "default": {
         "expiry": "8760h"
       },
       "profiles": {
         "kubernetes": {
           "usages": [
             "signing",
             "key encipherment",
             "server auth",
             "client auth"
           ],
           "expiry": "8760h"
         }
       }
     }
   }
   EOF
   
   tee ca-csr.json << EOF
   {
     "CN": "kubernetes",
     "key": {
       "algo": "rsa",
       "size": 2048
     },
     "names":[{
       "C": "<country>",
       "ST": "<state>",
       "L": "<city>",
       "O": "<organization>",
       "OU": "<organization unit>"
     }]
   }
   EOF
   
   cfssl gencert -initca ca-csr.json | cfssljson -bare ca
   
   tee server-csr.json << EOF
   {
       "CN": "*.it.just.works",
       "hosts":     [
           "*.it.just.works"
       ],
       "key":     {
           "algo": "rsa",
           "size": 2048
       },
       "names": [    {
           "C": "US",
           "L": "CA",
           "ST": "San Francisco"
       }]
   }
   EOF
   cfssl gencert -ca=ca.pem -ca-key=ca-key.pem --config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
   

3. Configure the OsDpl resource depending on the needs of your deployment.

   OsDpl is a Kubernetes CR that describes the OpenStack cluster deployment. The resource is validated with the help of the OpenAPI v3 schema. For more information about the fields and their description, refer to MOS Reference Architecture: OpenStackDeployment resource.

4. If required, enable DPDK, huge pages, and other supported Telco features as described in Advanced OpenStack configuration (optional).

5. To the openstackdeployment object, add information about the TLS certificates:

   • ssl:public_endpoints:ca_cert - CA certificate content (ca.pem)

   • ssl:public_endpoints:api_cert - server certificate content (server.pem)

   • ssl:public_endpoints:api_key - server private key (server-key.pem)

6. Verify that the Load Balancer network does not overlap your corporate or internal Kubernetes networks, for example, Calico IP pools. Also, verify that the pool of Load Balancer network is big enough to provide IP addresses for all Amphora VMs (loadbalancers).

   If required, reconfigure the Octavia network settings using the following sample structure:

   spec:
     services:
       load-balancer:
         octavia:
           values:
             octavia:
               settings:
                 lbmgmt_cidr: "10.255.0.0/16"
                 lbmgmt_subnet_start: "10.255.1.0"
                 lbmgmt_subnet_end: "10.255.255.254"
   

7. Trigger the OpenStack deployment:

   kubectl apply -f openstackdeployment.yaml
   

8. Monitor the status of your OpenStack deployment:

   kubectl -n openstack get pods
   kubectl -n openstack describe osdpl osh-dev
   

9. Assess the current status of the OpenStack deployment using the status section output in the OsDpl resource:

   1. Get the OsDpl YAML file:

      kubectl -n openstack get osdpl osh-dev -o yaml
      

   2. Analyze the status output using the detailed description in MOS Reference Architecture: OpenStackDeployment resource: The Status elements.

10. Verify that the OpenStack cluster has been deployed:

    clinet_pod_name=$(kubectl -n openstack get pods -l application=keystone,component=client  | grep keystone-client | head -1 | awk '{print $1}')
    kubectl -n openstack exec -it $clinet_pod_name -- openstack service list
    

    Example of a positive system response:

    +----------------------------------+---------------+----------------+
    | ID                               | Name          | Type           |
    +----------------------------------+---------------+----------------+
    | 159f5c7e59784179b589f933bf9fc6b0 | cinderv3      | volumev3       |
    | 6ad762f04eb64a31a9567c1c3e5a53b4 | keystone      | identity       |
    | 7e265e0f37e34971959ce2dd9eafb5dc | heat          | orchestration  |
    | 8bc263babe9944cdb51e3b5981a0096b | nova          | compute        |
    | 9571a49d1fdd4a9f9e33972751125f3f | placement     | placement      |
    | a3f9b25b7447436b85158946ca1c15e2 | neutron       | network        |
    | af20129d67a14cadbe8d33ebe4b147a8 | heat-cfn      | cloudformation |
    | b00b5ad18c324ac9b1c83d7eb58c76f5 | radosgw-swift | object-store   |
    | b28217da1116498fa70e5b8d1b1457e5 | cinderv2      | volumev2       |
    | e601c0749ce5425c8efb789278656dd4 | glance        | image          |
    +----------------------------------+---------------+----------------+
    

Advanced OpenStack configuration (optional)¶

This section includes configuration information for available advanced Mirantis OpenStack for Kubernetes features that include DPDK with the Neutron OVS back end, huge pages, CPU pinning, and other Enhanced Platform Awareness (EPA) capabilities.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=8-19"

spec:
  nodes:
    <NODE-LABEL>::<NODE-LABEL-VALUE>:
      services:
        compute:
          nova_compute:
            values:
              conf:
                nova:
                  compute:
                    cpu_dedicated_set: 2-17
                    cpu_shared_set: 18-47

Access OpenStack after deployment¶

This section contains the guidelines on how to access your MOS OpenStack environment.

kubectl -n openstack exec \
  $(kubectl -n openstack get pod -l application=keystone,component=client -ojsonpath='{.items[*].metadata.name}') \
  -ti -- bash

Troubleshoot an OpenStack deployment¶

This section provides the general debugging instructions for your OpenStack on Kubernetes deployment. Start your troubleshooting with the determination of the failing component that can include the OpenStack Operator, HelmBundle Controller, a particular pod or service.

kubectl -n osh-system get helmbundles openstack-operator -o yaml

status:
  releaseStatuses:
    openstack-operator:
      attempt: 1
      chart: openstack-controller/openstack-operator
      finishedAt: "2020-04-21T11:24:08Z"
      hash: caa6a811e5540333b4a02d74f7ec855460d97f81efdd69ed13d516256cbf9f81
      notes: ""
      revision: 1
      status: DEPLOYED
      success: true
      version: 0.1.9-55

kubectl -n osh-system get pods

NAME                                  READY   STATUS    RESTARTS   AGE
helm-controller-0                     2/2     Running   0          70m
local-volume-provisioner-2xsl5        1/1     Running   0          70m
local-volume-provisioner-fqdf9        1/1     Running   0          70m
local-volume-provisioner-ncclp        1/1     Running   0          70m
openstack-operator-7f4b5cf88c-4tj26   5/5     Running   0          70m

kubectl -n osh-system logs -f <openstack-operator-xxxx> -c osdpl

kubectl -n openstack get osdpl osh-dev

NAME      AGE   DEPLOYED   DRAFT
osh-dev   22h   false      false

kubectl -n openstack logs -f placement-api-84669d79b5-49drw -c init

Entrypoint WARNING: 2020/04/21 11:52:50 entrypoint.go:72: Resolving dependency Job placement-ks-user in namespace openstack failed: Job Job placement-ks-user in namespace openstack is not completed yet .
Entrypoint WARNING: 2020/04/21 11:52:52 entrypoint.go:72: Resolving dependency Job placement-ks-endpoints in namespace openstack failed: Job Job placement-ks-endpoints in namespace openstack is not completed yet .

kubectl -n openstack-ceph-shared get secrets openstack-ceph-keys

NAME                  TYPE     DATA   AGE
openstack-ceph-keys   Opaque   7      23h

Tungsten Fabric deployment prerequisites¶

Before you proceed with the actual Tungsten Fabric (TF) deployment, verify that your deployment meets the following prerequisites:

1. Your MOS OpenStack cluster is deployed as described in Deploy OpenStack with the compute-tf preset enabled.

2. Your MOS OpenStack cluster uses the correct value of features:neutron:tunnel_interface in the openstackdeployment object. The TF Operator will consume this value through the shared secret and use it as a network interface from the underlay network to create encapsulated tunnels with the tenant networks.

   Warning

   TF uses features:neutron:tunnel_interface to create the vhost0 virtual interface and transfers the IP configuration from the tunnel_interface to the virtual one. Therefore, plan this interface as a dedicated physical interface for TF overlay networks.

3. The Kubernetes nodes are labeled according to the TF node roles:

   Tungsten Fabric (TF) node roles¶

   Node role	Description	Kubernetes labels	Minimal count
   TF control plane	Hosts the TF control plane services such as database, messaging, api, svc, config.	tfconfig=enabled tfcontrol=enabled tfwebui=enabled tfconfigdb=enabled	3
   TF analytics	Hosts the TF analytics services.	tfanalytics=enabled tfanalyticsdb=enabled	3
   TF vRouter	Hosts the TF vRouter module and vRouter agent.	tfvrouter=enabled	Varies
   TF vRouter DPDK Technical Preview	Hosts the TF vRouter agent in DPDK mode.	tfvrouter-dpdk=enabled	Varies

   Note

   TF supports only Kubernetes OpenStack workloads. Therefore, you should label OpenStack compute nodes with the tfvrouter=enabled label.

   Note

   Do not specify the openstack-gateway=enabled and openvswitch=enabled labels for the MOS deployments with TF as a networking back end for OpenStack.

Deploy Tungsten Fabric¶

Deployment of Tungsten Fabric (TF) is managed by the tungstenfabric-operator Helm resource in a respective MOS ClusterRelease.

To deploy TF:

1. Verify that you have completed all prerequisite steps as described in Tungsten Fabric deployment prerequisites.

2. Create tungstenfabric.yaml with the TF resource configuration. For example:

   apiVersion: operator.tf.mirantis.com/v1alpha1
   kind: TFOperator
   metadata:
     name: openstack-tf
     namespace: tf
   spec:
     settings:
       orchestrator: openstack
   

3. Trigger the TF deployment:

   kubectl apply -f tungstenfabric.yaml
   

4. Verify that TF has been successfully deployed:

   kubectl get pods -n tf
   

   The successfully deployed TF services should appear in the Running status in the system response.

Access the Tungsten Fabric web UI¶

The Tungsten Fabric (TF) web UI allows for easy and fast TF resources configuration, monitoring, and debugging. You can access the TF web UI through either the Ingress service or the Kubernetes Service directly. TLS termination for the https protocol is performed through the Ingress service.

To access the TF web UI through Ingress:

1. Log in to a local machine running Ubuntu 18.04 where kubectl is installed.

2. Obtain and export kubeconfig of your managed cluster as described in Mirantis Container Cloud Operations Guide: Connect to a Container Cloud managed cluster.

3. Obtain the password of the Admin user:

   kubectl -n openstack exec -it $(kubectl -n openstack get pod -l application=keystone,component=client -o jsonpath='{.items[0].metadata.name}') -- env | grep PASS
   

4. Obtain the external IP address of the Ingress service:

   kubectl -n openstack get services ingress
   

   Example of system response:

   NAME      TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                                      AGE
   ingress   LoadBalancer   10.96.32.97   10.172.1.101   80:34234/TCP,443:34927/TCP,10246:33658/TCP   4h56m
   

   Note

   Do not use the EXTERNAL-IP value to directly access the TF web UI. Instead, use the FQDN from the list below.

5. Obtain the FQDN of tf-webui:

   Note

   The command below outputs all host names assigned to the TF web UI service. Use one of them.

   kubectl -n tf get ingress tf-webui -o custom-columns=HOSTS:.spec.rules[*].host
   

6. Configure DNS to access the TF web UI host as described in Configure DNS to access OpenStack.

7. Use your favorite browser to access the TF web UI at https://<FQDN-WEBUI>.

Troubleshoot the Tungsten Fabric deployment¶

This section provides the general debugging instructions for your Tungsten Fabric (TF) on Kubernetes deployment.

host tf-webui.it.just.works
# or
ping tf-webui.it.just.works
# or
dig host tf-webui.it.just.works

kubectl -n openstack get svc ingress -o custom-columns=HOSTS:.status.loadBalancer.ingress[*].ip