Mirantis OpenStack for Kubernetes Reference Architecture¶

Hardware requirements¶

This section provides hardware requirements for the Mirantis OpenStack for Kubernetes (MOS) cluster.

The MOS reference architecture includes the following node types:

• OpenStack control plane node and StackLight node

  Host OpenStack control plane services such as database, messaging, API, schedulers conductors, and L3 and L2 agents, as well as the StackLight components.

• Tenant gateway node

  Optional, hosts OpenStack gateway services including L2, L3, and DHCP agents. The tenant gateway nodes are combined with OpenStack control plane nodes. The strict requirement is a dedicated physical network (bond) for tenant network traffic.

• Tungsten Fabric control plane node

  Required only if Tungsten Fabric (TF) is enabled as a back end for the OpenStack networking. These nodes host the TF control plane services such as Cassandra database, messaging, API, control, and configuration services.

• Tungsten Fabric analytics node

  Required only if TF is enabled as a back end for the OpenStack networking. These nodes host the TF analytics services such as Cassandra, ZooKeeper and collector.

• Compute node

  Hosts OpenStack Compute services such as QEMU, L2 agents, and others.

• Infrastructure nodes

  Runs underlying Kubernetes cluster management services. The MOS reference configuration requires minimum three infrastructure nodes.

The table below specifies the hardware resources the MOS reference architecture recommends for each node type.

0

OpenStack gateway services can optionally be moved to separate nodes.

1(1,2)

TF control plane and analytics nodes can be combined with a respective addition of RAM, CPU, and disk space to the hardware hosts. Though, Mirantis does not recommend such configuration for production environments as the risk of the cluster downtime if one of the nodes unexpectedly fails increases.

2

• A Ceph cluster with 3 Ceph nodes does not provide hardware fault tolerance and is not eligible for recovery operations, such as a disk or an entire node replacement.

• A Ceph cluster uses the replication factor that equals 3. If the number of Ceph OSDs is less than 3, a Ceph cluster moves to the degraded state with the write operations restriction until the number of alive Ceph OSDs equals the replication factor again.

Components collocation¶

MOS uses Kubernetes labels to place components onto hosts. For the default locations of components, see Hardware requirements. Additionally, MOS supports component collocation. This is mostly useful for OpenStack compute and Ceph nodes. For component collocation, consider the following recommendations:

• When calculating hardware requirements for nodes, consider the requirements for all collocated components.

• When performing maintenance on a node with collocated components, execute the maintenance plan for all of them.

• When combining other services with the OpenStack compute host, verify that reserved_host_* has increased accordingly to the needs of collocated components by using node-specific overrides for the compute service.

Infrastructure requirements¶

This section lists the infrastructure requirements for the Mirantis OpenStack for Kubernetes (MOS) reference architecture.

OpenStack Operator¶

The OpenStack Operator component is a combination of the following entities:

OpenStack Controller¶

The OpenStack Controller runs in a set of containers in a pod in Kubernetes. The OpenStack Controller is deployed as a Deployment with 1 replica only. The failover is provided by Kubernetes that automatically restarts the failed containers in a pod.

However, given the recommendation to use a separate Kubernetes cluster for each OpenStack deployment, the controller in envisioned mode for operation and deployment will only manage a single OpenStackDeployment resource, making the proper HA much less of an issue.

The OpenStack Controller is written in Python using Kopf, as a Python framework to build Kubernetes operators, and Pykube, as a Kubernetes API client.

Using Kubernetes API, the controller subscribes to changes to resources of kind: OpenStackDeployment, and then reacts to these changes by creating, updating, or deleting appropriate resources in Kubernetes.

The basic child resources managed by the controller are of kind: HelmBundle. They are rendered from templates taking into account an appropriate values set from the main and features fields in the OpenStackDeployment resource.

Then, the common fields are merged to resulting data structures. Lastly, the services fields are merged providing the final and precise override for any value in any Helm release to be deployed or upgraded.

The constructed HelmBundle resources are then supplied to the Kubernetes API. The HelmBundle controller picks up the changes in these resources, similarly to the OpenStack Operator, and translates to the Helm releases, deploying, updating, or deleting native Kubernetes resources.

OpenStack Controller containers¶

Container	Description
osdpl	The core container that handles changes in the osdpl object.
helmbundle	The container that watches the helmbundle objects and reports their statuses to the osdpl object in status:children. See Status OsDpl elements for details.
health	The container that watches all Kubernetes native resources, such as Deployments, Daemonsets, Statefulsets, and reports their statuses to the osdpl object in status:health. See Status OsDpl elements for details.
secrets	The container that provides data exchange between different components such as Ceph.
node	The container that handles the node events.

OpenStackDeployment admission controller¶

The CustomResourceDefinition resource in Kubernetes uses the OpenAPI Specification version 2 to specify the schema of the resource defined. The Kubernetes API outright rejects the resources that do not pass this schema validation.

The language of the schema, however, is not expressive enough to define a specific validation logic that may be needed for a given resource. For this purpose, Kubernetes enables the extension of its API with Dynamic Admission Control.

For the OpenStackDeployment (OsDpl) CR the ValidatingAdmissionWebhook is a natural choice. It is deployed as part of OpenStack Controller by default and performs specific extended validations when an OsDpl CR is created or updated.

The inexhaustive list of additional validations includes:

• Deny the OpenStack version downgrade

• Deny the OpenStack version skip-level upgrade

• Deny the OpenStack master version deployment

• Deny upgrade to the OpenStack master version

• Deny upgrade if any part of an OsDpl CR specification changes along with the OpenStack version

Under specific circumstances, it may be viable to disable the admission controller, for example, when you attempt to deploy or upgrade to the master version of OpenStack.

Warning

Mirantis does not support MOS deployments performed without the OpenStackDeployment admission controller enabled. Disabling of the OpenStackDeployment admission controller is only allowed in staging non-production environments.

To disable the admission controller, ensure that the following structures and values are present in the openstack-controller HelmBundle resource:

apiVersion: lcm.mirantis.com/v1alpha1
kind: HelmBundle
metadata:
  name: openstack-operator
  namespace: osh-system
spec:
  releases:
  - name: openstack-operator
    values:
      admission:
        enabled: false

At that point, all safeguards except for those expressed by the CR definition are disabled.

OpenStackDeployment custom resource¶

The resource of kind OpenStackDeployment (OsDpl) is a custom resource (CR) defined by a resource of kind CustomResourceDefinition. This section is intended to provide a detailed overview of the OsDpl configuration including the definition of its main elements as well as the configuration of extra OpenStack services that do no belong to standard deployment profiles.

OsDpl standard configuration¶

The detailed information about schema of an OpenStackDeployment (OsDpl) custom resource can be obtained by running:

kubectl get crd openstackdeployments.lcm.mirantis.com -oyaml

The definition of a particular OpenStack deployment can be obtained by running:

kubectl -n openstack get osdpl -oyaml

For the detailed description of the OsDpl main elements, see the tables below:

• Main OsDpl elements

• Spec OsDpl elements

• Status OsDpl elements

Example of an OsDpl CR of minimum configuration:

apiVersion: lcm.mirantis.com/v1alpha1
kind: OpenStackDeployment
metadata:
  name: openstack-cluster
  namespace: openstack
spec:
  openstack_version: ussuri
  preset: compute
  size: tiny
  internal_domain_name: cluster.local
  public_domain_name: it.just.works
  features:
    ssl:
      public_endpoints:
        api_cert: |-
          # Update server certificate content
        api_key: |-
          # Update server private key content
        ca_cert: |-
          # Update CA certificate content
    neutron:
      tunnel_interface: ens3
      external_networks:
        - physnet: physnet1
          interface: veth-phy
          bridge: br-ex
          network_types:
           - flat
          vlan_ranges: null
          mtu: null
      floating_network:
        enabled: False
    nova:
      live_migration_interface: ens3
      images:
        backend: local

---

Main OsDpl elements¶

apiVersion¶

Specifies the version of the Kubernetes API that is used to create this object.

kind¶

Specifies the kind of the object.

metadata:name¶

Specifies the name of metadata. Should be set in compliance with the Kubernetes resource naming limitations.

metadata:namespace¶

Specifies the metadata namespace. While technically it is possible to deploy OpenStack on top of Kubernetes in other than openstack namespace, such configuration is not included in the MOS system integration test plans. Therefore, we do not recommend such scenario.

Warning

Both OpenStack and Kubernetes platforms provide resources to applications. When OpenStack is running on top of Kubernetes, Kubernetes is completely unaware of OpenStack-native workloads, such as virtual machines, for example.

For better results and stability, Mirantis recommends using a dedicated Kubernetes cluster for OpenStack, so that OpenStack and auxiliary services, Ceph, and StackLight are the only Kubernetes applications running in the cluster.

spec¶

Contains the data that defines the OpenStack deployment and configuration. It has both high-level and low-level sections.

The very basic values that must be provided include:

spec:
  openstack_version:
  preset:
  size:
  internal_domain_name:
  public_domain_name:

For the detailed description of the spec subelements, see Spec OsDpl elements

---

Spec OsDpl elements¶

openstack_version¶

Specifies the OpenStack release to deploy.

preset¶

String that specifies the name of the preset, a predefined configuration for the OpenStack cluster. A preset includes:

• A set of enabled services that includes virtualization, bare metal management, secret management, and others

• Major features provided by the services, such as VXLAN encapsulation of the tenant traffic

• Integration of services

Every supported deployment profile incorporates an OpenStack preset. Refer to Deployment profiles for the list of possible values.

size¶

String that specifies the size category for the OpenStack cluster. The size category defines the internal configuration of the cluster such as the number of replicas for service workers and timeouts, etc.

The list of supported sizes include:

• tiny - for approximately 10 OpenStack compute nodes

• small - for approximately 50 OpenStack compute nodes

• medium - for approximately 100 OpenStack compute nodes

internal_domain_name¶

Specifies the internal DNS name used inside the Kubernetes cluster on top of which the OpenStack cloud is deployed.

public_domain_name¶

Specifies the public DNS name for OpenStack services. This is a base DNS name that must be accessible and resolvable by API clients of your OpenStack cloud. It will be present in the OpenStack endpoints as presented by the OpenStack Identity service catalog.

The TLS certificates used by the OpenStack services (see below) must also be issued to this DNS name.

features¶

Contains the top-level collections of settings for the OpenStack deployment that potentially target several OpenStack services. The section where the customizations should take place.

features:services¶

Contains a list of extra OpenStack services to deploy. Extra OpenStack services are services that are not included into preset.

features:ssl¶

Contains the content of SSL/TLS certificates (server, key, CA bundle) used to enable a secure communication to public OpenStack API services.

These certificates must be issued to the DNS domain specified in the public_domain_name field.

features:neutron:tunnel_interface¶

Defines the name of the NIC device on the actual host that will be used for Neutron.

We recommend setting up your Kubernetes hosts in such a way that networking is configured identically on all of them, and names of the interfaces serving the same purpose or plugged into the same network are consistent across all physical nodes.

features:neutron:dns_servers¶

Defines the list of IPs of DNS servers that are accessible from virtual networks. Used as default DNS servers for VMs.

features:neutron:external_networks¶

Contains the data structure that defines external (provider) networks on top of which the Neutron networking will be created.

features:neutron:floating_network¶

If enabled, must contain the data structure defining the floating IP network that will be created for Neutron to provide external access to your Nova instances.

features:nova:live_migration_interface¶

Specifies the name of the NIC device on the actual host that will be used by Nova for the live migration of instances.

We recommend setting up your Kubernetes hosts in such a way that networking is configured identically on all of them, and names of the interfaces serving the same purpose or plugged into the same network are consistent across all physical nodes.

features:barbican:backends:vault¶

Specifies the object containing the Vault parameters to connect to Barbican. The list of supported options includes:

• enabled - boolean parameter indicating that the Vault back end is enabled

• approle_role_id - Vault app role ID

• approle_secret_id - secret ID created for the app role

• vault_url - URL of the Vault server

• use_ssl - enables the SSL encryption. Since MOS does not currently support the Vault SSL encryption, the use_ssl parameter should be set to false.

features:nova:images:backend¶

Defines the type of storage for Nova to use on the compute hosts for the images that back up the instances.

The list of supported options include:

• local - the local storage is used. The pros include faster operation, failure domain independency from the external storage. The cons include local space consumption and less performant and robust live migration with block migration.

• ceph - instance images are stored in a Ceph pool shared across all Nova hypervisors. The pros include faster image start, faster and more robust live migration. The cons include considerably slower IO performance, workload operations direct dependency on Ceph cluster availability and performance.

features:keystone:keycloak¶

Defines parameters to connect to the Keycloak identity provider.

features:keystone:domain_specific_configuration¶

Defines the domain-specific configuration and is useful for integration with LDAP. An example of OsDpl with LDAP integration, which will create a separate domain.with.ldap domain and configure it to use LDAP as an identity driver:

spec:
  features:
    keystone:
      domain_specific_configuration:
        enabled: true
        domains:
        - name: domain.with.ldap
          config:
            assignment:
              driver: keystone.assignment.backends.sql.Assignment
            identity:
              driver: ldap
            ldap:
              chase_referrals: false
              group_allow_create: false
              group_allow_delete: false
              group_allow_update: false
              group_desc_attribute: description
              group_id_attribute: cn
              group_member_attribute: member
              group_name_attribute: ou
              group_objectclass: groupOfNames
              page_size: 0
              password: XXXXXXXXX
              query_scope: sub
              suffix: dc=mydomain,dc=com
              url: ldap://ldap01.mydomain.com,ldap://ldap02.mydomain.com
              user: uid=openstack,ou=people,o=mydomain,dc=com
              user_allow_create: false
              user_allow_delete: false
              user_allow_update: false
              user_enabled_attribute: enabled
              user_enabled_default: false
              user_enabled_invert: true
              user_enabled_mask: 0
              user_id_attribute: uid
              user_mail_attribute: mail
              user_name_attribute: uid
              user_objectclass: inetOrgPerson

features:telemetry:mode¶

Specifies the Telemetry mode, which determines the permitted actions for the Telemetry services. The only supported value is autoscaling that allows for autoscaling of instances with HOT templates according to predefined conditions related to load of an instance and rules in the alarming service. The accounting mode support is being under development.

Caution

To enable the Telemetry mode, the corresponding services including the alarming, event, metering, and metric services should be specified in spec:services.

features:logging¶

Specifies the standard logging levels for OpenStack services that include the following, at increasing severity: TRACE, DEBUG, INFO, AUDIT, WARNING, ERROR, and CRITICAL. For example:

spec:
  features:
    logging:
      nova:
        level: DEBUG

features:horizon:themes¶

^{Available since MOS Ussuri Update}

Defines the list of custom OpenStack Dashboard themes. Content of the archive file with a theme depends on the level of customization and can include static files, Django templates, and other artifacts. For the details, refer to OpenStack official documentation: Customizing Horizon Themes.

spec:
  features:
    horizon:
      themes:
        - name: theme_name
          description: The brand new theme
          url: https://<path to .tgz file with the contents of custom theme>
          sha256summ: <SHA256 checksum of the archive above>

artifacts¶

A low-level section that defines the base URI prefixes for images and binary artifacts.

common¶

A low-level section that defines values that will be passed to all OpenStack (spec:common:openstack) or auxiliary (spec:common:infra) services Helm charts.

Structure example:

spec:
  artifacts:
  common:
    openstack:
      values:
    infra:
      values:

services¶

A section of the lowest level, enables the definition of specific values to pass to specific Helm charts on a one-by-one basis:

Warning

Mirantis does not recommend changing the default settings for spec:artifacts, spec:common, and spec:services elements. Customizations can compromise the OpenStack deployment update and upgrade processes.

---

Status OsDpl elements¶

The status element¶

Contains information about the current status of an OpenStack deployment, which cannot be changed by the user.

status:children¶

Specifies the current status of Helm releases that are managed by the OpenStack Operator. The possible values include:

• True - when the Helm chart is in the deployed state.

• False - when an error occurred during the Helm chart deployment.

An example of children output:

children:
  openstack-block-storage: true
  openstack-compute: true
  openstack-coordination: true
  ...

status:deployed¶

Shows an overall status of all Helm releases. Shows True when all children are in the deployed state.

status:fingerprint¶

Is the MD5 hash of the body:spec object. It is passed to all child HelmBundles to the body:metadata:lcm.mirantis.com/openstack-controller-fingerprint object. Also, it enables detecting the OsDpl resource version used when applying the child (HelmBundle) resource.

status:version¶

Contains the version of the OpenStack Operator that processes the OsDpl resource. And, similarly to fingerprint, it enables detecting the version of the OpenStack Operator that processed the child (HelmBundle) resource.

status:health¶

While status:children shows information about any deployed child HelmBundle resources, status:health shows the actual status of the deployed Kubernetes resources. The resources may be created as different Kubernetes objects, such as Deployments, Statefulsets, or DaemonSets. Possible values include:

• Ready - all pods from the resource are in the Ready state

• Unhealthy - not all pods from the resource are in the Ready state

• Progressing - Kubernetes resource is updating

• Unknown - other, unrecognized states

An example of the health output:

health:
  barbican:
    api:
      generation: 4
      status: Ready
    rabbitmq:
      generation: 1
      status: Ready
  cinder:
    api:
      generation: 4
      status: Ready
    backup:
      generation: 2
      status: Ready
    rabbitmq:
      generation: 1
      status: Ready
    ...
  ...

status:kopf¶

Contains the structure that is used by the Kopf library to store its internal data.

Integration with Identity Access Management (IAM)¶

Mirantis Container Cloud uses the Identity and access management (IAM) service for users and permission management. This section describes how you can integrate your OpenStack deployment with Keycloak through the OpenID connect.

To enable integration on the OpenStack side, define the following parameters in your openstackdeployment custom resource:

spec:
  features:
    keystone:
      keycloak:
        enabled: true
        url: <https://my-keycloak-instance>
        # optionally ssl cert validation might be disabled
        oidc:
           OIDCSSLValidateServer: false
           OIDCOAuthSSLValidateServer: false

The configuration above will trigger the creation of the os client in Keycloak. The role management and assignment should be configured separately on a particular deployment.

See also

• Mirantis Container Cloud Reference Architecture: Identity and access management

• Mirantis Container Cloud Operations Guide: IAM roles

Bare metal OsDpl configuration¶

The Bare metal (Ironic) service is an extra OpenStack service that can be deployed by the OpenStack Operator. This section provides the baremetal-specific configuration options of the OsDpl resource.

To install bare metal services, add the baremetal keyword to the spec:features:services list:

spec:
  features:
    services:
      - baremetal

Note

All bare metal services are scheduled to the nodes with the openstack-control-plane: enabled label.

Ironic agent deployment images¶

To provision a user image onto a bare metal server, Ironic boots a node with a ramdisk image. Depending on the node’s deploy interface and hardware, the ramdisk may require different drivers (agents). MOS provides tinyIPA-based ramdisk images and uses the direct deploy interface with the ipmitool power interface.

Example of agent_images configuration:

spec:
  features:
    ironic:
       agent_images:
         base_url: https://binary.mirantis.com/openstack/bin/ironic/tinyipa
         initramfs: tinyipa-stable-ussuri-20200617101427.gz
         kernel: tinyipa-stable-ussuri-20200617101427.vmlinuz

Since the bare metal nodes hardware may require additional drivers, you may need to build a deploy ramdisk for particular hardware. For more information, see Ironic Python Agent Builder. Be sure to create a ramdisk image with the version of Ironic Python Agent appropriate for your OpenStack release.

Networking¶

Ironic supports the flat and multitenancy networking modes.

The flat networking mode assumes that all bare metal nodes are pre-connected to a single network that cannot be changed during the virtual machine provisioning.

Example of the OsDpl resource illustrating the configuration for the flat network mode:

spec:
  features:
    services:
      - baremetal
    neutron:
      external_networks:
        - bridge: ironic-pxe
          interface: <baremetal-interface>
          network_types:
            - flat
          physnet: ironic
          vlan_ranges: null
    ironic:
       # The name of neutron network used for provisioning/cleaning.
       baremetal_network_name: ironic-provisioning
       networks:
         # Neutron baremetal network definition.
         baremetal:
           physnet: ironic
           name: ironic-provisioning
           network_type: flat
           external: true
           shared: true
           subnets:
             - name: baremetal-subnet
               range: 10.13.0.0/24
               pool_start: 10.13.0.100
               pool_end: 10.13.0.254
               gateway: 10.13.0.11
       # The name of interface where provision services like tftp and ironic-conductor
       # are bound.
       provisioning_interface: br-baremetal

---

The multitenancy network mode uses the neutron Ironic network interface to share physical connection information with Neutron. This information is handled by Neutron ML2 drivers when plugging a Neutron port to a specific network. MOS supports the networking-generic-switch Neutron ML2 driver out of the box.

Example of the OsDpl resource illustrating the configuration for the multitenancy network mode:

spec:
  features:
    services:
      - baremetal
    neutron:
      tunnel_interface: ens3
      external_networks:
        - physnet: physnet1
          interface: <physnet1-interface>
          bridge: br-ex
          network_types:
            - flat
          vlan_ranges: null
          mtu: null
        - physnet: ironic
          interface: <physnet-ironic-interface>
          bridge: ironic-pxe
          network_types:
            - vlan
          vlan_ranges: 1000:1099
    ironic:
      # The name of interface where provision services like tftp and ironic-conductor
      # are bound.
      provisioning_interface: <baremetal-interface>
      baremetal_network_name: ironic-provisioning
      networks:
        baremetal:
          physnet: ironic
          name: ironic-provisioning
          network_type: vlan
          segmentation_id: 1000
          external: true
          shared: false
          subnets:
            - name: baremetal-subnet
              range: 10.13.0.0/24
              pool_start: 10.13.0.100
              pool_end: 10.13.0.254
              gateway: 10.13.0.11

See also

OpenStack official documentation: networking-generic-switch

Node-specific settings¶

Caution

This feature is available starting from MOS Ussuri Update.

Depending on the use case, you may need to configure the same application components differently on different hosts. MOS enables you to easily perform the required configuration through node-specific overrides at the OpenStack Controller side.

The limitation of using the node-specific overrides is that they override only the configuration settings while other components, such as startup scripts and others, should be reconfigured as well.

Caution

The overrides have been implemented in a similar way to the OpenStack node and node label specific DaemonSet configurations. Though, the OpenStack Controller node-specific settings conflict with the upstream OpenStack node and node label specific DaemonSet configurations. Therefore, we do not recommend configuring node and node label overrides.

The node-specific settings are activated through the spec:nodes section of the OsDpl CR. The spec:nodes section contains the following subsections:

• features- implements overrides for a limited subset of fields and is constructed similarly to spec::features

• services - similarly to spec::services, enables you to override settings in general for the components running as DaemonSets.

Example configuration:

spec:
  nodes:
    <NODE-LABEL>::<NODE-LABEL-VALUE>:
    features:
      # Detailed information about features might be found at
      # openstack_controller/admission/validators/nodes/schema.yaml
    services:
      <service>:
        <chart>:
          <chart_daemonset_name>:
            values:
              # Any value from specific helm chart

See also

MOS Deployment Guide: Enable DPDK

OpenStack on Kubernetes architecture¶

OpenStack and auxiliary services are running as containers in the kind: Pod Kubernetes resources. All long-running services are governed by one of the ReplicationController-enabled Kubernetes resources, which include either kind: Deployment, kind: StatefulSet, or kind: DaemonSet.

The placement of the services is mostly governed by the Kubernetes node labels. The labels affecting the OpenStack services include:

• openstack-control-plane=enabled - the node hosting most of the OpenStack control plane services.

• openstack-compute-node=enabled - the node serving as a hypervisor for Nova. The virtual machines with tenants workloads are created there.

• openvswitch=enabled - the node hosting Neutron L2 agents and OpenvSwitch pods that manage L2 connection of the OpenStack networks.

• openstack-gateway=enabled - the node hosting Neutron L3, Metadata and DHCP agents, Octavia Health Manager, Worker and Housekeeping components.

spec:
  features:
    services:
    - object-storage

spec:
  features:
    barbican:
      backends:
        vault:
          enabled: true
          approle_role_id: <APPROLE_ROLE_ID>
          approle_secret_id: <APPROLE_SECRET_ID>
          vault_url: <VAULT_SERVER_URL>
          use_ssl: false

spec:
  features:
    services:
    - tempest

spec:
  features:
    services:
    - alarming
    - event
    - metering
    - metric

OpenStack database architecture¶

A complete setup of a MariaDB Galera cluster for OpenStack is illustrated in the following image:

MariaDB server pods are running a Galera multi-master cluster. Clients requests are forwarded by the Kubernetes mariadb service to the mariadb-server pod that has the primary label. Other pods from the mariadb-server StatefulSet have the backup label. Labels are managed by the mariadb-controller pod.

The MariaDB controller periodically checks the readiness of the mariadb-server pods and sets the primary label to it if the following requirements are met:

• The primary label has not already been set on the pod.

• The pod is in the ready state.

• The pod is not being terminated.

• The pod name has the lowest integer suffix among other ready pods in the StatefulSet. For example, between mariadb-server-1 and mariadb-server-2, the pod with the mariadb-server-1 name is preferred.

Otherwise, the MariaDB controller sets the backup label. This means that all SQL requests are passed only to one node while other two nodes are in the backup state and replicate the state from the primary node. The MariaDB clients are connecting to the mariadb service.

OpenStack and Ceph controllers integration¶

The integration between Ceph and OpenStack controllers is implemented through the shared Kubernetes openstack-ceph-shared namespace. Both controllers have access to this namespace to read and write the Kubernetes kind: Secret objects.

As Ceph is required and only supported back end for several OpenStack services, all necessary Ceph pools must be specified in the configuration of the kind: MiraCeph custom resource as part of the deployment. Once the Ceph cluster is deployed, the Ceph controller posts the information required by the OpenStack services to be properly configured as a kind: Secret object into the openstack-ceph-shared namespace. The OpenStack controller watches this namespace. Once the corresponding secret is created, the OpenStack controller transforms this secret to the data structures expected by the OpenStack-Helm charts. Even if an OpenStack installation is triggered at the same time as a Ceph cluster deployment, the OpenStack controller halts the deployment of the OpenStack services that depend on Ceph availability until the secret in the shared namespace is created by the Ceph controller.

For the configuration of Ceph RADOS Gateway as an OpenStack Object Storage, the reverse process takes place. The OpenStack controller waits for the OpenStack-Helm to create a secret with OpenStack Identity (Keystone) credentials that RADOS Gateway must use to validate the OpenStack Identity tokens, and posts it back to the same openstack-ceph-shared namespace in the format suitable for consumption by the Ceph controller. The Ceph controller then reads this secret and reconfigures RADOS Gateway accordingly.

OpenStack and StackLight integration¶

StackLight integration with OpenStack includes automatic discovery of RabbitMQ credentials for notifications and OpenStack credentials for OpenStack API metrics. For details, see the openstack.rabbitmq.credentialsConfig and openstack.telegraf.credentialsConfig parameters description in MOS Operations Guide: StackLight configuration parameters.

OpenStack and Tungsten Fabric integration¶

The levels of integration between OpenStack and Tungsten Fabric (TF) include:

• Controllers integration

• Services integration

Physical networks layout¶

The image below illustrates the recommended physical networks layout for a Mirantis OpenStack for Kubernetes (MOS) deployment with Ceph.

The image below illustrates the Ceph storage physical network layout.

Network types¶

When planning your OpenStack environment, consider what types of traffic your workloads generate and design your network accordingly. If you anticipate that certain types of traffic, such as storage replication, will likely consume a significant amount of network bandwidth, you may want to move that traffic to a dedicated network interface to avoid performance degradation.

A Mirantis OpenStack for Kubernetes (MOS) deployment typically requires the following networks:

The MOS deployment additionally requires the following networks:

The way of mapping of the logical networks described above to physical networks and interfaces on nodes depends on the cloud size and configuration. We recommend placing OpenStack networks on a dedicated physical interface (bond) that is not shared with storage and Kubernetes management network to minimize the influence on each other.

Performance optimization¶

To improve the goodput, we recommend that you enable jumbo frames where possible. The jumbo frames have to be enabled on the whole path of the packets traverse. If one of the network components cannot handle jumbo frames, the network path uses the smallest MTU.

To provide fault tolerance of a single NIC, we recommend using the link aggregation, such as bonding. The link aggregation is useful for linear scaling of bandwidth, load balancing, and fault protection. Depending on the hardware equipment, different types of bonds might be supported. Use the multi-chassis link aggregation as it provides fault tolerance at the device level. For example, MLAG on Arista equipment or vPC on Cisco equipment.

The Linux kernel supports the following bonding modes:

• active-backup

• balance-xor

• 802.3ad (LACP)

• balance-tlb

• balance-alb

Since LACP is the IEEE standard 802.3ad supported by the majority of network platforms, we recommend using this bonding mode.

Deployment architecture¶

Mirantis OpenStack for Kubernetes (MOS) deploys the StackLight stack as a release of a Helm chart that contains the helm-controller and HelmBundle custom resources. The StackLight HelmBundle consists of a set of Helm charts describing the StackLight components. Apart from the OpenStack-specific components below, StackLight also includes the components described in Mirantis Container Cloud Reference Architecture: Deployment architecture.

During the StackLight deployment, you can define the HA or non-HA StackLight architecture type. For details, see Mirantis Container Cloud Reference Architecture: StackLight database modes.

Monitored components¶

StackLight measures, analyzes, and reports in a timely manner about failures that may occur in the following Mirantis OpenStack for Kubernetes (MOS) components and their sub-components. Apart from the components below, StackLight also monitors the components listed in Mirantis Container Cloud Reference Architecture: Monitored components.

• libvirt

• Memcached

• MariaDB

• NTP

• OpenStack (Barbican, Cinder, Designate, Glance, Heat, Horizon, Ironic, Keystone, Neutron, Nova, Octavia)

• OpenStack SSL certificates

• Open vSwitch

• RabbitMQ

• Tungsten Fabric (Casandra, Kafka, Redis, ZooKeeper) ^{Available since MOS 21.1}

Elasticsearch and Prometheus storage sizing¶

During the deployment of a new cluster, you must specify the Elasticsearch retention time and Persistent Volume Claim (PVC) size, Prometheus PVC, retention time, and retention size. When configuring an existing cluster, you can only set Elasticsearch retention time, Prometheus retention time, and retention size.

The following table describes the recommendations for both Elasticsearch and Prometheus retention size and PVC size for a cluster with 34 nodes. Retention time depends on the space allocated for the data. To calculate the required retention time, use the {retention time} = {retention size} / {amount of data per day} formula.

Tungsten Fabric known limitations¶

This section contains a summary of the Tungsten Fabric upstream features and use cases not supported in MOS, features and use cases offered as Technology Preview in the current product release if any, and known limitations of Tungsten Fabric in integration with other product components.

Tungsten Fabric cluster overview¶

All services of Tungsten Fabric are delivered as separate containers, which are deployed by the Tungsten Fabric Operator (TFO). Each container has an INI-based configuration file that is available on the host system. The configuration file is generated automatically upon the container start and is based on environment variables provided by the TFO through Kubernetes ConfigMaps.

The main Tungsten Fabric containers run with the host network as DeploymentSet, without using the Kubernetes networking layer. The services listen directly on the host network interface.

The following diagram describes the minimum production installation of Tungsten Fabric with a Mirantis OpenStack for Kubernetes (MOS) deployment.

Tungsten Fabric components¶

This section describes the Tungsten Fabric services and their distribution across the Mirantis OpenStack for Kubernetes (MOS) deployment.

The Tungsten Fabric services run mostly as DaemonSets in a separate container for each service. The deployment and update processes are managed by the Tungsten Fabric operator. However, Kubernetes manages the probe checks and restart of broken containers.

The following tables describe the Tungsten Fabric services:

• Configuration and control services in Tungsten Fabric controller containers

• Analytics services in Tungsten Fabric analytics containers

• vRouter services on the OpenStack compute nodes

• Third-party services for Tungsten Fabric

• Tungsten Fabric plugin services on the OpenStack controller nodes

Tungsten Fabric operator¶

The Tungsten Fabric operator (TFO) is based on the operator SDK project. The operator SDK is a framework that uses the controller-runtime library to make writing operators easier by providing:

• High-level APIs and abstractions to write the operational logic more intuitively.

• Tools for scaffolding and code generation to bootstrap a new project fast.

• Extensions to cover common operator use cases.

The TFO deploys the following sub-operators. Each sub-operator handles a separate part of a TF deployment:

Besides the sub-operators that deploy TF services, TFO uses operators to deploy and maintain third-party services, such as different types of storage, cache, message system, and so on. The following table describes all third-party operators:

The following diagram illustrates a simplified TFO workflow:

Tungsten Fabric traffic flow¶

This section describes the types of traffic and traffic flow directions in a Mirantis OpenStack for Kubernetes (MOS) cluster.

User interface and API traffic

The following diagram illustrates all types of UI and API traffic in a MOS cluster, including the monitoring and OpenStack API traffic. The OpenStack Dashboard pod hosts Horizon and acts as a proxy for all other types of traffic. TLS termination is also performed for this type of traffic.

SDN traffic

SDN or Tungsten Fabric traffic goes through the overlay Data network and processes east-west and north-south traffic for applications that run in a MOS cluster. This network segment typically contains tenant networks as separate MPLS-over-GRE and MPLS-over-UDP tunnels. The traffic load depends on the workload.

The control traffic between the Tungsten Fabric controllers, edge routers, and vRouters uses the XMPP with TLS and iBGP protocols. Both protocols produce low traffic that does not affect MPLS over GRE and MPLS over UDP traffic. However, this traffic is critical and must be reliably delivered. Mirantis recommends configuring higher QoS for this type of traffic.

The following diagram displays both MPLS over GRE/MPLS over UDP and iBGP and XMPP traffic examples in a MOS cluster:

Tungsten Fabric vRouter¶

The Tungsten Fabric vRouter provides data forwarding to an OpenStack tenant instance and reports statistics to the Tungsten Fabric analytics service. The Tungsten Fabric vRouter is installed on all OpenStack compute nodes. Mirantis OpenStack for Kubernetes (MOS) supports the kernel-based deployment of the Tungsten Fabric vRouter.

The vRouter agent acts as a local control plane. Each Tungsten Fabric vRouter agent is connected to at least two Tungsten Fabric controllers in an active-active redundancy mode. The Tungsten Fabric vRouter agent is responsible for all networking-related functions including routing instances, routes, and so on.

The Tungsten Fabric vRouter uses different gateways for the control and data planes. For example, the Linux system gateway is located on the management network, and the Tungsten Fabric gateway is located on the data plane network.

The following diagram illustrates the Tungsten Fabric kernel vRouter setup by the TF operator:

On the diagram above, the following types of networks interfaces are used:

• eth0 - for the management (PXE) network (eth1 and eth2 are the slave interfaces of Bond0)

• Bond0.x - for the MKE control plane network

• Bond0.y - for the MKE data plane network

Tungsten Fabric integration with OpenStack Octavia¶

MOS ensures Octavia with Tungsten Fabric integration by OpenStack Octavia Driver with Tungsten Fabric HAProxy as a back end.

Octavia Tungsten Fabric Driver supports creation, update, and deletion operations with the following entities:

• Load balancers

  Note

  For a load balancer creation operation, the driver supports only the vip-subnet-id argument, the vip-network-id argument is not supported.

• Listeners

• Pools

• Health monitors

Octavia Tungsten Fabric Driver does not support the following functionality:

• L7 load balancing capabilities, such as L7 policies, L7 rules, and others

• Setting specific availability zones for load balancers and their resources

• Using of the UDP protocol

• Operations with Octavia quotas

• Operations with Octavia flavors