MOS Reference Architecture latest documentation

Mirantis OpenStack for Kubernetes Reference Architecture


About this documentation set

This documentation provides information on how to deploy and operate a Mirantis OpenStack for Kubernetes (MOS) environment. The documentation is intended to help operators to understand the core concepts of the product. The documentation provides sufficient information to deploy and operate the solution.

The information provided in this documentation set is being constantly improved and amended based on the feedback and kind requests from the consumers of MOS.

The following table lists the guides included in the documentation set you are reading:

Guides list



MOS Reference Architecture

Learn the fundamentals of MOS reference architecture to appropriately plan your deployment

MOS Deployment Guide

Deploy a MOS environment of a preferred configuration using supported deployment profiles tailored to the demands of specific business cases

MOS Operations Guide

Operate your MOS environment

MOS Release notes

Learn about new features and bug fixes in the current MOS version

The MOS documentation home page contains references to all guides included in this documentation set. For your convenience, we provide all guides in HTML (default), single-page HTML, PDF, and ePUB formats. To use the preferred format of a guide, select the required option from the Formats menu next to the guide title.

Intended audience

This documentation is intended for engineers who have the basic knowledge of Linux, virtualization and containerization technologies, Kubernetes API and CLI, Helm and Helm charts, Mirantis Kubernetes Engine (MKE), and OpenStack.

Technology Preview support scope

This documentation set includes description of the Technology Preview features. A Technology Preview feature provide early access to upcoming product innovations, allowing customers to experience the functionality and provide feedback during the development process. Technology Preview features may be privately or publicly available and neither are intended for production use. While Mirantis will provide support for such features through official channels, normal Service Level Agreements do not apply. Customers may be supported by Mirantis Customer Support or Mirantis Field Support.

As Mirantis considers making future iterations of Technology Preview features generally available, we will attempt to resolve any issues that customers experience when using these features.

During the development of a Technology Preview feature, additional components may become available to the public for testing. Because Technology Preview features are being under development, Mirantis cannot guarantee the stability of such features. As a result, if you are using Technology Preview features, you may not be able to seamlessly upgrade to subsequent releases of that feature. Mirantis makes no guarantees that Technology Preview features will be graduated to a generally available product release.

The Mirantis Customer Success Organization may create bug reports on behalf of support cases filed by customers. These bug reports will then be forwarded to the Mirantis Product team for possible inclusion in a future release.

Documentation history

The following table contains the released revision of the documentation set you are reading:

Release date


November 05, 2020

MOS GA release

December 23, 2020

MOS GA Update release

March 01, 2021

MOS 21.1


This documentation set uses the following conventions in the HTML format:

Documentation conventions



boldface font

Inline CLI tools and commands, titles of the procedures and system response examples, table titles

monospaced font

Files names and paths, Helm charts parameters and their values, names of packages, nodes names and labels, and so on

italic font

Information that distinguishes some concept or term


External links and cross-references, footnotes

Main menu > menu item

GUI elements that include any part of interactive user interface and menu navigation


Some extra, brief information


The Note block

Messages of a generic meaning that may be useful for the user


The Caution block

Information that prevents a user from mistakes and undesirable consequences when following the procedures


The Warning block

Messages that include details that can be easily missed, but should not be ignored by the user and are valuable before proceeding

See also

The See also block

List of references that may be helpful for understanding of some related tools, concepts, and so on

Learn more

The Learn more block

Used in the Release Notes to wrap a list of internal references to the reference architecture, deployment and operation procedures specific to a newly implemented product feature


Mirantis OpenStack for Kubernetes (MOS) is a virtualization platform that provides an infrastructure for cloud-ready applications, in combination with reliability and full control over the data.

MOS alloys OpenStack, an open-source cloud infrastructure software, with application management techniques used in Kubernetes ecosystem that include container isolation, state enforcement, declarative definition of deployments, and others.

MOS integrates with Mirantis Container Cloud to rely on its capabilities for bare-metal infrastructure provisioning, Kubernetes cluster management, and continuous delivery of the stack components.

MOS simplifies the work of a cloud operator by automating all major cloud life cycle management routines including cluster updates and upgrades.

Deployment profiles

A Mirantis OpenStack for Kubernetes (MOS) deployment profile is a thoroughly tested and officially supported reference architecture that is guaranteed to work at a specific scale and is tailored to the demands of a specific business case, such as generic IaaS cloud, Network Function Virtualisation infrastructure, Edge Computing, and others.

A deployment profile is defined as a combination of:

  • Services and features the cloud offers to its users.

  • Non-functional characteristics that users and operators should expect when running the profile on top of a reference hardware configuration. Including, but not limited to:

    • Performance characteristics, such as an average network throughput between VMs in the same virtual network.

    • Reliability characteristics, such as the cloud API error response rate when recovering a failed controller node.

    • Scalability characteristics, such as the total amount of virtual routers tenants can run simultaneously.

  • Hardware requirements - the specification of physical servers, and networking equipment required to run the profile in production.

  • Deployment parameters that an operator for the cloud can tweak within a certain range without being afraid of breaking the cloud or losing support.

In addition, the following items may be included in a definition:

  • Compliance-driven technical requirements, such as TLS encryption of all external API endpoints.

  • Foundation-level software components, such as Tungsten Fabric or Open vSwitch as a back end for the networking service.


Mirantis reserves the right to revise the technical implementation of any profile at will while preserving its definition - the functional and non-functional characteristics that operators and users are known to rely on.

MOS supports a huge list of different deployment profiles to address a wide variety of business tasks. The table below includes the profiles for the most common use cases.


Some components of a MOS cluster are mandatory and are being installed during the managed cluster deployment by MCC regardless of the deployment profile in use. StackLight is one of the cluster components that are enabled by default. See MCC Operations Guide for details.

Supported deployment profiles


OpenStackDeployment CR Preset


Cloud Provider Infrastructure (CPI)


Provides the core set of the services an IaaS vendor would need including some extra functionality. The profile is designed to support up 50-70 compute nodes and a reasonable number of storage nodes. 0

The core set of services provided by the profile includes:

  • Compute (Nova with Ceph as a back end for root and ephemeral storage)

  • Images (Glance)

  • Networking (Neutron with Open vSwitch as a back end)

  • Identity (Keystone)

  • Block Storage (Cinder with Ceph as a back end)

  • Orchestration (Heat)

  • Load balancing (Octavia)

  • DNS (Designate)

  • Secret Management (Barbican with SimpleCrypto back end)

  • Web front end (Horizon)

  • Bare metal provisioning (Ironic) 1 2

  • Telemetry (aodh, Panko, Ceilometer, and Gnocchi) 3

CPI with Tungsten Fabric


A variation of the CPI profile 1 with Tugsten Fabric as a back end for networking.


The supported node count is approximate and may vary depending on the hardware, cloud configuration, and planned workload.


Ironic is an optional component for the CPI profile. See Bare metal OsDpl configuration for details.


Ironic is not supported for the CPI with Tungsten Fabric profile. See Tungsten Fabric known limitations for details.


Telemetry services are optional components and should be enabled together through the list of services to be deployed in the OpenStackDeployment CR as described in Deployment Guide: Deploy an OpenStack cluster.

Components overview

Mirantis OpenStack for Kubernetes (MOS) includes the following key design elements:

  • HelmBundle Operator

    The HelmBundle Operator is the realization of the Kubernetes Operator pattern that provides a Kubernetes custom resource of the HelmBundle kind and code running inside a pod in Kubernetes. This code handles changes, such as creation, update, and deletion, in the Kubernetes resources of this kind by deploying, updating, and deleting groups of Helm releases from specified Helm charts with specified values.

  • OpenStack

    The OpenStack platform manages virtual infrastructure resources, including virtual servers, storage devices, networks, and networking services, such as load balancers, as well as provides management functions to the tenant users.

    Various OpenStack services are running as pods in Kubernetes and are represented as appropriate native Kubernetes resources, such as Deployments, StatefulSets, and DaemonSets.

    For a simple, resilient, and flexible deployment of OpenStack and related services on top of a Kubernetes cluster, MOS uses OpenStack-Helm that provides a required collection of the Helm charts.

    Also, MOS uses OpenStack Operator as the realization of the Kubernetes Operator pattern. The OpenStack Operator provides a custom Kubernetes resource of the OpenStackDeployment kind and code running inside a pod in Kubernetes. This code handles changes such as creation, update, and deletion in the Kubernetes resources of this kind by deploying, updating, and deleting groups of the HelmBundle resources handled by the HelmBundle Operator to manage OpenStack in Kubernetes through the OpenStack-Helm charts.

  • Ceph

    Ceph is a distributed storage platform that provides storage resources, such as objects and virtual block devices, to virtual and physical infrastructure.

    MOS uses Rook as the implementation of the Kubernetes Operator pattern that manages resources of the CephCluster kind to deploy and manage Ceph services as pods on top of Kubernetes to provide Ceph-based storage to the consumers, which include OpenStack services, such as Volume and Image services, and underlying Kubernetes through Ceph CSI (Container Storage Interface).

    The Ceph controller is the implementation of the Kubernetes Operator pattern, that manages resources of the MiraCeph kind to simplify management of the Rook-based Ceph clusters.

  • StackLight Logging, Monitoring, and Alerting

    The StackLight component is responsible for collection, analysis, and visualization of critical monitoring data from physical and virtual infrastructure, as well as alerting and error notifications through a configured communication system, such as email. StackLight includes the following key sub-components:

    • Prometheus

    • Elasticsearch

    • Fluentd

    • Kibana


Hardware requirements

This section provides hardware requirements for the Mirantis OpenStack for Kubernetes (MOS) cluster.


A MOS managed cluster is deployed by a Mirantis Container Cloud baremetal-based management cluster. For the hardware requirements for this kind of management clusters, see Mirantis Container Cloud Reference Architecture: Reference hardware configuration.

The MOS reference architecture includes the following node types:

  • OpenStack control plane node and StackLight node

    Host OpenStack control plane services such as database, messaging, API, schedulers conductors, and L3 and L2 agents, as well as the StackLight components.

  • Tenant gateway node

    Optional, hosts OpenStack gateway services including L2, L3, and DHCP agents. The tenant gateway nodes are combined with OpenStack control plane nodes. The strict requirement is a dedicated physical network (bond) for tenant network traffic.

  • Tungsten Fabric control plane node

    Required only if Tungsten Fabric (TF) is enabled as a back end for the OpenStack networking. These nodes host the TF control plane services such as Cassandra database, messaging, API, control, and configuration services.

  • Tungsten Fabric analytics node

    Required only if TF is enabled as a back end for the OpenStack networking. These nodes host the TF analytics services such as Cassandra, ZooKeeper and collector.

  • Compute node

    Hosts OpenStack Compute services such as QEMU, L2 agents, and others.

  • Infrastructure nodes

    Runs underlying Kubernetes cluster management services. The MOS reference configuration requires minimum three infrastructure nodes.

The table below specifies the hardware resources the MOS reference architecture recommends for each node type.

Hardware requirements

Node type

# of servers

CPU cores # per server

RAM per server, GB

Disk space per server, GB

NICs # per server

OpenStack control plane, gateway 0, and StackLight nodes




1 SSD x 500
2 SSD x 1000 3


Tenant gateway (optional)




1 SSD x 500


Tungsten Fabric control plane nodes 1




1 SSD x 500


Tungsten Fabric analytics nodes 1




1 SSD x 1000


Compute node

3 (varies)



1 SSD x 500 4


Infrastructure node (Kubernetes cluster management)

3 5



1 SSD x 500


Infrastructure node (Ceph) 2




1 SSD x 500
2 HDDs x 2000



The exact hardware specifications and number of nodes depend on a cloud configuration and scaling needs.


OpenStack gateway services can optionally be moved to separate nodes.


TF control plane and analytics nodes can be combined with a respective addition of RAM, CPU, and disk space to the hardware hosts. Though, Mirantis does not recommend such configuration for production environments as the risk of the cluster downtime if one of the nodes unexpectedly fails increases.

  • A Ceph cluster with 3 Ceph nodes does not provide hardware fault tolerance and is not eligible for recovery operations, such as a disk or an entire node replacement.

  • A Ceph cluster uses the replication factor that equals 3. If the number of Ceph OSDs is less than 3, a Ceph cluster moves to the degraded state with the write operations restriction until the number of alive Ceph OSDs equals the replication factor again.

  • 1 SSD x 500 for operating system

  • 1 SSD x 1000 for OpenStack LVP

  • 1 SSD x 1000 for StackLight LVP


When Nova is used with local folders, additional capacity is required depending on the VM images size.


For nodes hardware requirements, refer to Container Cloud Reference Architecture: Managed cluster hardware configuration.


If you are looking to try MOS and do not have much hardware at your disposal, you can deploy it in a virtual environment, for example, on top of another OpenStack cloud using the sample Heat templates.

Please mind, the tooling is provided for reference only and is not a part of the product itself. Mirantis does not guarantee its interoperability with the latest MOS version.

Components collocation


This feature is available as technical preview. Use such configuration for testing and evaluation purposes only.

MOS uses Kubernetes labels to place components onto hosts. For the default locations of components, see Hardware requirements. Additionally, MOS supports component collocation. This is mostly useful for OpenStack compute and Ceph nodes. For component collocation, consider the following recommendations:

  • When calculating hardware requirements for nodes, consider the requirements for all collocated components.

  • When performing maintenance on a node with collocated components, execute the maintenance plan for all of them.

  • When combining other services with the OpenStack compute host, verify that reserved_host_* has increased accordingly to the needs of collocated components by using node-specific overrides for the compute service.

Infrastructure requirements

This section lists the infrastructure requirements for the Mirantis OpenStack for Kubernetes (MOS) reference architecture.

Infrastructure requirements




MetalLB exposes external IP addresses to access applications in a Kubernetes cluster.


The Kubernetes Ingress NGINX controller is used to expose OpenStack services outside of a Kubernetes deployment. Access to the Ingress services is allowed only by its FQDN. Therefore, DNS is a mandatory infrastructure service for an OpenStack on Kubernetes deployment.


OpenStack Operator

The OpenStack Operator component is a combination of the following entities:

OpenStack Controller

The OpenStack Controller runs in a set of containers in a pod in Kubernetes. The OpenStack Controller is deployed as a Deployment with 1 replica only. The failover is provided by Kubernetes that automatically restarts the failed containers in a pod.

However, given the recommendation to use a separate Kubernetes cluster for each OpenStack deployment, the controller in envisioned mode for operation and deployment will only manage a single OpenStackDeployment resource, making the proper HA much less of an issue.

The OpenStack Controller is written in Python using Kopf, as a Python framework to build Kubernetes operators, and Pykube, as a Kubernetes API client.

Using Kubernetes API, the controller subscribes to changes to resources of kind: OpenStackDeployment, and then reacts to these changes by creating, updating, or deleting appropriate resources in Kubernetes.

The basic child resources managed by the controller are of kind: HelmBundle. They are rendered from templates taking into account an appropriate values set from the main and features fields in the OpenStackDeployment resource.

Then, the common fields are merged to resulting data structures. Lastly, the services fields are merged providing the final and precise override for any value in any Helm release to be deployed or upgraded.

The constructed HelmBundle resources are then supplied to the Kubernetes API. The HelmBundle controller picks up the changes in these resources, similarly to the OpenStack Operator, and translates to the Helm releases, deploying, updating, or deleting native Kubernetes resources.

OpenStack Controller containers




The core container that handles changes in the osdpl object.


The container that watches the helmbundle objects and reports their statuses to the osdpl object in status:children. See Status OsDpl elements for details.


The container that watches all Kubernetes native resources, such as Deployments, Daemonsets, Statefulsets, and reports their statuses to the osdpl object in status:health. See Status OsDpl elements for details.


The container that provides data exchange between different components such as Ceph.


The container that handles the node events.


OpenStackDeployment admission controller

The CustomResourceDefinition resource in Kubernetes uses the OpenAPI Specification version 2 to specify the schema of the resource defined. The Kubernetes API outright rejects the resources that do not pass this schema validation.

The language of the schema, however, is not expressive enough to define a specific validation logic that may be needed for a given resource. For this purpose, Kubernetes enables the extension of its API with Dynamic Admission Control.

For the OpenStackDeployment (OsDpl) CR the ValidatingAdmissionWebhook is a natural choice. It is deployed as part of OpenStack Controller by default and performs specific extended validations when an OsDpl CR is created or updated.

The inexhaustive list of additional validations includes:

  • Deny the OpenStack version downgrade

  • Deny the OpenStack version skip-level upgrade

  • Deny the OpenStack master version deployment

  • Deny upgrade to the OpenStack master version

  • Deny upgrade if any part of an OsDpl CR specification changes along with the OpenStack version

Under specific circumstances, it may be viable to disable the admission controller, for example, when you attempt to deploy or upgrade to the master version of OpenStack.


Mirantis does not support MOS deployments performed without the OpenStackDeployment admission controller enabled. Disabling of the OpenStackDeployment admission controller is only allowed in staging non-production environments.

To disable the admission controller, ensure that the following structures and values are present in the openstack-controller HelmBundle resource:

kind: HelmBundle
  name: openstack-operator
  namespace: osh-system
  - name: openstack-operator
        enabled: false

At that point, all safeguards except for those expressed by the CR definition are disabled.

OpenStackDeployment custom resource

The resource of kind OpenStackDeployment (OsDpl) is a custom resource (CR) defined by a resource of kind CustomResourceDefinition. This section is intended to provide a detailed overview of the OsDpl configuration including the definition of its main elements as well as the configuration of extra OpenStack services that do no belong to standard deployment profiles.

OsDpl standard configuration

The detailed information about schema of an OpenStackDeployment (OsDpl) custom resource can be obtained by running:

kubectl get crd -oyaml

The definition of a particular OpenStack deployment can be obtained by running:

kubectl -n openstack get osdpl -oyaml

For the detailed description of the OsDpl main elements, see the tables below:

Example of an OsDpl CR of minimum configuration:

kind: OpenStackDeployment
  name: openstack-cluster
  namespace: openstack
  openstack_version: ussuri
  preset: compute
  size: tiny
  internal_domain_name: cluster.local
        api_cert: |-
          # Update server certificate content
        api_key: |-
          # Update server private key content
        ca_cert: |-
          # Update CA certificate content
      tunnel_interface: ens3
        - physnet: physnet1
          interface: veth-phy
          bridge: br-ex
           - flat
          vlan_ranges: null
          mtu: null
        enabled: False
      live_migration_interface: ens3
        backend: local

Main OsDpl elements

Specifies the version of the Kubernetes API that is used to create this object.


Specifies the kind of the object.


Specifies the name of metadata. Should be set in compliance with the Kubernetes resource naming limitations.


Specifies the metadata namespace. While technically it is possible to deploy OpenStack on top of Kubernetes in other than openstack namespace, such configuration is not included in the MOS system integration test plans. Therefore, we do not recommend such scenario.


Both OpenStack and Kubernetes platforms provide resources to applications. When OpenStack is running on top of Kubernetes, Kubernetes is completely unaware of OpenStack-native workloads, such as virtual machines, for example.

For better results and stability, Mirantis recommends using a dedicated Kubernetes cluster for OpenStack, so that OpenStack and auxiliary services, Ceph, and StackLight are the only Kubernetes applications running in the cluster.


Contains the data that defines the OpenStack deployment and configuration. It has both high-level and low-level sections.

The very basic values that must be provided include:


For the detailed description of the spec subelements, see Spec OsDpl elements.

Spec OsDpl elements

Specifies the OpenStack release to deploy.


String that specifies the name of the preset, a predefined configuration for the OpenStack cluster. A preset includes:

  • A set of enabled services that includes virtualization, bare metal management, secret management, and others

  • Major features provided by the services, such as VXLAN encapsulation of the tenant traffic

  • Integration of services

Every supported deployment profile incorporates an OpenStack preset. Refer to Deployment profiles for the list of possible values.


String that specifies the size category for the OpenStack cluster. The size category defines the internal configuration of the cluster such as the number of replicas for service workers and timeouts, etc.

The list of supported sizes include:

  • tiny - for approximately 10 OpenStack compute nodes

  • small - for approximately 50 OpenStack compute nodes

  • medium - for approximately 100 OpenStack compute nodes


Specifies the internal DNS name used inside the Kubernetes cluster on top of which the OpenStack cloud is deployed.


Specifies the public DNS name for OpenStack services. This is a base DNS name that must be accessible and resolvable by API clients of your OpenStack cloud. It will be present in the OpenStack endpoints as presented by the OpenStack Identity service catalog.

The TLS certificates used by the OpenStack services (see below) must also be issued to this DNS name.


Contains the top-level collections of settings for the OpenStack deployment that potentially target several OpenStack services. The section where the customizations should take place.


Contains a list of extra OpenStack services to deploy. Extra OpenStack services are services that are not included into preset.


Available since MOS Ussuri Update

Enables the object storage and provides a RADOS Gateway Swift API that is compatible with the OpenStack Swift API. To enable the service, add object-storage to the service list:

    - object-storage

To create the RADOS Gateway pool in Ceph, see Container Cloud Operations Guide: Enable Ceph RGW Object Storage.


Available since MOS 21.2, Technical Preview

Enables Masakari, the OpenStack service that ensures high availability of instances running on a host. To enable the service, add instance-ha to the service list:

    - instance-ha

Enables tests against a deployed OpenStack cloud:

    - tempest

Contains the content of SSL/TLS certificates (server, key, CA bundle) used to enable a secure communication to public OpenStack API services.

These certificates must be issued to the DNS domain specified in the public_domain_name field.


Defines the name of the NIC device on the actual host that will be used for Neutron.

We recommend setting up your Kubernetes hosts in such a way that networking is configured identically on all of them, and names of the interfaces serving the same purpose or plugged into the same network are consistent across all physical nodes.


Defines the list of IPs of DNS servers that are accessible from virtual networks. Used as default DNS servers for VMs.


Contains the data structure that defines external (provider) networks on top of which the Neutron networking will be created.


If enabled, must contain the data structure defining the floating IP network that will be created for Neutron to provide external access to your Nova instances.


Specifies the name of the NIC device on the actual host that will be used by Nova for the live migration of instances.

We recommend setting up your Kubernetes hosts in such a way that networking is configured identically on all of them, and names of the interfaces serving the same purpose or plugged into the same network are consistent across all physical nodes.


Specifies the object containing the Vault parameters to connect to Barbican. The list of supported options includes:

  • enabled - boolean parameter indicating that the Vault back end is enabled

  • approle_role_id - Vault app role ID

  • approle_secret_id - secret ID created for the app role

  • vault_url - URL of the Vault server

  • use_ssl - enables the SSL encryption. Since MOS does not currently support the Vault SSL encryption, the use_ssl parameter should be set to false.

If the Vault back end is used, configure it properly using the following parameters:

          enabled: true
          approle_role_id: <APPROLE_ROLE_ID>
          approle_secret_id: <APPROLE_SECRET_ID>
          vault_url: <VAULT_SERVER_URL>
          use_ssl: false


Since MOS does not currently support the Vault SSL encryption, set the use_ssl parameter to false.


Defines the type of storage for Nova to use on the compute hosts for the images that back up the instances.

The list of supported options include:

  • local - the local storage is used. The pros include faster operation, failure domain independency from the external storage. The cons include local space consumption and less performant and robust live migration with block migration.

  • ceph - instance images are stored in a Ceph pool shared across all Nova hypervisors. The pros include faster image start, faster and more robust live migration. The cons include considerably slower IO performance, workload operations direct dependency on Ceph cluster availability and performance.

  • lvm Available since MOS 21.2, Technical Preview - instance images and ephemeral images are stored on a local Logical Volume. If specified, features:nova:images:lvm:volume_group must be set to an available LVM Volume Group, by default, nova-vol. For details, see MOS Deployment Guide: Enable LVM ephemeral storage.


Defines parameters to connect to the Keycloak identity provider. For details, see Integration with Identity Access Management (IAM).


Defines the domain-specific configuration and is useful for integration with LDAP. An example of OsDpl with LDAP integration, which will create a separate domain.with.ldap domain and configure it to use LDAP as an identity driver:

        enabled: true
        - name: domain.with.ldap
          enabled: true
              driver: keystone.assignment.backends.sql.Assignment
              driver: ldap
              chase_referrals: false
              group_allow_create: false
              group_allow_delete: false
              group_allow_update: false
              group_desc_attribute: description
              group_id_attribute: cn
              group_member_attribute: member
              group_name_attribute: ou
              group_objectclass: groupOfNames
              page_size: 0
              password: XXXXXXXXX
              query_scope: sub
              suffix: dc=mydomain,dc=com
              url: ldap://,ldap://
              user: uid=openstack,ou=people,o=mydomain,dc=com
              user_allow_create: false
              user_allow_delete: false
              user_allow_update: false
              user_enabled_attribute: enabled
              user_enabled_default: false
              user_enabled_invert: true
              user_enabled_mask: 0
              user_id_attribute: uid
              user_mail_attribute: mail
              user_name_attribute: uid
              user_objectclass: inetOrgPerson

Specifies the Telemetry mode, which determines the permitted actions for the Telemetry services. The only supported value is autoscaling that allows for autoscaling of instances with HOT templates according to predefined conditions related to load of an instance and rules in the alarming service. The accounting mode support is being under development.


To enable the Telemetry mode, the corresponding services including the alarming, event, metering, and metric services should be specified in spec:features:services:

    - alarming
    - event
    - metering
    - metric

Specifies the standard logging levels for OpenStack services that include the following, at increasing severity: TRACE, DEBUG, INFO, AUDIT, WARNING, ERROR, and CRITICAL. For example:

        level: DEBUG

Available since MOS Ussuri Update

Defines the list of custom OpenStack Dashboard themes. Content of the archive file with a theme depends on the level of customization and can include static files, Django templates, and other artifacts. For the details, refer to OpenStack official documentation: Customizing Horizon Themes.

        - name: theme_name
          description: The brand new theme
          url: https://<path to .tgz file with the contents of custom theme>
          sha256summ: <SHA256 checksum of the archive above>

A low-level section that defines the base URI prefixes for images and binary artifacts.


A low-level section that defines values that will be passed to all OpenStack (spec:common:openstack) or auxiliary (spec:common:infra) services Helm charts.

Structure example:


A section of the lowest level, enables the definition of specific values to pass to specific Helm charts on a one-by-one basis:


Mirantis does not recommend changing the default settings for spec:artifacts, spec:common, and spec:services elements. Customizations can compromise the OpenStack deployment update and upgrade processes.

Status OsDpl elements
The status element

Contains information about the current status of an OpenStack deployment, which cannot be changed by the user.


Specifies the current status of Helm releases that are managed by the OpenStack Operator. The possible values include:

  • True - when the Helm chart is in the deployed state.

  • False - when an error occurred during the Helm chart deployment.

An example of children output:

  openstack-block-storage: true
  openstack-compute: true
  openstack-coordination: true

Shows an overall status of all Helm releases. Shows True when all children are in the deployed state.


Is the MD5 hash of the body:spec object. It is passed to all child HelmBundles to the object. Also, it enables detecting the OsDpl resource version used when applying the child (HelmBundle) resource.


Contains the version of the OpenStack Operator that processes the OsDpl resource. And, similarly to fingerprint, it enables detecting the version of the OpenStack Operator that processed the child (HelmBundle) resource.


While status:children shows information about any deployed child HelmBundle resources, status:health shows the actual status of the deployed Kubernetes resources. The resources may be created as different Kubernetes objects, such as Deployments, Statefulsets, or DaemonSets. Possible values include:

  • Ready - all pods from the resource are in the Ready state

  • Unhealthy - not all pods from the resource are in the Ready state

  • Progressing - Kubernetes resource is updating

  • Unknown - other, unrecognized states

An example of the health output:

      generation: 4
      status: Ready
      generation: 1
      status: Ready
      generation: 4
      status: Ready
      generation: 2
      status: Ready
      generation: 1
      status: Ready

Contains the structure that is used by the Kopf library to store its internal data.

Integration with Identity Access Management (IAM)

Mirantis Container Cloud uses the Identity and access management (IAM) service for users and permission management. This section describes how you can integrate your OpenStack deployment with Keycloak through the OpenID connect.

To enable integration on the OpenStack side, define the following parameters in your openstackdeployment custom resource:

        enabled: true
        url: <https://my-keycloak-instance>
        # optionally ssl cert validation might be disabled
           OIDCSSLValidateServer: false
           OIDCOAuthSSLValidateServer: false

The configuration above will trigger the creation of the os client in Keycloak. The role management and assignment should be configured separately on a particular deployment.

Bare metal OsDpl configuration

The Bare metal (Ironic) service is an extra OpenStack service that can be deployed by the OpenStack Operator. This section provides the baremetal-specific configuration options of the OsDpl resource.

To install bare metal services, add the baremetal keyword to the spec:features:services list:

      - baremetal


All bare metal services are scheduled to the nodes with the openstack-control-plane: enabled label.

Ironic agent deployment images

To provision a user image onto a bare metal server, Ironic boots a node with a ramdisk image. Depending on the node’s deploy interface and hardware, the ramdisk may require different drivers (agents). MOS provides tinyIPA-based ramdisk images and uses the direct deploy interface with the ipmitool power interface.

Example of agent_images configuration:

         initramfs: tinyipa-stable-ussuri-20200617101427.gz
         kernel: tinyipa-stable-ussuri-20200617101427.vmlinuz

Since the bare metal nodes hardware may require additional drivers, you may need to build a deploy ramdisk for particular hardware. For more information, see Ironic Python Agent Builder. Be sure to create a ramdisk image with the version of Ironic Python Agent appropriate for your OpenStack release.


Ironic supports the flat and multitenancy networking modes.

The flat networking mode assumes that all bare metal nodes are pre-connected to a single network that cannot be changed during the virtual machine provisioning.

Example of the OsDpl resource illustrating the configuration for the flat network mode:

      - baremetal
        - bridge: ironic-pxe
          interface: <baremetal-interface>
            - flat
          physnet: ironic
          vlan_ranges: null
       # The name of neutron network used for provisioning/cleaning.
       baremetal_network_name: ironic-provisioning
         # Neutron baremetal network definition.
           physnet: ironic
           name: ironic-provisioning
           network_type: flat
           external: true
           shared: true
             - name: baremetal-subnet
       # The name of interface where provision services like tftp and ironic-conductor
       # are bound.
       provisioning_interface: br-baremetal

The multitenancy network mode uses the neutron Ironic network interface to share physical connection information with Neutron. This information is handled by Neutron ML2 drivers when plugging a Neutron port to a specific network. MOS supports the networking-generic-switch Neutron ML2 driver out of the box.

Example of the OsDpl resource illustrating the configuration for the multitenancy network mode:

      - baremetal
      tunnel_interface: ens3
        - physnet: physnet1
          interface: <physnet1-interface>
          bridge: br-ex
            - flat
          vlan_ranges: null
          mtu: null
        - physnet: ironic
          interface: <physnet-ironic-interface>
          bridge: ironic-pxe
            - vlan
          vlan_ranges: 1000:1099
      # The name of interface where provision services like tftp and ironic-conductor
      # are bound.
      provisioning_interface: <baremetal-interface>
      baremetal_network_name: ironic-provisioning
          physnet: ironic
          name: ironic-provisioning
          network_type: vlan
          segmentation_id: 1000
          external: true
          shared: false
            - name: baremetal-subnet
Node-specific settings


This feature is available starting from MOS Ussuri Update.

Depending on the use case, you may need to configure the same application components differently on different hosts. MOS enables you to easily perform the required configuration through node-specific overrides at the OpenStack Controller side.

The limitation of using the node-specific overrides is that they override only the configuration settings while other components, such as startup scripts and others, should be reconfigured as well.


The overrides have been implemented in a similar way to the OpenStack node and node label specific DaemonSet configurations. Though, the OpenStack Controller node-specific settings conflict with the upstream OpenStack node and node label specific DaemonSet configurations. Therefore, we do not recommend configuring node and node label overrides.

The node-specific settings are activated through the spec:nodes section of the OsDpl CR. The spec:nodes section contains the following subsections:

  • features- implements overrides for a limited subset of fields and is constructed similarly to spec::features

  • services - similarly to spec::services, enables you to override settings in general for the components running as DaemonSets.

Example configuration:

        # Detailed information about features might be found at
        # openstack_controller/admission/validators/nodes/schema.yaml
                # Any value from specific helm chart

OpenStack on Kubernetes architecture

OpenStack and auxiliary services are running as containers in the kind: Pod Kubernetes resources. All long-running services are governed by one of the ReplicationController-enabled Kubernetes resources, which include either kind: Deployment, kind: StatefulSet, or kind: DaemonSet.

The placement of the services is mostly governed by the Kubernetes node labels. The labels affecting the OpenStack services include:

  • openstack-control-plane=enabled - the node hosting most of the OpenStack control plane services.

  • openstack-compute-node=enabled - the node serving as a hypervisor for Nova. The virtual machines with tenants workloads are created there.

  • openvswitch=enabled - the node hosting Neutron L2 agents and OpenvSwitch pods that manage L2 connection of the OpenStack networks.

  • openstack-gateway=enabled - the node hosting Neutron L3, Metadata and DHCP agents, Octavia Health Manager, Worker and Housekeeping components.



OpenStack is an infrastructure management platform. Mirantis OpenStack for Kubernetes (MOS) uses Kubernetes mostly for orchestration and dependency isolation. As a result, multiple OpenStack services are running as privileged containers with host PIDs and Host Networking enabled. You must ensure that at least the user with the credentials used by Helm/Tiller (administrator) is capable of creating such Pods.

Infrastructure services




While the underlying Kubernetes cluster is configured to use Ceph CSI for providing persistent storage for container workloads, for some types of workloads such networked storage is suboptimal due to latency.

This is why the separate local-volume-provisioner CSI is deployed and configured as an additional storage class. Local Volume Provisioner is deployed as kind: DaemonSet.


A single WSREP (Galera) cluster of MariaDB is deployed as the SQL database to be used by all OpenStack services. It uses the storage class provided by Local Volume Provisioner to store the actual database files. The service is deployed as kind: StatefulSet of a given size, which is no less than 3, on any openstack-control-plane node. For details, see OpenStack database architecture.


RabbitMQ is used as a messaging bus between the components of the OpenStack services.

A separate instance of RabbitMQ is deployed for each OpenStack service that needs a messaging bus for intercommunication between its components.

An additional, separate RabbitMQ instance is deployed to serve as a notification messages bus for OpenStack services to post their own and listen to notifications from other services. StackLight also uses this message bus to collect notifications for monitoring purposes.

Each RabbitMQ instance is a single node and is deployed as kind: StatefulSet.


A single multi-instance of the Memcached service is deployed to be used by all OpenStack services that need caching, which are mostly HTTP API services.


A separate instance of etcd is deployed to be used by Cinder, which require Distributed Lock Management for coordination between its components.


Is deployed as kind: DaemonSet.

Image pre-caching

A special kind: DaemonSet is deployed and updated each time the kind: OpenStackDeployment resource is created or updated. Its purpose is to pre-cache container images on Kubernetes nodes, and thus, to minimize possible downtime when updating container images.

This is especially useful for containers used in kind: DaemonSet resources, as during the image update Kubernetes starts to pull the new image only after the container with the old image is shut down.

OpenStack services



Identity (Keystone)

Uses MySQL back end by default.

keystoneclient - a separate kind: Deployment with a pod that has the OpenStack CLI client as well as relevant plugins installed, and OpenStack admin credentials mounted. Can be used by administrator to manually interact with OpenStack APIs from within a cluster.

Image (Glance)

Supported back end is RBD (Ceph is required).

Volume (Cinder)

Supported back end is RBD (Ceph is required).

Network (Neutron)

Supported back ends are Open vSwitch and Tungsten Fabric.


Compute (Nova)

Supported hypervisor is Qemu/KVM through libvirt library.

Dashboard (Horizon)

DNS (Designate)

Supported back end is PowerDNS.

Load Balancer (Octavia)

RADOS Gateway Object Storage (SWIFT) Available since MOS Ussuri Update

Provides the object storage and a RADOS Gateway Swift API that is compatible with the OpenStack Swift API. You can manually enable the service in the OpenStackDeployment CR as described in Deployment Guide: Deploy an OpenStack cluster.

Instance HA (Masakari) Available since MOS 21.2, Technical Preview

An OpenStack service that ensures high availability of instances running on a host. You can manually enable Masakari in the OpenStackDeployment CR as described in Deployment Guide: Deploy an OpenStack cluster.

Orchestration (Heat)

Key Manager (Barbican)

The supported back ends include:

  • The built-in Simple Crypto, which is used by default

  • Vault

    Vault by HashiCorp is a third-party system and is not installed by MOS. Hence, the Vault storage back end should be available elsewhere on the user environment and accessible from the MOS deployment.

    If the Vault back end is used, you can configure Vault in the OpenStackDeployment CR as described in Deployment Guide: Deploy an OpenStack cluster.


Runs tests against a deployed OpenStack cloud. You can manually enable Tempest in the OpenStackDeployment CR as described in Deployment Guide: Deploy an OpenStack cluster.


Telemetry services include alarming (aodh), event storage (Panko), metering (Ceilometer), and metric (Gnocchi). All services should be enabled together through the list of services to be deployed in the OpenStackDeployment CR as described in Deployment Guide: Deploy an OpenStack cluster.

OpenStack database architecture

A complete setup of a MariaDB Galera cluster for OpenStack is illustrated in the following image:


MariaDB server pods are running a Galera multi-master cluster. Clients requests are forwarded by the Kubernetes mariadb service to the mariadb-server pod that has the primary label. Other pods from the mariadb-server StatefulSet have the backup label. Labels are managed by the mariadb-controller pod.

The MariaDB controller periodically checks the readiness of the mariadb-server pods and sets the primary label to it if the following requirements are met:

  • The primary label has not already been set on the pod.

  • The pod is in the ready state.

  • The pod is not being terminated.

  • The pod name has the lowest integer suffix among other ready pods in the StatefulSet. For example, between mariadb-server-1 and mariadb-server-2, the pod with the mariadb-server-1 name is preferred.

Otherwise, the MariaDB controller sets the backup label. This means that all SQL requests are passed only to one node while other two nodes are in the backup state and replicate the state from the primary node. The MariaDB clients are connecting to the mariadb service.

OpenStack and Ceph controllers integration

The integration between Ceph and OpenStack controllers is implemented through the shared Kubernetes openstack-ceph-shared namespace. Both controllers have access to this namespace to read and write the Kubernetes kind: Secret objects.


As Ceph is required and only supported back end for several OpenStack services, all necessary Ceph pools must be specified in the configuration of the kind: MiraCeph custom resource as part of the deployment. Once the Ceph cluster is deployed, the Ceph controller posts the information required by the OpenStack services to be properly configured as a kind: Secret object into the openstack-ceph-shared namespace. The OpenStack controller watches this namespace. Once the corresponding secret is created, the OpenStack controller transforms this secret to the data structures expected by the OpenStack-Helm charts. Even if an OpenStack installation is triggered at the same time as a Ceph cluster deployment, the OpenStack controller halts the deployment of the OpenStack services that depend on Ceph availability until the secret in the shared namespace is created by the Ceph controller.

For the configuration of Ceph RADOS Gateway as an OpenStack Object Storage, the reverse process takes place. The OpenStack controller waits for the OpenStack-Helm to create a secret with OpenStack Identity (Keystone) credentials that RADOS Gateway must use to validate the OpenStack Identity tokens, and posts it back to the same openstack-ceph-shared namespace in the format suitable for consumption by the Ceph controller. The Ceph controller then reads this secret and reconfigures RADOS Gateway accordingly.

OpenStack and StackLight integration

StackLight integration with OpenStack includes automatic discovery of RabbitMQ credentials for notifications and OpenStack credentials for OpenStack API metrics. For details, see the openstack.rabbitmq.credentialsConfig and openstack.telegraf.credentialsConfig parameters description in MOS Operations Guide: StackLight configuration parameters.

OpenStack and Tungsten Fabric integration

The levels of integration between OpenStack and Tungsten Fabric (TF) include:

Controllers integration

The integration between the OpenStack and TF controllers is implemented through the shared Kubernetes openstack-tf-shared namespace. Both controllers have access to this namespace to read and write the Kubernetes kind: Secret objects.

The OpenStack controller posts the data into the openstack-tf-shared namespace required by the TF services. The TF controller watches this namespace. Once an appropriate secret is created, the TF controller obtains it into the internal data structures for further processing.

The OpenStack controller includes the following data for the TF controller:

  • tunnel_inteface

    Name of the network interface for the TF data plane. This interface is used by TF for the encapsulated traffic for overlay networks.

  • Keystone authorization information

    Keystone Administrator credentials and an up-and-running IAM service are required for the TF controller to initiate the deployment process.

  • Nova metadata information

    Required for the TF vRrouter agent service.

Also, the OpenStack Controller watches the openstack-tf-shared namespace for the vrouter_port parameter that defines the vRouter port number and passes it to the nova-compute pod.

Services integration

The list of the OpenStack services that are integrated with TF through their API include:

  • neutron-server - integration is provided by the contrail-neutron-plugin component that is used by the neutron-server service for transformation of the API calls to the TF API compatible requests.

  • nova-compute - integration is provided by the contrail-nova-vif-driver and contrail-vrouter-api packages used by the nova-compute service for interaction with the TF vRouter to the network ports.

  • octavia-api - integration is provided by the Octavia TF Driver that enables you to use OpenStack CLI and Horizon for operations with load balancers. See Tungsten Fabric integration with OpenStack Octavia for details.


TF is not integrated with the following OpenStack services:

  • DNS service (Designate)

  • Key management (Barbican)


Depending on the size of an OpenStack environment and the components that you use, you may want to have a single or multiple network interfaces, as well as run different types of traffic on a single or multiple VLANs.

This section provides the recommendations for planning the network configuration and optimizing the cloud performance.

Physical networks layout

The image below illustrates the recommended physical networks layout for a Mirantis OpenStack for Kubernetes (MOS) deployment with Ceph.


The image below illustrates the Ceph storage physical network layout.


Network types

When planning your OpenStack environment, consider what types of traffic your workloads generate and design your network accordingly. If you anticipate that certain types of traffic, such as storage replication, will likely consume a significant amount of network bandwidth, you may want to move that traffic to a dedicated network interface to avoid performance degradation.

A Mirantis OpenStack for Kubernetes (MOS) deployment typically requires the following networks:

L3 networks for Kubernetes



Common/PXE network

The network used for the provisioning of bare metal servers.

Management network

The network used for managing of bare metal servers.

Kubernetes workloads network

The network used for communication between containers in Kubernetes.

Storage access network (Ceph)

The network used for accessing the Ceph storage. We recommended that it is placed on a dedicated hardware interface.

Storage replication network (Ceph)

The network used for the storage replication (Ceph). To ensure low latency and fast access, place the network on a dedicated hardware interface.

External networks (MetalLB)

The routable network used for external IP addresses of the Kubernetes LoadBalancer services managed by MetalLB.


When selecting subnets, ensure that the subnet ranges do not overlap with the internal subnets’ ranges. Otherwise, the users’ internal resources will not be available from the deployed Container Cloud managed cluster.

The MOS deployment additionally requires the following networks:

L3 networks for MOS

Service name




Provider networks

Typically, a routable network used to provide the external access to OpenStack instances (a floating network). Can be used by the OpenStack services such as Ironic, Manila, and others, to connect their management resources.


Overlay networks (virtual networks)

The network used to provide denied, secure tenant networks with the help of the tunneling mechanism (VLAN/GRE/VXLAN). If the VXLAN and GRE encapsulation takes place, the IP address assignment is required on interfaces at the node level.


Live migration network

The network used by the OpenStack compute service (Nova) to transfer data during live migration. Depending on the cloud needs, it can be placed on a dedicated physical network not to affect other networks during live migration. The IP address assignment is required on interfaces at the node level.

The way of mapping of the logical networks described above to physical networks and interfaces on nodes depends on the cloud size and configuration. We recommend placing OpenStack networks on a dedicated physical interface (bond) that is not shared with storage and Kubernetes management network to minimize the influence on each other.

Performance optimization

To improve the goodput, we recommend that you enable jumbo frames where possible. The jumbo frames have to be enabled on the whole path of the packets traverse. If one of the network components cannot handle jumbo frames, the network path uses the smallest MTU.

To provide fault tolerance of a single NIC, we recommend using the link aggregation, such as bonding. The link aggregation is useful for linear scaling of bandwidth, load balancing, and fault protection. Depending on the hardware equipment, different types of bonds might be supported. Use the multi-chassis link aggregation as it provides fault tolerance at the device level. For example, MLAG on Arista equipment or vPC on Cisco equipment.

The Linux kernel supports the following bonding modes:

  • active-backup

  • balance-xor

  • 802.3ad (LACP)

  • balance-tlb

  • balance-alb

Since LACP is the IEEE standard 802.3ad supported by the majority of network platforms, we recommend using this bonding mode.


Mirantis OpenStack for Kubernetes (MOS) uses Ceph as a distributed storage system for block and object storage. For more information, refer to Mirantis Container Cloud Reference Architecture: Storage.


StackLight is the logging, monitoring, and alerting solution that provides a single pane of glass for cloud maintenance and day-to-day operations as well as offers critical insights into cloud health including operational information about the components deployed with Mirantis OpenStack for Kubernetes (MOS). StackLight is based on Prometheus, an open-source monitoring solution and a time series database, and Elasticsearch, the logs and notifications storage.

Deployment architecture

Mirantis OpenStack for Kubernetes (MOS) deploys the StackLight stack as a release of a Helm chart that contains the helm-controller and HelmBundle custom resources. The StackLight HelmBundle consists of a set of Helm charts describing the StackLight components. Apart from the OpenStack-specific components below, StackLight also includes the components described in Mirantis Container Cloud Reference Architecture: Deployment architecture.

During the StackLight deployment, you can define the HA or non-HA StackLight architecture type. For details, see Mirantis Container Cloud Reference Architecture: StackLight database modes.

OpenStack-specific StackLight components overview

StackLight component


Prometheus native exporters and endpoints

Export the existing metrics as Prometheus metrics and include:

  • libvirt-exporter

  • memcached-exporter

  • mysql-exporter

  • rabbitmq-exporter

  • tungstenfabric-exporter Available since MOS 21.1

Telegraf OpenStack plugin

Collects and processes the OpenStack metrics.

Monitored components

StackLight measures, analyzes, and reports in a timely manner about failures that may occur in the following Mirantis OpenStack for Kubernetes (MOS) components and their sub-components. Apart from the components below, StackLight also monitors the components listed in Mirantis Container Cloud Reference Architecture: Monitored components.

  • libvirt

  • Memcached

  • MariaDB

  • NTP

  • OpenStack (Barbican, Cinder, Designate, Glance, Heat, Horizon, Ironic, Keystone, Neutron, Nova, Octavia)

  • OpenStack SSL certificates

  • Open vSwitch

  • RabbitMQ

  • Tungsten Fabric (Casandra, Kafka, Redis, ZooKeeper) Available since MOS 21.1

Elasticsearch and Prometheus storage sizing


Calculations in this document are based on numbers from a real-scale test cluster with 34 nodes. The exact space required for metrics and logs must be calculated depending on the ongoing cluster operations. Some operations force the generation of additional metrics and logs. The values below are approximate. Use them only as recommendations.

During the deployment of a new cluster, you must specify the Elasticsearch retention time and Persistent Volume Claim (PVC) size, Prometheus PVC, retention time, and retention size. When configuring an existing cluster, you can only set Elasticsearch retention time, Prometheus retention time, and retention size.

The following table describes the recommendations for both Elasticsearch and Prometheus retention size and PVC size for a cluster with 34 nodes. Retention time depends on the space allocated for the data. To calculate the required retention time, use the {retention time} = {retention size} / {amount of data per day} formula.


Required space per day



StackLight in non-HA mode:
  • 202 - 253 GB for the entire cluster

  • ~6 - 7.5 GB for a single node

StackLight in HA mode:
  • 404 - 506 GB for the entire cluster

  • ~12 - 15 GB for a single node

When setting Persistent Volume Claim Size for Elasticsearch during the cluster creation, take into account that it defines the PVC size for a single instance of the Elasticsearch cluster. StackLight in HA mode has 3 Elasticsearch instances. Therefore, for a total Elasticsearch capacity, multiply the PVC size by 3.


  • 11 GB for the entire cluster

  • ~400 MB for a single node

Every Prometheus instance stores the entire database. Multiple replicas store multiple copies of the same data. Therefore, treat the Prometheus PVC size as the capacity of Prometheus in the cluster. Do not sum them up.

Prometheus has built-in retention mechanisms based on the database size and time series duration stored in the database. Therefore, if you miscalculate the PVC size, retention size set to ~1 GB less than the PVC size will prevent disk overfilling.

Tungsten Fabric

Tungsten Fabric provides basic L2/L3 networking to an OpenStack environment running on the MKE cluster and includes the IP address management, security groups, floating IP addresses, and routing policies functionality. Tungsten Fabric is based on overlay networking, where all virtual machines are connected to a virtual network with encapsulation (MPLSoGRE, MPLSoUDP, VXLAN). This enables you to separate the underlay Kubernetes management network. A workload requires an external gateway, such as a hardware EdgeRouter or a simple gateway to route the outgoing traffic.

The Tungsten Fabric vRouter uses different gateways for the control and data planes.

Tungsten Fabric known limitations

This section contains a summary of the Tungsten Fabric upstream features and use cases not supported in MOS, features and use cases offered as Technology Preview in the current product release if any, and known limitations of Tungsten Fabric in integration with other product components.

Tungsten Fabric known limitations

Feature or use case



Tungsten Fabric web UI

Provided as is

MOS provides the TF web UI as is and does not include this service in the support Service Level Agreement

Automatic generation of network port records in DNSaaS (Designate)

Not supported

As a workaround, you can use the Tungsten Fabric built-in DNS service that enables virtual machines to resolve each other names

Secret management (Barbican)

Not supported

It is not possible to use the certificates stored in Barbican to terminate HTTPs on a load balancer in a Tungsten Fabric deployment

Role Based Access Control (RBAC) for Neutron objects

Not supported

Advanced Tungsten Fabric features

Not supported

Tungsten Fabric does not support the following upstream advanced features:

  • Service Function Chaining

  • Production ready multi-site SDN

Technical Preview

  • DPDK Available since MOS Ussuri Update

  • SR-IOV Available since MOS 21.2

Tungsten Fabric cluster overview

All services of Tungsten Fabric are delivered as separate containers, which are deployed by the Tungsten Fabric Operator (TFO). Each container has an INI-based configuration file that is available on the host system. The configuration file is generated automatically upon the container start and is based on environment variables provided by the TFO through Kubernetes ConfigMaps.

The main Tungsten Fabric containers run with the host network as DeploymentSet, without using the Kubernetes networking layer. The services listen directly on the host network interface.

The following diagram describes the minimum production installation of Tungsten Fabric with a Mirantis OpenStack for Kubernetes (MOS) deployment.


Tungsten Fabric components

This section describes the Tungsten Fabric services and their distribution across the Mirantis OpenStack for Kubernetes (MOS) deployment.

The Tungsten Fabric services run mostly as DaemonSets in a separate container for each service. The deployment and update processes are managed by the Tungsten Fabric operator. However, Kubernetes manages the probe checks and restart of broken containers.

The following tables describe the Tungsten Fabric services:

Configuration and control services in Tungsten Fabric controller containers

Service name

Service description


Exposes a REST-based interface for the Tungsten Fabric API.


Collects data of the Tungsten Fabric configuration processes and sends it to the Tungsten Fabric collector.


Communicates with the cluster gateways using BGP and with the vRouter agents using XMPP, as well as redistributes appropriate networking information.


Collects the Tungsten Fabric controller process data and sends this information to the Tungsten Fabric collector.


Manages physical networking devices using netconf or ovsdb. In multi-node deployments, it operates in the active-backup mode.


Using the named service, provides the DNS service to the VMs spawned on different compute nodes. Each vRouter node connects to two Tungsten Fabric controller containers that run the dns process.


The customized Berkeley Internet Name Domain (BIND) daemon of Tungsten Fabric that manages DNS zones for the dns service.


Listens to configuration changes performed by a user and generates corresponding system configuration objects. In multi-node deployments, it works in the active-backup mode.


Listens to configuration changes of service-template and service-instance, as well as spawns and monitors virtual machines for the firewall, analyzer services, and so on. In multi-node deployments, it works in the active-backup mode.


Consists of the webserver and jobserver services. Provides the Tungsten Fabric web UI.

Analytics services in Tungsten Fabric analytics containers

Service name

Service description


Evaluates and manages the alarms rules.


Provides a REST API to interact with the Cassandra analytics database.


Collects all Tungsten Fabric analytics process data and sends this information to the Tungsten Fabric collector.


Provisions the init model if needed. Collects data of the database process and sends it to the Tungsten Fabric collector.


Collects and analyzes data from all Tungsten Fabric services.


Handles the queries to access data from the Cassandra database.


Receives the authorization and configuration of the physical routers from the config-nodemgr service, polls the physical routers using the Simple Network Management Protocol (SNMP), and uploads the data to the Tungsten Fabric collector.


Reads the SNMP information from the physical router user-visible entities (UVEs), creates a neighbor list, and writes the neighbor information to the physical router UVEs. The Tungsten Fabric web UI uses the neighbor list to display the physical topology.

vRouter services on the OpenStack compute nodes

Service name

Service description


Connects to the Tungsten Fabric controller container and the Tungsten Fabric DNS system using the Extensible Messaging and Presence Protocol (XMPP).


Collects the supervisor vrouter data and sends it to the Tungsten Fabric collector.

Third-party services for Tungsten Fabric

Service name

Service description


  • On the Tungsten Fabric control plane nodes, maintains the configuration data of the Tungsten Fabric cluster.

  • On the Tungsten Fabric analytics nodes, stores the collector service data.


The Kubernetes operator that enables the Cassandra clusters creation and management.


Handles the messaging bus and generates alarms across the Tungsten Fabric analytics containers.


The Kubernetes operator that enabels Kafka clusters creation and management.


Stores the physical router UVE storage and serves as a messaging bus for event notifications.


The Kubernetes operator that enables Redis clusters creation and management.


Holds the active-backup status for the device-manager, svc-monitor, and the schema-transformer services. This service is also used for mapping of the Tungsten Fabric resources names to UUIDs.


The Kubernetes operator that enables ZooKeeper clusters creation and management.


Exchanges messages between API servers and original request senders.


The Kubernetes operator that enables RabbitMQ clusters creation and management.

Tungsten Fabric plugin services on the OpenStack controller nodes

Service name

Service description


The Neutron server that includes the Tungsten Fabric plugin.


The Octavia API that includes the Tungsten Fabric Octavia driver.

Tungsten Fabric operator

The Tungsten Fabric operator (TFO) is based on the operator SDK project. The operator SDK is a framework that uses the controller-runtime library to make writing operators easier by providing:

  • High-level APIs and abstractions to write the operational logic more intuitively.

  • Tools for scaffolding and code generation to bootstrap a new project fast.

  • Extensions to cover common operator use cases.

The TFO deploys the following sub-operators. Each sub-operator handles a separate part of a TF deployment:

TFO sub-operators




Deploys the Tungsten Fabric control services, such as:

  • Control

  • DNS

  • Control NodeManager


Deploys the Tungsten Fabric configuration services, such as:

  • API

  • Service monitor

  • Schema transformer

  • Device manager

  • Configuration NodeManager

  • Database NodeManager


Deploys the Tungsten Fabric analytics services, such as:

  • API

  • Collector

  • Alarm

  • Alarm-gen

  • SNMP

  • Topology

  • Alarm NodeManager

  • Database NodeManager

  • SNMP NodeManager


Deploys a vRouter on each compute node with the following services:

  • vRouter agent

  • NodeManager


Deploys the following web UI services:

  • Web server

  • Job server


Deploys the following tools to verify the TF deployment status:

  • TF-status

  • TF-status aggregator


An operator to run Tempest tests.

Besides the sub-operators that deploy TF services, TFO uses operators to deploy and maintain third-party services, such as different types of storage, cache, message system, and so on. The following table describes all third-party operators:

TFO third-party sub-operators




An upstream operator that automates the Cassandra HA storage operations for the configuration and analytics data.


An upstream operator for deployment and automation of a ZooKeeper cluster.


An operator for the Kafka cluster used by analytics services.


An upstream operator that automates the Redis cluster deployment and keeps it healthy.


An operator for the messaging system based on RabbitMQ.

The following diagram illustrates a simplified TFO workflow:


Tungsten Fabric traffic flow

This section describes the types of traffic and traffic flow directions in a Mirantis OpenStack for Kubernetes (MOS) cluster.

User interface and API traffic

The following diagram illustrates all types of UI and API traffic in a MOS cluster, including the monitoring and OpenStack API traffic. The OpenStack Dashboard pod hosts Horizon and acts as a proxy for all other types of traffic. TLS termination is also performed for this type of traffic.


SDN traffic

SDN or Tungsten Fabric traffic goes through the overlay Data network and processes east-west and north-south traffic for applications that run in a MOS cluster. This network segment typically contains tenant networks as separate MPLS-over-GRE and MPLS-over-UDP tunnels. The traffic load depends on the workload.

The control traffic between the Tungsten Fabric controllers, edge routers, and vRouters uses the XMPP with TLS and iBGP protocols. Both protocols produce low traffic that does not affect MPLS over GRE and MPLS over UDP traffic. However, this traffic is critical and must be reliably delivered. Mirantis recommends configuring higher QoS for this type of traffic.

The following diagram displays both MPLS over GRE/MPLS over UDP and iBGP and XMPP traffic examples in a MOS cluster:


Tungsten Fabric vRouter

The Tungsten Fabric vRouter provides data forwarding to an OpenStack tenant instance and reports statistics to the Tungsten Fabric analytics service. The Tungsten Fabric vRouter is installed on all OpenStack compute nodes. Mirantis OpenStack for Kubernetes (MOS) supports the kernel-based deployment of the Tungsten Fabric vRouter.

The vRouter agent acts as a local control plane. Each Tungsten Fabric vRouter agent is connected to at least two Tungsten Fabric controllers in an active-active redundancy mode. The Tungsten Fabric vRouter agent is responsible for all networking-related functions including routing instances, routes, and so on.

The Tungsten Fabric vRouter uses different gateways for the control and data planes. For example, the Linux system gateway is located on the management network, and the Tungsten Fabric gateway is located on the data plane network.

The following diagram illustrates the Tungsten Fabric kernel vRouter setup by the TF operator:


On the diagram above, the following types of networks interfaces are used:

  • eth0 - for the management (PXE) network (eth1 and eth2 are the slave interfaces of Bond0)

  • Bond0.x - for the MKE control plane network

  • Bond0.y - for the MKE data plane network

Tungsten Fabric integration with OpenStack Octavia

MOS ensures Octavia with Tungsten Fabric integration by OpenStack Octavia Driver with Tungsten Fabric HAProxy as a back end.

Octavia Tungsten Fabric Driver supports creation, update, and deletion operations with the following entities:

  • Load balancers


    For a load balancer creation operation, the driver supports only the vip-subnet-id argument, the vip-network-id argument is not supported.

  • Listeners

  • Pools

  • Health monitors

Octavia Tungsten Fabric Driver does not support the following functionality:

  • L7 load balancing capabilities, such as L7 policies, L7 rules, and others

  • Setting specific availability zones for load balancers and their resources

  • Using of the UDP protocol

  • Operations with Octavia quotas

  • Operations with Octavia flavors


Octavia Tungsten Fabric Driver enables you to manage the load balancer resources through the OpenStack CLI or OpenStack Horizon. Do not perform any operations on the load balancer resources through the Tungsten Fabric web UI because in this case the changes will not be reflected on the OpenStack side.