This documentation provides information on how to deploy and operate a Mirantis OpenStack for Kubernetes (MOS) environment. The documentation is intended to help operators to understand the core concepts of the product. The documentation provides sufficient information to deploy and operate the solution.
The information provided in this documentation set is being constantly improved and amended based on the feedback and kind requests from the consumers of MOS.
The following table lists the guides included in the documentation set you are reading:
Guide |
Purpose |
---|---|
MOS Reference Architecture |
Learn the fundamentals of MOS reference architecture to appropriately plan your deployment |
MOS Deployment Guide |
Deploy a MOS environment of a preferred configuration using supported deployment profiles tailored to the demands of specific business cases |
MOS Operations Guide |
Operate your MOS environment |
MOS Release notes |
Learn about new features and bug fixes in the current MOS version |
The MOS documentation home page contains references to all guides included in this documentation set. For your convenience, we provide all guides in HTML (default), single-page HTML, PDF, and ePUB formats. To use the preferred format of a guide, select the required option from the Formats menu next to the guide title.
This documentation is intended for engineers who have the basic knowledge of Linux, virtualization and containerization technologies, Kubernetes API and CLI, Helm and Helm charts, Mirantis Kubernetes Engine (MKE), and OpenStack.
This documentation set includes description of the Technology Preview features. A Technology Preview feature provide early access to upcoming product innovations, allowing customers to experience the functionality and provide feedback during the development process. Technology Preview features may be privately or publicly available and neither are intended for production use. While Mirantis will provide support for such features through official channels, normal Service Level Agreements do not apply. Customers may be supported by Mirantis Customer Support or Mirantis Field Support.
As Mirantis considers making future iterations of Technology Preview features generally available, we will attempt to resolve any issues that customers experience when using these features.
During the development of a Technology Preview feature, additional components may become available to the public for testing. Because Technology Preview features are being under development, Mirantis cannot guarantee the stability of such features. As a result, if you are using Technology Preview features, you may not be able to seamlessly upgrade to subsequent releases of that feature. Mirantis makes no guarantees that Technology Preview features will be graduated to a generally available product release.
The Mirantis Customer Success Organization may create bug reports on behalf of support cases filed by customers. These bug reports will then be forwarded to the Mirantis Product team for possible inclusion in a future release.
The following table contains the released revision of the documentation set you are reading:
Release date |
Description |
---|---|
November 05, 2020 |
MOS GA release |
December 23, 2020 |
MOS GA Update release |
March 01, 2021 |
MOS 21.1 |
This documentation set uses the following conventions in the HTML format:
Convention |
Description |
---|---|
boldface font |
Inline CLI tools and commands, titles of the procedures and system response examples, table titles |
|
Files names and paths, Helm charts parameters and their values, names of packages, nodes names and labels, and so on |
italic font |
Information that distinguishes some concept or term |
External links and cross-references, footnotes |
|
Main menu > menu item |
GUI elements that include any part of interactive user interface and menu navigation |
Superscript |
Some extra, brief information |
Note The Note block |
Messages of a generic meaning that may be useful for the user |
Caution The Caution block |
Information that prevents a user from mistakes and undesirable consequences when following the procedures |
Warning The Warning block |
Messages that include details that can be easily missed, but should not be ignored by the user and are valuable before proceeding |
See also The See also block |
List of references that may be helpful for understanding of some related tools, concepts, and so on |
Learn more The Learn more block |
Used in the Release Notes to wrap a list of internal references to the reference architecture, deployment and operation procedures specific to a newly implemented product feature |
Mirantis OpenStack for Kubernetes (MOS) enables the operator to create, scale, update, and upgrade OpenStack deployments on Kubernetes through a declarative API.
The Kubernetes built-in features, such as flexibility, scalability, and declarative resource definition make MOS a robust solution.
The detailed plan of any Mirantis OpenStack for Kubernetes (MOS) deployment is determined on a per-cloud basis. For the MOS reference architecture and design overview, see MOS Reference Architecture.
Also, read through Mirantis Container Cloud Reference Architecture: Container Cloud bare metal as a MOS managed cluster is deployed on top of a baremetal-based Container Cloud management cluster.
The bare metal management system enables the Infrastructure Operator to deploy Container Cloud on a set of bare metal servers. It also enables Container Cloud to deploy MOS managed clusters on bare metal servers without a pre-provisioned operating system.
To provision your bare metal management cluster, refer to Mirantis Container Cloud Deployment Guide: Deploy a baremetal-based management cluster
Before you proceed with the OpenStack deployment, you need to configure the host operating system to run OpenStack.
To calculate the required number of hosts, read through Hardware requirements.
The OpenStack Nova and Neutron services may require additional kernel modules to be loaded for normal operation. This section instructs you on how to enable the required kernel modules.
To enable KVM
A kernel-based Virtual Machine (KVM) is a virtualization module in the Linux kernel that enables the kernel to function as a hypervisor. The KVM-based virtualization is recommended on compute hosts to provide high performance VMs. See OpenStack Ussuri official documentation: Enable KVM for details.
Note
Some systems require that you enable the VT support in the system BIOS. To enable the VT support, refer to the BIOS guides of the manufacturer of your server motherboard.
Mirantis OpenStack for Kubernetes (MOS) typically requires common/PXE, management, tenant, and storage types of networks. For the details, refer to MOS Reference Architecture: Networking.
This section describes how to configure persistent networking on hosts with three and five NICs using the Mirantis Container Cloud L2 Templates.
This section provides recommended configuration for networking on servers with three NICs that include:
eno1
that is used as a PXE interface
ens3f1
and ens3f2
that are used for bond0
To configure networking:
Create subnets. For example:
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
name: storage-backend
namespace: managed-ns
spec:
cidr: 10.12.0.0/24
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
name: storage-frontend
namespace: managed-ns
spec:
cidr: 10.12.1.0/24
Create the openstack-example-3nic
L2 template.
Caution
The bootstrapping engine automatically assigns an IP address
to the PXE nic 0
NIC. To prevent the IP duplication during updates,
do not assign the IP address manually.
Example template:
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
labels:
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
name: openstack-example-3nic
namespace: managed-ns
spec:
clusterRef: managed-cluster
autoIfMappingPrio:
- provision
- eno
- ens
- enp
npTemplate: |-
version: 2
ethernets:
{{nic 0}}:
# IMPORTANT: Do not assign an IP address here explicitly
# to prevent IP duplication issues. The IP will be assigned
# automatically by the bootstrapping engine.
# addresses: []
match:
macaddress: {{mac 0}}
set-name: {{nic 0}}
mtu: 1500
{{nic 1}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 1}}
set-name: {{nic 1}}
mtu: 9000
{{nic 2}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 2}}
set-name: {{nic 2}}
mtu: 9000
bonds:
bond0:
interfaces:
- {{nic 1}}
- {{nic 2}}
vlans:
pr-floating:
id: 403
link: bond0
stor-frontend:
id: 404
link: bond0
addresses:
- {{ip "stor-frontend:storage-frontend"}}
mtu: 9000
stor-backend:
id: 405
link: bond0
addresses:
- {{ip "stor-backend:storage-backend"}}
mtu: 9000
This section provides recommended configuration for networking on servers with five NICs that include:
eno1
that is used as a PXE interface
ens3f1
and ens3f2
that are used for bond0
ens5f1
and ens5f2
that are used for bond1
To configure networking:
Create subnets. For example:
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
name: storage-backend
namespace: managed-ns
spec:
cidr: 10.12.0.0/24
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
name: storage-frontend
namespace: managed-ns
spec:
cidr: 10.12.1.0/24
---
apiVersion: ipam.mirantis.com/v1alpha1
kind: Subnet
metadata:
labels:
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
name: tenant-tunnel
namespace: managed-ns
spec:
cidr: 10.12.3.0/24
Create the openstack-example-5nic
L2 template.
Caution
The bootstrapping engine automatically assigns an IP address
to the PXE nic 0
NIC. To prevent the IP duplication during updates,
do not assign the IP address manually.
Example template:
apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
labels:
kaas.mirantis.com/provider: baremetal
kaas.mirantis.com/region: region-one
name: openstack-example-5nic
namespace: managed-ns
spec:
clusterRef: managed-cluster
autoIfMappingPrio:
- provision
- eno
- ens
- enp
npTemplate: |-
version: 2
ethernets:
{{nic 0}}:
# IMPORTANT: Do not assign an IP address here explicitly
# to prevent IP duplication issues. The IP will be assigned
# automatically by the bootstrapping engine.
# addresses: []
match:
macaddress: {{mac 0}}
set-name: {{nic 0}}
mtu: 1500
{{nic 1}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 1}}
set-name: {{nic 1}}
mtu: 9000
{{nic 2}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 2}}
set-name: {{nic 2}}
mtu: 9000
{{nic 3}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 3}}
set-name: {{nic 3}}
mtu: 9000
{{nic 4}}:
dhcp4: false
dhcp6: false
match:
macaddress: {{mac 4}}
set-name: {{nic 4}}
mtu: 9000
bonds:
bond0:
interfaces:
- {{nic 1}}
- {{nic 2}}
bond1:
interfaces:
- {{nic 3}}
- {{nic 4}}
vlans:
pr-floating:
id: 403
link: bond1
stor-frontend:
id: 404
link: bond0
addresses:
- {{ip "stor-frontend:storage-frontend"}}
mtu: 9000
stor-backend:
id: 405
link: bond1
addresses:
- {{ip "stor-backend:storage-backend"}}
mtu: 9000
tenant-tunnel:
id: 406
link: bond1
addresses:
- {{ip "tenant-tunnel:tenant-tunnel"}}
mtu: 9000
TungstenFabric services use the /etc/hosts
file as a source of information
for services configuration and registration in the database.
Caution
For a MOS with Tungsten Fabric deployment, the proper hosts file configuration is a requirement.
To configure the hosts file:
On each node that will be used for running TungstenFabric services, verify
that the /etc/hosts
file has an entry with a local hostname and
appropriate IP address from the Management network according to
MOS Reference Architecture: Networking.
For example:
127.0.0.1 localhost
<management-ip> <node-hostname>
....
After you prepare the host operating system as described in Configure host operating system, proceed to creating a MOS cluster.
A MOS cluster is deployed as a Container Cloud managed cluster through the Container Cloud web UI. For a detailed procedure, refer to Mirantis Container Cloud Operations Guide: Create a managed cluster.
To properly configure StackLight, consider the recommendations described in Elasticsearch and Prometheus storage sizing.
Note
Once you have created a MOS managed cluster, some StackLight alerts may raise as false-positive until you deploy the MOS OpenStack environment.
After you create a MOS managed cluster and configure operating systems for the bare metal hosts, proceed with adding the bare metal hosts to your deployment through the Mirantis Container Cloud web UI.
For the detailed procedure, refer to Mirantis Container Cloud Operations Guide: Add a bare metal host.
Note
To calculate the required number of hosts, read through Hardware requirements.
After you add bare metal hosts to the MOS managed cluster, create Kubernetes machines in your cluster using the Mirantis Container Cloud web UI.
For the detailed procedure, refer to Mirantis Container Cloud Operations Guide: Add a machine.
When adding the machines, verify that you label the Kubernetes nodes according to the OpenStack node roles:
Node role |
Description |
Kubernetes labels |
Minimal count |
---|---|---|---|
OpenStack control plane |
Hosts the OpenStack control plane services such as database, messaging, API, schedulers, conductors, L3 and L2 agents. |
openstack-control-plane=enabled openstack-gateway=enabled openvswitch=enabled |
3 |
OpenStack compute |
Hosts the OpenStack compute services such as libvirt and L2 agents. |
openstack-compute-node=enabled openvswitch=enabled (for a deployment with Open vSwitch as a
back end for networking) |
Varies |
Note
MOS allows collocating components on the same host, for example, Ceph OSD and OpenStack compute. For details, see Mirantis Container Cloud Operations Guide: Add, remove, or reconfigure Ceph nodes.
This section describes how to deploy a Ceph cluster in a MOS managed cluster.
For Ceph cluster limitations, see Mirantis Container Cloud Reference Architecture: Limitations.
To deploy a Ceph cluster:
Deploy Ceph in the same Kubernetes cluster as described in Mirantis Container Cloud Operations Guide: Add a Ceph cluster.
Open the KaaSCephCluster
CR for editing as described in
Mirantis Container Cloud Operations Guide: Ceph advanced configuration.
Verify that the following snippet is present in the KaaSCephCluster
configuration:
network:
clusterNet: 10.10.10.0/24
publicNet: 10.10.11.0/24
Configure the pools for Image, Block Storage, and Compute services.
Note
Ceph validates the specified pools. Therefore, do not omit any of the following pools.
spec:
pools:
- default: true
deviceClass: hdd
name: kubernetes
replicated:
size: 2
role: kubernetes
- default: false
deviceClass: hdd
name: volumes
replicated:
size: 2
role: volumes
- default: false
deviceClass: hdd
name: vms
replicated:
size: 2
role: vms
- default: false
deviceClass: hdd
name: backup
replicated:
size: 2
role: backup
- default: false
deviceClass: hdd
name: images
replicated:
size: 2
role: images
- default: false
deviceClass: hdd
name: other
replicated:
size: 2
role: other
Once all pools are created, verify that an appropriate secret required for
a successful deployment of the OpenStack services that rely on Ceph is
created in the openstack-ceph-shared
namespace:
kubectl -n openstack-ceph-shared get secrets openstack-ceph-keys
Example of a positive system response:
NAME TYPE DATA AGE
openstack-ceph-keys Opaque 7 36m
This section instructs you on how to deploy OpenStack on top of Kubernetes as well as how to troubleshoot the deployment and access your OpenStack environment after deployment.
This section instructs you on how to deploy OpenStack on top of Kubernetes
using the OpenStack Controller and openstackdeployments.lcm.mirantis.com
(OsDpl) CR.
To deploy an OpenStack cluster:
Verify that you have pre-configured the networking according to MOS Reference Architecture: Networking.
Verify that the TLS certificates that will be required for the OpenStack cluster deployment have been pre-generated.
Note
The Transport Layer Security (TLS) protocol is mandatory on public endpoints.
Caution
To avoid certificates renewal with subsequent OpenStack
updates during which additional services with new public endpoints may
appear, we recommend using wildcard SSL certificates for public
endpoints. For example, *.it.just.works
, where it.just.works
is
a cluster public domain.
The sample code block below illustrates how to generate a self-signed
certificate for the it.just.works
domain. The procedure presumes
the cfssl and cfssljson tools are installed on the
machine.
mkdir cert && cd cert
tee ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "8760h"
}
}
}
}
EOF
tee ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names":[{
"C": "<country>",
"ST": "<state>",
"L": "<city>",
"O": "<organization>",
"OU": "<organization unit>"
}]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
tee server-csr.json << EOF
{
"CN": "*.it.just.works",
"hosts": [
"*.it.just.works"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [ {
"C": "US",
"L": "CA",
"ST": "San Francisco"
}]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem --config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
Create the openstackdeployment.yaml
file that will include the
OpenStack cluster deployment configuration.
Note
The resource of kind OpenStackDeployment
(OsDpl) is a custom
resource defined by a resource of kind
CustomResourceDefinition
. The resource is validated with
the help of the OpenAPI v3 schema.
Configure the OsDpl resource depending on the needs of your deployment. For the configuration details, refer to MOS Reference Architecture: OpenStackDeployment resource.
Example of an OsDpl CR of minimum configuration:
apiVersion: lcm.mirantis.com/v1alpha1
kind: OpenStackDeployment
metadata:
name: openstack-cluster
namespace: openstack
spec:
openstack_version: ussuri
preset: compute
size: tiny
internal_domain_name: cluster.local
public_domain_name: it.just.works
features:
ssl:
public_endpoints:
api_cert: |-
# Update server certificate content
api_key: |-
# Update server private key content
ca_cert: |-
# Update CA certificate content
neutron:
tunnel_interface: ens3
external_networks:
- physnet: physnet1
interface: veth-phy
bridge: br-ex
network_types:
- flat
vlan_ranges: null
mtu: null
floating_network:
enabled: False
nova:
live_migration_interface: ens3
images:
backend: local
If required, enable DPDK, huge pages, and other supported Telco features as described in Advanced OpenStack configuration (optional).
To the openstackdeployment
object, add information about the TLS
certificates:
ssl:public_endpoints:ca_cert
- CA certificate content (ca.pem
)
ssl:public_endpoints:api_cert
- server certificate content
(server.pem
)
ssl:public_endpoints:api_key
- server private key (server-key.pem
)
Verify that the Load Balancer network does not overlap your corporate
or internal Kubernetes networks, for example, Calico IP pools. Also,
verify that the pool of Load Balancer network is big enough to provide
IP addresses for all Amphora VMs (loadbalancers
).
If required, reconfigure the Octavia network settings using the following sample structure:
spec:
services:
load-balancer:
octavia:
values:
octavia:
settings:
lbmgmt_cidr: "10.255.0.0/16"
lbmgmt_subnet_start: "10.255.1.0"
lbmgmt_subnet_end: "10.255.255.254"
Trigger the OpenStack deployment:
kubectl apply -f openstackdeployment.yaml
Monitor the status of your OpenStack deployment:
kubectl -n openstack get pods
kubectl -n openstack describe osdpl osh-dev
Assess the current status of the OpenStack deployment using the
status
section output in the OsDpl resource:
Get the OsDpl YAML file:
kubectl -n openstack get osdpl osh-dev -o yaml
Analyze the status
output using the detailed description in
MOS Reference Architecture: OpenStackDeployment resource:
The Status elements.
Verify that the OpenStack cluster has been deployed:
clinet_pod_name=$(kubectl -n openstack get pods -l application=keystone,component=client | grep keystone-client | head -1 | awk '{print $1}')
kubectl -n openstack exec -it $clinet_pod_name -- openstack service list
Example of a positive system response:
+----------------------------------+---------------+----------------+
| ID | Name | Type |
+----------------------------------+---------------+----------------+
| 159f5c7e59784179b589f933bf9fc6b0 | cinderv3 | volumev3 |
| 6ad762f04eb64a31a9567c1c3e5a53b4 | keystone | identity |
| 7e265e0f37e34971959ce2dd9eafb5dc | heat | orchestration |
| 8bc263babe9944cdb51e3b5981a0096b | nova | compute |
| 9571a49d1fdd4a9f9e33972751125f3f | placement | placement |
| a3f9b25b7447436b85158946ca1c15e2 | neutron | network |
| af20129d67a14cadbe8d33ebe4b147a8 | heat-cfn | cloudformation |
| b00b5ad18c324ac9b1c83d7eb58c76f5 | radosgw-swift | object-store |
| b28217da1116498fa70e5b8d1b1457e5 | cinderv2 | volumev2 |
| e601c0749ce5425c8efb789278656dd4 | glance | image |
+----------------------------------+---------------+----------------+
This section includes configuration information for available advanced Mirantis OpenStack for Kubernetes features that include DPDK with the Neutron OVS back end, huge pages, CPU pinning, and other Enhanced Platform Awareness (EPA) capabilities.
Note
Consider this section as part of Deploy an OpenStack cluster.
Note
This feature is available as technical preview. Use such configuration for testing and evaluation purposes only.
Caution
This feature is available starting from MOS Ussuri Update.
This section instructs you on how to enable DPDK with the Neutron OVS back end.
To enable DPDK with OVS:
Verify that your deployment meets the following requirements:
The required drivers have been installed on the host operating system.
Different Poll Mode Driver (PMD) types may require different kernel drivers to properly work with NIC. For more information about the DPDK drivers, read DPDK official documentation: Linux Drivers and Overview of Networking Drivers.
The DPDK NICs are not used on the host operating system.
The huge pages feature is enabled on the host. See Enable huge pages for OpenStack for details.
Enable DPDK in the OsDpl custom resource through the
node specific overrides
settings. For example:
spec:
nodes:
<NODE-LABEL>::<NODE-LABEL-VALUE>:
features:
neutron:
dpdk:
bridges:
- ip_address: 10.12.2.80/24
name: br-phy
driver: igb_uio
enabled: true
nics:
- bridge: br-phy
name: nic01
pci_id: "0000:05:00.0"
tunnel_interface: br-phy
Note
Consider this section as part of Deploy an OpenStack cluster.
Note
This feature is available as technical preview. Use such configuration for testing and evaluation purposes only.
This section instructs you on how to enable SR-IOV with the Neutron OVS back end.
To enable SR-IOV with OVS:
Verify that your deployment meets the following requirements:
NICs with the SR-IOV support are installed
SR-IOV and VT-d are enabled in BIOS
Enable IOMMU in the kernel by configuring intel_iommu=on
in the GRUB
configuration file. Specify the parameter for compute nodes in
BareMetalHostProfile
in the grubConfig
section:
spec:
grubConfig:
defaultGrubOptions:
- 'GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX intel_iommu=on"'
Configure the OpenStackDeployment
CR to deploy OpenStack with the
VLAN tenant network encapsulation.
Caution
To ensure correct appliance of the configuration changes, configure VLAN segmentation during the initial OpenStack deployment.
Configuration example:
spec:
features:
neutron:
external_networks:
- bridge: br-ex
interface: pr-floating
mtu: null
network_types:
- flat
physnet: physnet1
vlan_ranges: null
- bridge: br-tenant
interface: bond0
network_types:
- vlan
physnet: tenant
vlan_ranges: 490:499,1420:1459
tenant_network_types:
- vlan
Enable SR-IOV in the OpenStackDeployment
CR through the
node-specific overrides settings. For example:
spec:
nodes:
<NODE-LABEL>::<NODE-LABEL-VALUE>:
features:
neutron:
sriov:
enabled: true
nics:
- device: enp10s0f1
num_vfs: 7
physnet: tenant
Note
Consider this section as part of Deploy an OpenStack cluster.
Caution
This feature is available starting from MOS Ussuri Update.
The section describes how to perform advanced configuration for the OpenStack compute nodes. Such configuration can be required in some specific use cases, such as DPDK, SR-IOV, or huge pages features usage.
Note
Consider this section as part of Deploy an OpenStack cluster.
The huge pages OpenStack feature provides essential performance improvements for applications that are highly memory IO-bound. Huge pages should be enabled on a per compute node basis.
Since NUMATopologyFilter
is enabled in a MOS deployment by default,
to activate the feature, you need to enable huge pages on the host as
described in Mirantis Container Cloud Operations Guide:
Enable huge pages in a host profile.
Note
The multi-size huge pages are not fully supported by Kubernetes before 1.19. Therefore, define only one size in kernel parameters.
Note
Consider this section as part of Deploy an OpenStack cluster.
CPU isolation allows for better performance of some HPC applications,
such as Open vSwitch with DPDK. To configure CPU isolation, add the
isolcpus
parameter to the GRUB configuration. For example:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=8-19"
Use the instruction from Mirantis Container Cloud Operations Guide: Enable huge pages in a host profile as an example procedure for GRUB parameters configuration.
Note
Consider this section as part of Deploy an OpenStack cluster.
The majority of CPU topologies features are activated by NUMATopologyFilter
that is enabled by default. Such features do not require any further service
configuration and can be used directly on a vanilla MOS deployment. The list
of the CPU topologies features includes, for example:
NUMA placement policies
CPU pinning policies
CPU thread pinning policies
CPU topologies
To enable libvirt CPU pinning through the node-specific overrides in the
OpenStackDeployment
custom resource, use the following sample
configuration structure:
spec:
nodes:
<NODE-LABEL>::<NODE-LABEL-VALUE>:
services:
compute:
nova_compute:
values:
conf:
nova:
compute:
cpu_dedicated_set: 2-17
cpu_shared_set: 18-47
Note
Consider this section as part of Deploy an OpenStack cluster.
The Peripheral Component Interconnect (PCI) passthrough feature in OpenStack allows full access and direct control over physical PCI devices in guests. The mechanism applies to any kind of PCI devices including a Network Interface Card (NIC), Graphics Processing Unit (GPU), and any other device that can be attached to a PCI bus. The only requirement for the guest to properly use the device is to correctly install the driver.
To enable PCI passthrough in a MOS deployment:
For Linux X86 compute nodes, verify that the following features are enabled on the host:
VT-d in BIOS
IOMMU on the host operating system as described in Enable SR-IOV with OVS.
Configure the nova-api
service that is scheduled
on OpenStack controllers nodes. To generate the alias for PCI
in nova.conf
, add the alias configuration through the
OpenStackDeployment
CR.
Note
When configuring PCI with SR-IOV on the same host, the values
specified in alias
take precedence. Therefore, add the SR-IOV
devices to passthrough_whitelist
explicitly.
For example:
spec:
services:
compute:
nova:
values:
conf:
nova:
pci:
alias: '{ "vendor_id":"8086", "product_id":"154d", "device_type":"type-PF", "name":"a1" }'
Configure the nova-compute
service that is scheduled
on OpenStack compute nodes. To enable Nova to pass PCI devices
to virtual machines, configure the passthrough_whitelist
section in nova.conf``through the node-specific overrides in the
``OpenStackDeployment
CR. For example:
spec:
nodes:
<NODE-LABEL>::<NODE-LABEL-VALUE>:
services:
compute:
nova_compute:
values:
conf:
nova:
pci:
alias: '{ "vendor_id":"8086", "product_id":"154d", "device_type":"type-PF", "name":"a1" }'
passthrough_whitelist: '{ "vendor_id": "8086", "product_id": "10fb" }'
This section contains the guidelines on how to access your MOS OpenStack environment.
The OpenStack services are exposed through the Ingress NGINX controller.
To configure DNS to access your OpenStack environment:
Obtain the external IP address of the Ingress service:
kubectl -n openstack get services ingress
Example of system response:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress LoadBalancer 10.96.32.97 10.172.1.101 80:34234/TCP,443:34927/TCP,10246:33658/TCP 4h56m
Select from the following options:
If you have a corporate DNS server, update your corporate DNS service and create appropriate DNS records for all OpenStack public endpoints.
To obtain the full list of public endpoints:
kubectl -n openstack get ingress -ocustom-columns=NAME:.metadata.name,HOSTS:spec.rules[*].host | awk '/cluster-fqdn/ {print $2}'
Example of system response:
barbican.it.just.works
cinder.it.just.works
cloudformation.it.just.works
designate.it.just.works
glance.it.just.works
heat.it.just.works
horizon.it.just.works
keystone.it.just.works
neutron.it.just.works
nova.it.just.works
novncproxy.it.just.works
octavia.it.just.works
placement.it.just.works
If you do not have a corporate DNS server, perform one of the following steps:
Add the appropriate records to /etc/hosts
locally. For example:
10.172.1.101 barbican.it.just.works
10.172.1.101 cinder.it.just.works
10.172.1.101 cloudformation.it.just.works
10.172.1.101 designate.it.just.works
10.172.1.101 glance.it.just.works
10.172.1.101 heat.it.just.works
10.172.1.101 horizon.it.just.works
10.172.1.101 keystone.it.just.works
10.172.1.101 neutron.it.just.works
10.172.1.101 nova.it.just.works
10.172.1.101 novncproxy.it.just.works
10.172.1.101 octavia.it.just.works
10.172.1.101 placement.it.just.works
Deploy your DNS server on top of Kubernetes:
Deploy a standalone CoreDNS server by including the following
configuration into coredns.yaml
:
apiVersion: lcm.mirantis.com/v1alpha1
kind: HelmBundle
metadata:
name: coredns
namespace: osh-system
spec:
repositories:
- name: hub_stable
url: https://kubernetes-charts.storage.googleapis.com
releases:
- name: coredns
chart: hub_stable/coredns
version: 1.8.1
namespace: coredns
values:
image:
repository: mirantis.azurecr.io/openstack/extra/coredns
tag: "1.6.9"
isClusterService: false
servers:
- zones:
- zone: .
scheme: dns://
use_tcp: false
port: 53
plugins:
- name: cache
parameters: 30
- name: errors
# Serves a /health endpoint on :8080, required for livenessProbe
- name: health
# Serves a /ready endpoint on :8181, required for readinessProbe
- name: ready
# Required to query kubernetes API for data
- name: kubernetes
parameters: cluster.local
- name: loadbalance
parameters: round_robin
# Serves a /metrics endpoint on :9153, required for serviceMonitor
- name: prometheus
parameters: 0.0.0.0:9153
- name: forward
parameters: . /etc/resolv.conf
- name: file
parameters: /etc/coredns/it.just.works.db it.just.works
serviceType: LoadBalancer
zoneFiles:
- filename: it.just.works.db
domain: it.just.works
contents: |
it.just.works. IN SOA sns.dns.icann.org. noc.dns.icann.org. 2015082541 7200 3600 1209600 3600
it.just.works. IN NS b.iana-servers.net.
it.just.works. IN NS a.iana-servers.net.
it.just.works. IN A 1.2.3.4
*.it.just.works. IN A 1.2.3.4
Update the public IP address of the Ingress service:
sed -i 's/1.2.3.4/10.172.1.101/' release/ci/30-coredns.yaml
kubectl apply -f release/ci/30-coredns.yaml
Verify that the DNS resolution works properly:
Assign an external IP to the service:
kubectl -n coredns patch service coredns-coredns --type='json' -p='[{"op": "replace", "path": "/spec/ports", "value": [{"name": "udp-53", "port": 53, "protocol": "UDP", "targetPort": 53}]}]'
kubectl -n coredns patch service coredns-coredns --type='json' -p='[{"op": "replace", "path": "/spec/type", "value":"LoadBalancer"}]'
Obtain the external IP address of CoreDNS:
kubectl -n coredns get service coredns-coredns
Example of system response:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coredns-coredns ClusterIP 10.96.178.21 10.172.1.102 53/UDP,53/TCP 25h
Point your machine to use the correct DNS. It is 10.172.1.102
in the example system response above.
If you plan to launch Tempest tests or use the OpenStack client from
a keystone-client-XXX
pod, verify that the Kubernetes built-in
DNS service is configured to resolve your public FQDN records by
adding your public domain to Corefile
. For example,
to add the it.just.works
domain:
kubectl -n kube-system get configmap coredns -oyaml
Example of system response:
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
it.just.works:53 {
errors
cache 30
forward . 10.96.178.21
}
This section explains how to access your OpenStack environment as the Admin user.
Before you proceed, verify that you can access the Kubernetes API and have
privileges to read secrets from the openstack
namespace in Kubernetes or
you are able to exec to the pods in this namespace.
You can use the built-in admin CLI client and execute the openstack
CLI commands from a dedicated pod deployed in the openstack
namespace:
kubectl -n openstack exec \
$(kubectl -n openstack get pod -l application=keystone,component=client -ojsonpath='{.items[*].metadata.name}') \
-ti -- bash
This pod has python-openstackclient
and all required plugins already
installed. Also, this pod has cloud admin credentials stored as appropriate
shell environment variables for the openstack CLI command to
consume.
Configure the external DNS resolution for OpenStack services as described in Configure DNS to access OpenStack.
Obtain the password of the Admin user:
kubectl -n openstack get secret keystone-keystone-admin -ojsonpath='{.data.OS_PASSWORD}' | base64 -d
Access Horizon through your browser using its public service.
For example, https://horizon.it.just.works
.
To log in, specify the admin
user name and default
domain.
If the OpenStack Identity service has been deployed with the OpenID Connect
integration:
From the Authenticate using drop-down menu, select OpenID Connect.
Click Connect. You will be redirected to your identity provider to proceed with the authentication.
Note
If OpenStack has been deployed with self-signed TLS certificates for public endpoints, you may get a warning about an untrusted certificate. To proceed, allow the connection.
To be able to access your OpenStack environment using CLI, you need to set the required environment variables that are stored in an OpenStack RC environment file. You can either download a project-specific file from Horizon, which is the easiest way, or create an environment file.
To access OpenStack through CLI, select from the following options:
Download and source the OpenStack RC file:
Log in to Horizon as described in Access an OpenStack environment through Horizon.
Download the openstackrc
or clouds.yaml
file from the Web
interface.
On any shell from which you want to run OpenStack commands, source the environment file for the respective project.
Create and source the OpenStack RC file:
Configure the external DNS resolution for OpenStack services as described in Configure DNS to access OpenStack.
Create a stub of the OpenStack RC file:
cat << EOF > openstackrc
export OS_PASSWORD=$(kubectl -n openstack get secret keystone-keystone-admin -ojsonpath='{.data.OS_PASSWORD}' | base64 -d)
export OS_USERNAME=admin
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default
export OS_REGION_NAME=RegionOne
export OS_INTERFACE=public
export OS_IDENTITY_API_VERSION="3"
EOF
Add the Keystone public endpoint to this file as the OS_AUTH_URL
variable. For example, for the domain name used throughout this guide:
echo export OS_AUTH_URL=https://keystone.it.just.works >> openstackrc
Source the obtained data into the shell:
source <openstackrc>
Now, you can use the openstack CLI as usual. For example:
openstack user list
+----------------------------------+-----------------+
| ID | Name |
+----------------------------------+-----------------+
| dc23d2d5ee3a4b8fae322e1299f7b3e6 | internal_cinder |
| 8d11133d6ef54349bd014681e2b56c7b | admin |
+----------------------------------+-----------------+
Note
If OpenStack was deployed with self-signed TLS certificates for public endpoints, you may need to use the openstack CLI client with certificate validation disabled. For example:
openstack --insecure user list
This section provides the general debugging instructions for your OpenStack on Kubernetes deployment. Start your troubleshooting with the determination of the failing component that can include the OpenStack Operator, HelmBundle Controller, a particular pod or service.
Note
For Kubernetes cluster debugging and troubleshooting, refer to Kubernetes official documentation: Troubleshoot clusters and Docker Enterprise v3.0 documentation: Monitor and troubleshoot.
The HelmBundle controller is running in the following containers:
tiller
- runs Tiller, the server portion of Helm
controller
- contains processes that handle the HelmBundle object and
deploy it though Tiller
Download the Helm client to connect to Tiller:
export HELM_VERSION=v2.13.1
wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz
tar -xf helm-${HELM_VERSION}-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
On the Kubernetes node, run the following command to set up port forwarding:
kubectl port-forward -n osh-system helm-controller-0 44134:44134
Set up alias to use port forwarding:
alias helm="helm --host=localhost:44134"
Verify the Helm releases statuses:
helm list
If a Helm release is not in the DEPLOYED
state, obtain the details
from the output of the following command:
helm history --col-width 1000000 <release-name>
To verify the status of a HelmBundle release:
kubectl -n osh-system get helmbundles openstack-operator -o yaml
Example of a system response:
status:
releaseStatuses:
openstack-operator:
attempt: 1
chart: openstack-controller/openstack-operator
finishedAt: "2020-04-21T11:24:08Z"
hash: caa6a811e5540333b4a02d74f7ec855460d97f81efdd69ed13d516256cbf9f81
notes: ""
revision: 1
status: DEPLOYED
success: true
version: 0.1.9-55
The OpenStack Controller is running in several containers in the
openstack-operator-xxxx
pod in the osh-system
namespace.
For the full list of containers and their roles, refer to
MOS Reference Architecture: Openstack controller containers.
To verify the status of the OpenStack Controller, run:
kubectl -n osh-system get pods
Example of a system response:
NAME READY STATUS RESTARTS AGE
helm-controller-0 2/2 Running 0 70m
local-volume-provisioner-2xsl5 1/1 Running 0 70m
local-volume-provisioner-fqdf9 1/1 Running 0 70m
local-volume-provisioner-ncclp 1/1 Running 0 70m
openstack-operator-7f4b5cf88c-4tj26 5/5 Running 0 70m
To verify the logs for the osdpl
container, run:
kubectl -n osh-system logs -f <openstack-operator-xxxx> -c osdpl
openstack-operator
pod is missing¶The openstack-operator
pod is created by the openstack-operator
HelmBundle object. If the OpenStack Controller pod is not present
in the namespace, Debugging the HelmBundle controller.
This section includes the ways to mitigate the most common issues with the OsDpl CR. We assume that you have already debugged the HelmBundle and OpenStack Controllers to rule out possible failures with these components as described in Debugging the HelmBundle controller and Debugging the OpenStack Controller.
osdpl
has DEPLOYED=false
¶Possible root cause: One or more HelmBundle releases have not been deployed successfully.
To determine if you are affected:
Verify the status of the osdpl
object:
kubectl -n openstack get osdpl osh-dev
Example of a system response:
NAME AGE DEPLOYED DRAFT
osh-dev 22h false false
To debug the issue:
Identify the failed release by assessing the status:children
section in the OsDpl resource:
Get the OsDpl YAML file:
kubectl -n openstack get osdpl osh-dev -o yaml
Analyze the status
output using the detailed description in
MOS Reference Architecture: OpenStackDeployment resource:
The Status elements.
For further debugging, refer to Debugging the HelmBundle controller.
Init
¶Possible root cause: MOS uses the Kubernetes entrypoint init container to
resolve dependencies between objects. If the pod is stuck in Init:0/X
,
this pod may be waiting for its dependencies.
To debug the issue:
Verify the missing dependencies:
kubectl -n openstack logs -f placement-api-84669d79b5-49drw -c init
Example of a system response:
Entrypoint WARNING: 2020/04/21 11:52:50 entrypoint.go:72: Resolving dependency Job placement-ks-user in namespace openstack failed: Job Job placement-ks-user in namespace openstack is not completed yet .
Entrypoint WARNING: 2020/04/21 11:52:52 entrypoint.go:72: Resolving dependency Job placement-ks-endpoints in namespace openstack failed: Job Job placement-ks-endpoints in namespace openstack is not completed yet .
Possible root cause: some OpenStack services depend on Ceph. These services
include OpenStack Image, OpenStack Compute, and OpenStack Block Storage.
If the HelmBundle releases for these services are not present, the
openstack-ceph-keys
secret may be missing in the openstack-ceph-shared
namespace.
To debug the issue:
Verify that the Ceph Controller has created the openstack-ceph-keys
secret in the openstack-ceph-shared
namespace:
kubectl -n openstack-ceph-shared get secrets openstack-ceph-keys
Example of a positive system response:
NAME TYPE DATA AGE
openstack-ceph-keys Opaque 7 23h
If the secret is not present, create one manually.
This section describes how to deploy Tungsten Fabric as a back end for networking for your MOS environment.
Caution
Before you proceed with the Tungsten Fabric deployment, read through MOS Reference Architecture: Tungsten Fabric known limitations.
Before you proceed with the actual Tungsten Fabric (TF) deployment, verify that your deployment meets the following prerequisites:
Your MOS OpenStack cluster is deployed as described in
Deploy OpenStack with the compute-tf
preset enabled.
Your MOS OpenStack cluster uses the correct value of
features:neutron:tunnel_interface
in the openstackdeployment
object.
The TF Operator will consume this value through the shared secret and use
it as a network interface from the underlay network to create encapsulated
tunnels with the tenant networks.
Warning
TF uses features:neutron:tunnel_interface
to create the vhost0
virtual interface and transfers the IP configuration from the
tunnel_interface
to the virtual one. Therefore, plan this interface
as a dedicated physical interface for TF overlay networks.
The Kubernetes nodes are labeled according to the TF node roles:
Node role |
Description |
Kubernetes labels |
Minimal count |
---|---|---|---|
TF control plane |
Hosts the TF control plane services such as |
tfconfig=enabled tfcontrol=enabled tfwebui=enabled tfconfigdb=enabled |
3 |
TF analytics |
Hosts the TF analytics services. |
tfanalytics=enabled tfanalyticsdb=enabled |
3 |
TF vRouter |
Hosts the TF vRouter module and vRouter agent. |
tfvrouter=enabled |
Varies |
TF vRouter DPDK Technical Preview |
Hosts the TF vRouter agent in DPDK mode. |
tfvrouter-dpdk=enabled |
Varies |
Note
TF supports only Kubernetes OpenStack workloads.
Therefore, you should label OpenStack compute nodes with
the tfvrouter=enabled
label.
Note
Do not specify the openstack-gateway=enabled
and openvswitch=enabled
labels for the MOS deployments
with TF as a networking back end for OpenStack.
Deployment of Tungsten Fabric (TF) is managed by the
tungstenfabric-operator
Helm resource in a respective MOS
ClusterRelease
.
To deploy TF:
Verify that you have completed all prerequisite steps as described in Tungsten Fabric deployment prerequisites.
Create tungstenfabric.yaml
with the TF resource configuration.
For example:
apiVersion: operator.tf.mirantis.com/v1alpha1
kind: TFOperator
metadata:
name: openstack-tf
namespace: tf
spec:
settings:
orchestrator: openstack
Trigger the TF deployment:
kubectl apply -f tungstenfabric.yaml
Verify that TF has been successfully deployed:
kubectl get pods -n tf
The successfully deployed TF services should appear in the Running
status
in the system response.
Starting from MOS 21.1, if you have enabled StackLight, enable
Tungsten Fabric monitoring by setting tungstenFabricMonitoring.enabled
to true
as described in MOS Operations Guide:
Configure StackLight.
The Tungsten Fabric (TF) web UI allows for easy and fast TF resources
configuration, monitoring, and debugging. You can access
the TF web UI through either the Ingress service or the Kubernetes Service
directly. TLS termination for the https
protocol is performed through the
Ingress service.
To access the TF web UI through Ingress:
Log in to a local machine running Ubuntu 18.04 where kubectl is installed.
Obtain and export kubeconfig
of your managed cluster as described in
Mirantis Container Cloud Operations Guide:
Connect to a Container Cloud managed cluster.
Obtain the password of the Admin user:
kubectl -n openstack exec -it $(kubectl -n openstack get pod -l application=keystone,component=client -o jsonpath='{.items[0].metadata.name}') -- env | grep PASS
Obtain the external IP address of the Ingress service:
kubectl -n openstack get services ingress
Example of system response:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress LoadBalancer 10.96.32.97 10.172.1.101 80:34234/TCP,443:34927/TCP,10246:33658/TCP 4h56m
Note
Do not use the EXTERNAL-IP
value to directly access the TF web
UI. Instead, use the FQDN from the list below.
Obtain the FQDN of tf-webui
:
Note
The command below outputs all host names assigned to the TF web UI service. Use one of them.
kubectl -n tf get ingress tf-webui -o custom-columns=HOSTS:.spec.rules[*].host
Configure DNS to access the TF web UI host as described in Configure DNS to access OpenStack.
Use your favorite browser to access the TF web UI at
https://<FQDN-WEBUI>
.
This section provides the general debugging instructions for your Tungsten Fabric (TF) on Kubernetes deployment.
To enable debug logging for the Tungsten Fabric (TF) services:
Open the TF custom resource for modification:
kubectl -n tf edit tfoperators.operator.tf.mirantis.com openstack-tf
Specify the LOG_LEVEL
variable with the SYS_DEBUG
value
for the required TF service. For example, for the config-api
service:
spec:
controllers:
tf-config:
api:
containers:
- name: api
env:
- name: LOG_LEVEL
value: SYS_DEBUG
Warning
After the TF custom resource modification, the pods
related to the affected services will be restarted. This rule does not
apply to the tf-vrouter-agent-<XXXXX>
pods as their update strategy
differs. Therefore, if you enable the debug logging for the services in
a tf-vrouter-agent-<XXXXX>
pod, restart this pod manually after you
modify the custom resource.
If you cannot access the Tungsten Fabric (TF) web UI service, verify that the FQDN of the TF web UI is resolvable on your PC by running one of the following commands:
host tf-webui.it.just.works
# or
ping tf-webui.it.just.works
# or
dig host tf-webui.it.just.works
All commands above should resolve the web UI domain name to the IP address
that should match the EXTERNAL-IPs
subnet dedicated to Kubernetes.
If the TF web UI domain name has not been resolved to the IP address, your PC
is using a different DNS or the DNS does not contain the record for the TF web
UI service. To resolve the issue, define the IP address of the Ingress service
from the openstack
namespace of Kubernetes in the hosts
file of your
machine. To obtain the Ingress IP address:
kubectl -n openstack get svc ingress -o custom-columns=HOSTS:.status.loadBalancer.ingress[*].ip
If the web UI domain name is resolvable but you still cannot access the service, verify the connectivity to the cluster.