Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Starting with MOSK 25.2, the MOSK documentation set covers all product layers, including MOSK management (formerly Container Cloud). This means everything you need is in one place. Some legacy names may remain in the code and documentation and will be updated in future releases. The separate Container Cloud documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Example of a complete template configuration for cluster creation¶

The following example contains all required objects of an advanced network and host configuration for a MOSK cluster.

The procedure below contains:

Various .yaml objects to be applied with a MOSK cluster kubeconfig
Useful comments inside the .yaml example files
Example hardware and configuration data, such as network, disk, auth, that must be updated accordingly to fit your cluster configuration
Example templates, such as l2template and baremetalhostprofline, that illustrate how to implement a specific configuration

Caution

The exemplary configuration described below is not production ready and is provided for illustration purposes only.

For illustration purposes, all files provided in this exemplary procedure are named by the Kubernetes object types:

Note

Before update of the management cluster to 2.29.0 (Cluster release 16.4.0), instead of BareMetalHostInventory, use the BareMetalHost object. For details, see BareMetalHost resource.

Caution

While the Cluster release of the management cluster is 16.4.0, BareMetalHostInventory operations are allowed to m:kaas@management-admin only. This limitation is lifted once the management cluster is updated to the Cluster release 16.4.1 or later.

mosk-ns_BareMetalHostInventory_cz7700-mosk-cluster-control-noefi.yaml
mosk-ns_BareMetalHostInventory_cz7741-mosk-cluster-control-noefi.yaml
mosk-ns_BareMetalHostInventory_cz7743-mosk-cluster-control-noefi.yaml
mosk-ns_BareMetalHostInventory_cz812-mosk-cluster-storage-worker-noefi.yaml
mosk-ns_BareMetalHostInventory_cz813-mosk-cluster-storage-worker-noefi.yaml
mosk-ns_BareMetalHostInventory_cz814-mosk-cluster-storage-worker-noefi.yaml
mosk-ns_BareMetalHostInventory_cz815-mosk-cluster-worker-noefi.yaml
mosk-ns_BareMetalHostProfile_bmhp-cluster-default.yaml
mosk-ns_BareMetalHostProfile_worker-storage1.yaml
mosk-ns_Cluster_mosk-cluster.yaml
mosk-ns_KaaSCephCluster_ceph-cluster-mosk-cluster.yaml
mosk-ns_L2Template_bm-1490-template-controls-netplan-cz7700-pxebond.yaml
mosk-ns_L2Template_bm-1490-template-controls-netplan.yaml
mosk-ns_L2Template_bm-1490-template-workers-netplan.yaml
mosk-ns_Machine_cz7700-mosk-cluster-control-noefi-.yaml
mosk-ns_Machine_cz7741-mosk-cluster-control-noefi-.yaml
mosk-ns_Machine_cz7743-mosk-cluster-control-noefi-.yaml
mosk-ns_Machine_cz812-mosk-cluster-storage-worker-noefi-.yaml
mosk-ns_Machine_cz813-mosk-cluster-storage-worker-noefi-.yaml
mosk-ns_Machine_cz814-mosk-cluster-storage-worker-noefi-.yaml
mosk-ns_Machine_cz815-mosk-cluster-worker-noefi-.yaml
mosk-ns_PublicKey_mosk-cluster-key.yaml
mosk-ns_cz7700-cred.yaml
mosk-ns_cz7741-cred.yaml
mosk-ns_cz7743-cred.yaml
mosk-ns_cz812-cred.yaml
mosk-ns_cz813-cred.yaml
mosk-ns_cz814-cred.yaml
mosk-ns_cz815-cred.yaml
mosk-ns_Subnet_lcm-nw.yaml
mosk-ns_Subnet_api-lb.yaml
mosk-ns_Subnet_metallb-public-for-mosk.yaml (obsolete)
mosk-ns_Subnet_metallb-public-for-extiface.yaml
mosk-ns_Subnet_storage-backend.yaml
mosk-ns_Subnet_storage-frontend.yaml
mosk-ns_MetalLBConfig-lb-mosk.yaml
mosk-ns_MetalLBConfigTemplate-lb-mosk-template.yaml (obsolete)
default_Namespace_mosk-ns.yaml

Caution

The procedure below presumes that you apply each new .yaml file using kubectl create -f <file_name.yaml>.

To create an example configuration for a MOSK cluster creation:

Verify that you have configured the following items:
1. All bmh nodes for PXE boot as described in Add a bare metal host using CLI
2. All physical NICs of the bmh nodes
3. All required physical subnets and routing
Create an empty .yaml file with the namespace object:
```
apiVersion: v1
```
Select from the following options:
Since MCC 2.21.0 (11.5.0, 7.11.0)
Create the required number of .yaml files with the BareMetalHostCredential objects for each bmh node with the unique name and authentication data. The following example contains one BareMetalHostCredential object:

Note

The kaas.mirantis.com/region label is removed from all MOSK objects in 24.1. Therefore, do not add the label starting with this release. On existing clusters updated to this release, or if added manually, MOSK ignores this label.
mosk-ns_cz815-cred.yaml
apiVersion: kaas.mirantis.com/v1alpha1 kind: BareMetalHostCredential metadata: name: cz815-cred namespace: mosk-ns labels: kaas.mirantis.com/region: region-one spec: username: admin password: value: supersecret
Before MCC 2.21.0 (11.4.0, 8.10.0, 7.10.0, or earlier)
Create the required number of .yaml files with the Secret objects for each bmh node with the unique name and authentication data. The following example contains one Secret object:
mosk-ns_cz815-cred.yaml
apiVersion: v1 data: password: YWRtaW4= username: ZW5naW5lZXI= kind: Secret metadata: labels: kaas.mirantis.com/credentials: 'true' kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one name: cz815-cred namespace: mosk-ns

Create a set of files with the bmh nodes configuration:

Since MOSK 25.1

Before MOSK 25.1

Verify that the inspecting phase has started:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get bmh -o wide

Example of system response:

NAME                                       STATUS STATE CONSUMER BMC           BOOTMODE ONLINE ERROR REGION
cz7700-mosk-cluster-control-noefi       OK     inspecting     192.168.1.12  legacy   true         region-one
cz7741-mosk-cluster-control-noefi       OK     inspecting     192.168.1.76  legacy   true         region-one
cz7743-mosk-cluster-control-noefi       OK     inspecting     192.168.1.78  legacy   true         region-one
cz812-mosk-cluster-storage-worker-noefi OK     inspecting     192.168.1.182 legacy   true         region-one

Wait for inspection to complete. Usually, it takes up to 15 minutes.

Collect the bmh hardware information to create the l2template and bmh objects:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get bmh -o wide

Example of system response:

NAME                                       STATUS STATE CONSUMER BMC           BOOTMODE ONLINE ERROR REGION
cz7700-mosk-cluster-control-noefi       OK     available      192.168.1.12  legacy   true         region-one
cz7741-mosk-cluster-control-noefi       OK     available      192.168.1.76  legacy   true         region-one
cz7743-mosk-cluster-control-noefi       OK     available      192.168.1.78  legacy   true         region-one
cz812-mosk-cluster-storage-worker-noefi OK     available      192.168.1.182 legacy   true         region-one

KUBECONFIG=kubeconfig kubectl -n mosk-ns get bmh cz7700-mosk-cluster-control-noefi -o yaml | less

Example of system response:

..
nics:
- ip: ""
  mac: 0c:c4:7a:1d:f4:a6
  model: 0x8086 0x10fb
  # discovered interfaces
  name: ens4f0
  pxe: false
  # temporary PXE address discovered from baremetal-mgmt
- ip: 172.16.170.30
  mac: 0c:c4:7a:34:52:04
  model: 0x8086 0x1521
  name: enp9s0f0
  pxe: true
  # duplicates temporary PXE address discovered from baremetal-mgmt
  # since we have fallback-bond configured on host
- ip: 172.16.170.33
  mac: 0c:c4:7a:34:52:05
  model: 0x8086 0x1521
  # discovered interfaces
  name: enp9s0f1
  pxe: false
...
storage:
- by_path: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
  model: Samsung SSD 850
  name: /dev/sda
  rotational: false
  sizeBytes: 500107862016
- by_path: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
  model: Samsung SSD 850
  name: /dev/sdb
  rotational: false
  sizeBytes: 500107862016
....

Create bare metal host profiles:

mosk-ns_BareMetalHostProfile_bmhp-cluster-default.yaml

apiVersion: metal3.io/v1alpha1
kind: BareMetalHostProfile
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: mosk-cluster
    # This label indicates that this profile will be default in
    # namespaces, so machines w\o exact profile selecting will use
    # this template
    kaas.mirantis.com/defaultBMHProfile: 'true'
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
  name: bmhp-cluster-default
  namespace: mosk-ns
spec:
  devices:
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
      minSize: 120Gi
      wipe: true
    partitions:
    - name: bios_grub
      partflags:
      - bios_grub
      size: 4Mi
      wipe: true
    - name: uefi
      partflags:
      - esp
      size: 200Mi
      wipe: true
    - name: config-2
      size: 64Mi
      wipe: true
    - name: lvm_dummy_part
      size: 1Gi
      wipe: true
    - name: lvm_root_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
      minSize: 30Gi
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-3
      minSize: 30Gi
      wipe: true
    partitions:
    - name: lvm_lvp_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-4
      wipe: true
  fileSystems:
  - fileSystem: vfat
    partition: config-2
  - fileSystem: vfat
    mountPoint: /boot/efi
    partition: uefi
  - fileSystem: ext4
    logicalVolume: root
    mountPoint: /
  - fileSystem: ext4
    logicalVolume: lvp
    mountPoint: /mnt/local-volumes/
  grubConfig:
    defaultGrubOptions:
    - GRUB_DISABLE_RECOVERY="true"
    - GRUB_PRELOAD_MODULES=lvm
    - GRUB_TIMEOUT=30
  kernelParameters:
    modules:
    - content: 'options kvm_intel nested=1'
      filename: kvm_intel.conf
    sysctl:
    # For the list of options prohibited to change, refer to
    # https://docs.mirantis.com/mke/3.7/install/predeployment/set-up-kernel-default-protections.html
      fs.aio-max-nr: '1048576'
      fs.file-max: '9223372036854775807'
      fs.inotify.max_user_instances: '4096'
      kernel.core_uses_pid: '1'
      kernel.dmesg_restrict: '1'
      net.ipv4.conf.all.rp_filter: '0'
      net.ipv4.conf.default.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.m-pub.rp_filter: '0'
      vm.max_map_count: '262144'
  logicalVolumes:
  - name: root
    size: 0
    vg: lvm_root
  - name: lvp
    size: 0
    vg: lvm_lvp
  postDeployScript: |
    #!/bin/bash -ex
    # used for test-debug only!
    echo "root:r00tme" | sudo chpasswd
    echo 'ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"' > /etc/udev/rules.d/60-ssd-scheduler.rules
    echo $(date) 'post_deploy_script done' >> /root/post_deploy_done

  preDeployScript: |
    #!/bin/bash -ex
    echo "$(date) pre_deploy_script done" >> /root/pre_deploy_done
  volumeGroups:
  - devices:
    - partition: lvm_root_part
    name: lvm_root
  - devices:
    - partition: lvm_lvp_part
    name: lvm_lvp
  - devices:
    - partition: lvm_dummy_part
    # here we can create lvm, but do not mount or format it somewhere
    name: lvm_forawesomeapp

mosk-ns_BareMetalHostProfile_worker-storage1.yaml

apiVersion: metal3.io/v1alpha1
kind: BareMetalHostProfile
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: mosk-cluster
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
  name: worker-storage1
  namespace: mosk-ns
spec:
  devices:
  - device:
      minSize: 120Gi
      wipe: true
    partitions:
    - name: bios_grub
      partflags:
      - bios_grub
      size: 4Mi
      wipe: true
    - name: uefi
      partflags:
      - esp
      size: 200Mi
      wipe: true
    - name: config-2
      size: 64Mi
      wipe: true
    # Create dummy partition w\o mounting
    - name: lvm_dummy_part
      size: 1Gi
      wipe: true
    - name: lvm_root_part
      size: 0
      wipe: true
  - device:
      # Will be used for Ceph, so required to be wiped
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
      minSize: 30Gi
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
      minSize: 30Gi
      wipe: true
    partitions:
    - name: lvm_lvp_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-3
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-4
      minSize: 30Gi
      wipe: true
    partitions:
      - name: lvm_lvp_part_sdf
        wipe: true
        size: 0
  fileSystems:
  - fileSystem: vfat
    partition: config-2
  - fileSystem: vfat
    mountPoint: /boot/efi
    partition: uefi
  - fileSystem: ext4
    logicalVolume: root
    mountPoint: /
  - fileSystem: ext4
    logicalVolume: lvp
    mountPoint: /mnt/local-volumes/
  grubConfig:
    defaultGrubOptions:
    - GRUB_DISABLE_RECOVERY="true"
    - GRUB_PRELOAD_MODULES=lvm
    - GRUB_TIMEOUT=30
  kernelParameters:
    modules:
    - content: 'options kvm_intel nested=1'
      filename: kvm_intel.conf
    sysctl:
    # For the list of options prohibited to change, refer to
    # https://docs.mirantis.com/mke/3.6/install/predeployment/set-up-kernel-default-protections.html
      fs.aio-max-nr: '1048576'
      fs.file-max: '9223372036854775807'
      fs.inotify.max_user_instances: '4096'
      kernel.core_uses_pid: '1'
      kernel.dmesg_restrict: '1'
      net.ipv4.conf.all.rp_filter: '0'
      net.ipv4.conf.default.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.m-pub.rp_filter: '0'
      vm.max_map_count: '262144'
  logicalVolumes:
  - name: root
    size: 0
    vg: lvm_root
  - name: lvp
    size: 0
    vg: lvm_lvp
  postDeployScript: |

    #!/bin/bash -ex

    # used for test-debug only! That would allow operator to logic via TTY.
    echo "root:r00tme" | sudo chpasswd
    # Just an example for enforcing "ssd" disks to be switched to use "deadline" i\o scheduler.
    echo 'ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"' > /etc/udev/   rules.d/60-ssd-scheduler.rules
    echo $(date) 'post_deploy_script done' >> /root/post_deploy_done

  preDeployScript: |
    #!/bin/bash -ex
    echo "$(date) pre_deploy_script done" >> /root/pre_deploy_done

  volumeGroups:
  - devices:
    - partition: lvm_root_part
    name: lvm_root
  - devices:
    - partition: lvm_lvp_part
    - partition: lvm_lvp_part_sdf
    name: lvm_lvp
  - devices:
    - partition: lvm_dummy_part
    name: lvm_forawesomeapp

Note

If you mount the /var directory, before configuring BareMetalHostProfile, review Mounting recommendations for the /var directory.

Create the L2Template objects:

Create the Subnet objects:

Create MetalLB configuration objects:

Since MCC 2.27.0 (17.2.0 and 16.2.0):

Before MCC 2.27.0 (17.1.0, 16.1.0, or earlier):

Before MCC 2.24.0 (12.7.0, 11.7.0, or earlier):

Create the PublicKey object for a MOSK cluster connection. For details, see PublicKey resource.

Create the Cluster object. For details, see Cluster resource.

Create the Machine objects linked to each bmh node. For details, see Machine resource.

Verify that the bmh nodes are in the provisioning state:

KUBECONFIG=kubectl kubectl -n mosk-ns get bmh  -o wide

Example of system response:

NAME                                  STATUS STATE          CONSUMER                                    BMC          BOOTMODE   ONLINE  ERROR REGION
cz7700-mosk-cluster-control-noefi  OK     provisioning   cz7700-mosk-cluster-control-noefi-8bkqw  192.168.1.12  legacy     true          region-one
cz7741-mosk-cluster-control-noefi  OK     provisioning   cz7741-mosk-cluster-control-noefi-42tp2  192.168.1.76  legacy     true          region-one
cz7743-mosk-cluster-control-noefi  OK     provisioning   cz7743-mosk-cluster-control-noefi-8cwpw  192.168.1.78  legacy     true          region-one
...

Wait until all bmh nodes are in the provisioned state.

Verify that the lcmmachine phase has started:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get lcmmachines  -o wide

Example of system response:

NAME                                       CLUSTERNAME       TYPE      STATE   INTERNALIP     HOSTNAME                                         AGENTVERSION
cz7700-mosk-cluster-control-noefi-8bkqw mosk-cluster   control   Deploy  172.16.170.153 kaas-node-803721b4-227c-4675-acc5-15ff9d3cfde2   v0.2.0-349-g4870b7f5
cz7741-mosk-cluster-control-noefi-42tp2 mosk-cluster   control   Prepare 172.16.170.152 kaas-node-6b8f0d51-4c5e-43c5-ac53-a95988b1a526   v0.2.0-349-g4870b7f5
cz7743-mosk-cluster-control-noefi-8cwpw mosk-cluster   control   Prepare 172.16.170.151 kaas-node-e9b7447d-5010-439b-8c95-3598518f8e0a   v0.2.0-349-g4870b7f5
...

Verify that the lcmmachine phase is complete and the Kubernetes cluster is created:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get lcmmachines  -o wide

Example of system response:

NAME                                       CLUSTERNAME       TYPE     STATE  INTERNALIP      HOSTNAME                                        AGENTVERSION
cz7700-mosk-cluster-control-noefi-8bkqw  mosk-cluster  control  Ready  172.16.170.153  kaas-node-803721b4-227c-4675-acc5-15ff9d3cfde2  v0.2.0-349-g4870b7f5
cz7741-mosk-cluster-control-noefi-42tp2  mosk-cluster  control  Ready  172.16.170.152  kaas-node-6b8f0d51-4c5e-43c5-ac53-a95988b1a526  v0.2.0-349-g4870b7f5
cz7743-mosk-cluster-control-noefi-8cwpw  mosk-cluster  control  Ready  172.16.170.151  kaas-node-e9b7447d-5010-439b-8c95-3598518f8e0a  v0.2.0-349-g4870b7f5
...

Create the KaaSCephCluster object:

Note

The storageDevices[].fullPath field is available since Container Cloud 2.25.0 (Cluster releases 17.0.0 and 16.0.0). For the clusters running earlier product versions, define the /dev/disk/by-id symlinks using storageDevices[].name instead.

Obtain kubeconfig of the newly created MOSK cluster:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get secrets mosk-cluster-kubeconfig -o jsonpath='{.data.admin\.conf}' | base64 -d |  tee mosk.kubeconfig

Verify the status of the Ceph cluster in your MOSK cluster:

KUBECONFIG=mosk.kubeconfig kubectl -n rook-ceph exec -it $(KUBECONFIG=mosk.kubeconfig kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph -s

Example of system response:

cluster:
  id:     e75c6abd-c5d5-4ae8-af17-4711354ff8ef
  health: HEALTH_OK
services:
  mon: 3 daemons, quorum a,b,c (age 55m)
  mgr: a(active, since 55m)
  osd: 3 osds: 3 up (since 54m), 3 in (since 54m)
data:
  pools:   1 pools, 32 pgs
  objects: 273 objects, 555 MiB
  usage:   4.0 GiB used, 1.6 TiB / 1.6 TiB avail
  pgs:     32 active+clean
io:
  client:   51 KiB/s wr, 0 op/s rd, 4 op/s wr