Mirantis Container Cloud (MCC) becomes part of Mirantis OpenStack for Kubernetes (MOSK)!

Now, the MOSK documentation set covers all product layers, including MOSK management (formerly Container Cloud). This means everything you need is in one place. Some legacy names may remain in the code and documentation and will be updated in future releases. The separate Container Cloud documentation site will be retired, so please update your bookmarks for continued easy access to the latest content.

Example of a complete template configuration for cluster creation¶

The following example contains all required objects of an advanced network and host configuration for a MOSK cluster.

The procedure below contains:

Various .yaml objects to be applied with a MOSK cluster kubeconfig
Useful comments inside the .yaml example files
Example hardware and configuration data, such as network, disk, auth, that must be updated accordingly to fit your cluster configuration
Example templates, such as l2template and baremetalhostprofline, that illustrate how to implement a specific configuration

Caution

The exemplary configuration described below is not production ready and is provided for illustration purposes only.

For illustration purposes, all files provided in this exemplary procedure are named by the Kubernetes object types:

mosk-ns_BareMetalHostInventory_cz7700-mosk-cluster-control-noefi.yaml
mosk-ns_BareMetalHostInventory_cz7741-mosk-cluster-control-noefi.yaml
mosk-ns_BareMetalHostInventory_cz7743-mosk-cluster-control-noefi.yaml
mosk-ns_BareMetalHostInventory_cz812-mosk-cluster-storage-worker-noefi.yaml
mosk-ns_BareMetalHostInventory_cz813-mosk-cluster-storage-worker-noefi.yaml
mosk-ns_BareMetalHostInventory_cz814-mosk-cluster-storage-worker-noefi.yaml
mosk-ns_BareMetalHostInventory_cz815-mosk-cluster-worker-noefi.yaml
mosk-ns_BareMetalHostProfile_bmhp-cluster-default.yaml
mosk-ns_BareMetalHostProfile_worker-storage1.yaml
mosk-ns_Cluster_mosk-cluster.yaml
mosk-ns_KaaSCephCluster_ceph-cluster-mosk-cluster.yaml
mosk-ns_L2Template_bm-1490-template-controls-netplan-cz7700-pxebond.yaml
mosk-ns_L2Template_bm-1490-template-controls-netplan.yaml
mosk-ns_L2Template_bm-1490-template-workers-netplan.yaml
mosk-ns_Machine_cz7700-mosk-cluster-control-noefi-.yaml
mosk-ns_Machine_cz7741-mosk-cluster-control-noefi-.yaml
mosk-ns_Machine_cz7743-mosk-cluster-control-noefi-.yaml
mosk-ns_Machine_cz812-mosk-cluster-storage-worker-noefi-.yaml
mosk-ns_Machine_cz813-mosk-cluster-storage-worker-noefi-.yaml
mosk-ns_Machine_cz814-mosk-cluster-storage-worker-noefi-.yaml
mosk-ns_Machine_cz815-mosk-cluster-worker-noefi-.yaml
mosk-ns_PublicKey_mosk-cluster-key.yaml
mosk-ns_cz7700-cred.yaml
mosk-ns_cz7741-cred.yaml
mosk-ns_cz7743-cred.yaml
mosk-ns_cz812-cred.yaml
mosk-ns_cz813-cred.yaml
mosk-ns_cz814-cred.yaml
mosk-ns_cz815-cred.yaml
mosk-ns_Subnet_lcm-nw.yaml
mosk-ns_Subnet_api-lb.yaml
mosk-ns_Subnet_metallb-public-for-extiface.yaml
mosk-ns_Subnet_storage-backend.yaml
mosk-ns_Subnet_storage-frontend.yaml
mosk-ns_MetalLBConfig-lb-mosk.yaml
default_Namespace_mosk-ns.yaml

Caution

The procedure below presumes that you apply each new .yaml file using kubectl create -f <file_name.yaml>.

To create an example configuration for a MOSK cluster creation:

Verify that you have configured the following items:
1. All bmh nodes for PXE boot as described in Add a bare metal host using CLI
2. All physical NICs of the bmh nodes
3. All required physical subnets and routing
Create an empty .yaml file with the namespace object:
```
apiVersion: v1
```
Create the required number of .yaml files with the BareMetalHostCredential objects for each bmh node with the unique name and authentication data. The following example contains one BareMetalHostCredential object:
mosk-ns_cz815-cred.yaml
apiVersion: kaas.mirantis.com/v1alpha1 kind: BareMetalHostCredential metadata: name: cz815-cred namespace: mosk-ns spec: username: admin password: value: supersecret

Create a set of files with the bmh nodes configuration:

Verify that the inspecting phase has started:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get bmh -o wide

Example of system response:

NAME                                       STATUS STATE CONSUMER BMC           BOOTMODE ONLINE ERROR
cz7700-mosk-cluster-control-noefi       OK     inspecting     192.168.1.12  legacy   true
cz7741-mosk-cluster-control-noefi       OK     inspecting     192.168.1.76  legacy   true
cz7743-mosk-cluster-control-noefi       OK     inspecting     192.168.1.78  legacy   true
cz812-mosk-cluster-storage-worker-noefi OK     inspecting     192.168.1.182 legacy   true

Wait for inspection to complete. Usually, it takes up to 15 minutes.

Collect the bmh hardware information to create the l2template and bmh objects:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get bmh -o wide

Example of system response:

NAME                                       STATUS STATE CONSUMER BMC           BOOTMODE ONLINE ERROR
cz7700-mosk-cluster-control-noefi       OK     available      192.168.1.12  legacy   true
cz7741-mosk-cluster-control-noefi       OK     available      192.168.1.76  legacy   true
cz7743-mosk-cluster-control-noefi       OK     available      192.168.1.78  legacy   true
cz812-mosk-cluster-storage-worker-noefi OK     available      192.168.1.182 legacy   true

KUBECONFIG=kubeconfig kubectl -n mosk-ns get bmh cz7700-mosk-cluster-control-noefi -o yaml | less

Example of system response:

..
nics:
- ip: ""
  mac: 0c:c4:7a:1d:f4:a6
  model: 0x8086 0x10fb
  # discovered interfaces
  name: ens4f0
  pxe: false
  # temporary PXE address discovered from baremetal-mgmt
- ip: 172.16.170.30
  mac: 0c:c4:7a:34:52:04
  model: 0x8086 0x1521
  name: enp9s0f0
  pxe: true
  # duplicates temporary PXE address discovered from baremetal-mgmt
  # since we have fallback-bond configured on host
- ip: 172.16.170.33
  mac: 0c:c4:7a:34:52:05
  model: 0x8086 0x1521
  # discovered interfaces
  name: enp9s0f1
  pxe: false
...
storage:
- by_path: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
  model: Samsung SSD 850
  name: /dev/sda
  rotational: false
  sizeBytes: 500107862016
- by_path: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
  model: Samsung SSD 850
  name: /dev/sdb
  rotational: false
  sizeBytes: 500107862016
....

Create bare metal host profiles:

mosk-ns_BareMetalHostProfile_bmhp-cluster-default.yaml

apiVersion: metal3.io/v1alpha1
kind: BareMetalHostProfile
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: mosk-cluster
    # This label indicates that this profile will be default in
    # namespaces, so machines w\o exact profile selecting will use
    # this template
    kaas.mirantis.com/defaultBMHProfile: 'true'
    kaas.mirantis.com/provider: baremetal
  name: bmhp-cluster-default
  namespace: mosk-ns
spec:
  devices:
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
      minSize: 120Gi
      wipe: true
    partitions:
    - name: bios_grub
      partflags:
      - bios_grub
      size: 4Mi
      wipe: true
    - name: uefi
      partflags:
      - esp
      size: 200Mi
      wipe: true
    - name: config-2
      size: 64Mi
      wipe: true
    - name: lvm_dummy_part
      size: 1Gi
      wipe: true
    - name: lvm_root_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
      minSize: 30Gi
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-3
      minSize: 30Gi
      wipe: true
    partitions:
    - name: lvm_lvp_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-4
      wipe: true
  fileSystems:
  - fileSystem: vfat
    partition: config-2
  - fileSystem: vfat
    mountPoint: /boot/efi
    partition: uefi
  - fileSystem: ext4
    logicalVolume: root
    mountPoint: /
  - fileSystem: ext4
    logicalVolume: lvp
    mountPoint: /mnt/local-volumes/
  grubConfig:
    defaultGrubOptions:
    - GRUB_DISABLE_RECOVERY="true"
    - GRUB_PRELOAD_MODULES=lvm
    - GRUB_TIMEOUT=30
  kernelParameters:
    modules:
    - content: 'options kvm_intel nested=1'
      filename: kvm_intel.conf
    sysctl:
    # For the list of options prohibited to change, refer to
    # https://docs.mirantis.com/mke/3.7/install/predeployment/set-up-kernel-default-protections.html
      fs.aio-max-nr: '1048576'
      fs.file-max: '9223372036854775807'
      fs.inotify.max_user_instances: '4096'
      kernel.core_uses_pid: '1'
      kernel.dmesg_restrict: '1'
      net.ipv4.conf.all.rp_filter: '0'
      net.ipv4.conf.default.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.m-pub.rp_filter: '0'
      vm.max_map_count: '262144'
  logicalVolumes:
  - name: root
    size: 0
    vg: lvm_root
  - name: lvp
    size: 0
    vg: lvm_lvp
  postDeployScript: |
    #!/bin/bash -ex
    # used for test-debug only!
    echo "root:r00tme" | sudo chpasswd
    echo 'ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"' > /etc/udev/rules.d/60-ssd-scheduler.rules
    echo $(date) 'post_deploy_script done' >> /root/post_deploy_done

  preDeployScript: |
    #!/bin/bash -ex
    echo "$(date) pre_deploy_script done" >> /root/pre_deploy_done
  volumeGroups:
  - devices:
    - partition: lvm_root_part
    name: lvm_root
  - devices:
    - partition: lvm_lvp_part
    name: lvm_lvp
  - devices:
    - partition: lvm_dummy_part
    # here we can create lvm, but do not mount or format it somewhere
    name: lvm_forawesomeapp

mosk-ns_BareMetalHostProfile_worker-storage1.yaml

apiVersion: metal3.io/v1alpha1
kind: BareMetalHostProfile
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: mosk-cluster
    kaas.mirantis.com/provider: baremetal
  name: worker-storage1
  namespace: mosk-ns
spec:
  devices:
  - device:
      minSize: 120Gi
      wipe: true
    partitions:
    - name: bios_grub
      partflags:
      - bios_grub
      size: 4Mi
      wipe: true
    - name: uefi
      partflags:
      - esp
      size: 200Mi
      wipe: true
    - name: config-2
      size: 64Mi
      wipe: true
    # Create dummy partition w\o mounting
    - name: lvm_dummy_part
      size: 1Gi
      wipe: true
    - name: lvm_root_part
      size: 0
      wipe: true
  - device:
      # Will be used for Ceph, so required to be wiped
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
      minSize: 30Gi
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
      minSize: 30Gi
      wipe: true
    partitions:
    - name: lvm_lvp_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-3
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-4
      minSize: 30Gi
      wipe: true
    partitions:
      - name: lvm_lvp_part_sdf
        wipe: true
        size: 0
  fileSystems:
  - fileSystem: vfat
    partition: config-2
  - fileSystem: vfat
    mountPoint: /boot/efi
    partition: uefi
  - fileSystem: ext4
    logicalVolume: root
    mountPoint: /
  - fileSystem: ext4
    logicalVolume: lvp
    mountPoint: /mnt/local-volumes/
  grubConfig:
    defaultGrubOptions:
    - GRUB_DISABLE_RECOVERY="true"
    - GRUB_PRELOAD_MODULES=lvm
    - GRUB_TIMEOUT=30
  kernelParameters:
    modules:
    - content: 'options kvm_intel nested=1'
      filename: kvm_intel.conf
    sysctl:
    # For the list of options prohibited to change, refer to
    # https://docs.mirantis.com/mke/3.6/install/predeployment/set-up-kernel-default-protections.html
      fs.aio-max-nr: '1048576'
      fs.file-max: '9223372036854775807'
      fs.inotify.max_user_instances: '4096'
      kernel.core_uses_pid: '1'
      kernel.dmesg_restrict: '1'
      net.ipv4.conf.all.rp_filter: '0'
      net.ipv4.conf.default.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.m-pub.rp_filter: '0'
      vm.max_map_count: '262144'
  logicalVolumes:
  - name: root
    size: 0
    vg: lvm_root
  - name: lvp
    size: 0
    vg: lvm_lvp
  postDeployScript: |

    #!/bin/bash -ex

    # used for test-debug only! That would allow operator to logic via TTY.
    echo "root:r00tme" | sudo chpasswd
    # Just an example for enforcing "ssd" disks to be switched to use "deadline" i\o scheduler.
    echo 'ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"' > /etc/udev/   rules.d/60-ssd-scheduler.rules
    echo $(date) 'post_deploy_script done' >> /root/post_deploy_done

  preDeployScript: |
    #!/bin/bash -ex
    echo "$(date) pre_deploy_script done" >> /root/pre_deploy_done

  volumeGroups:
  - devices:
    - partition: lvm_root_part
    name: lvm_root
  - devices:
    - partition: lvm_lvp_part
    - partition: lvm_lvp_part_sdf
    name: lvm_lvp
  - devices:
    - partition: lvm_dummy_part
    name: lvm_forawesomeapp

Note

If you mount the /var directory, before configuring BareMetalHostProfile, review Mounting recommendations for the /var directory.

Create the L2Template objects:

Create the Subnet objects:

Create MetalLB configuration objects:

Create the PublicKey object for a MOSK cluster connection. For details, see PublicKey resource.

Create the Cluster object. For details, see Cluster resource.

Create the Machine objects linked to each bmh node. For details, see Machine resource.

Verify that the bmh nodes are in the provisioning state:

KUBECONFIG=kubectl kubectl -n mosk-ns get bmh  -o wide

Example of system response:

NAME                                  STATUS STATE          CONSUMER                                    BMC          BOOTMODE   ONLINE  ERROR
cz7700-mosk-cluster-control-noefi  OK     provisioning   cz7700-mosk-cluster-control-noefi-8bkqw  192.168.1.12  legacy     true
cz7741-mosk-cluster-control-noefi  OK     provisioning   cz7741-mosk-cluster-control-noefi-42tp2  192.168.1.76  legacy     true
cz7743-mosk-cluster-control-noefi  OK     provisioning   cz7743-mosk-cluster-control-noefi-8cwpw  192.168.1.78  legacy     true
...

Wait until all bmh nodes are in the provisioned state.

Verify that the lcmmachine phase has started:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get lcmmachines  -o wide

Example of system response:

NAME                                       CLUSTERNAME       TYPE      STATE   INTERNALIP     HOSTNAME                                         AGENTVERSION
cz7700-mosk-cluster-control-noefi-8bkqw mosk-cluster   control   Deploy  172.16.170.153 kaas-node-803721b4-227c-4675-acc5-15ff9d3cfde2   v0.2.0-349-g4870b7f5
cz7741-mosk-cluster-control-noefi-42tp2 mosk-cluster   control   Prepare 172.16.170.152 kaas-node-6b8f0d51-4c5e-43c5-ac53-a95988b1a526   v0.2.0-349-g4870b7f5
cz7743-mosk-cluster-control-noefi-8cwpw mosk-cluster   control   Prepare 172.16.170.151 kaas-node-e9b7447d-5010-439b-8c95-3598518f8e0a   v0.2.0-349-g4870b7f5
...

Verify that the lcmmachine phase is complete and the Kubernetes cluster is created:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get lcmmachines  -o wide

Example of system response:

NAME                                       CLUSTERNAME       TYPE     STATE  INTERNALIP      HOSTNAME                                        AGENTVERSION
cz7700-mosk-cluster-control-noefi-8bkqw  mosk-cluster  control  Ready  172.16.170.153  kaas-node-803721b4-227c-4675-acc5-15ff9d3cfde2  v0.2.0-349-g4870b7f5
cz7741-mosk-cluster-control-noefi-42tp2  mosk-cluster  control  Ready  172.16.170.152  kaas-node-6b8f0d51-4c5e-43c5-ac53-a95988b1a526  v0.2.0-349-g4870b7f5
cz7743-mosk-cluster-control-noefi-8cwpw  mosk-cluster  control  Ready  172.16.170.151  kaas-node-e9b7447d-5010-439b-8c95-3598518f8e0a  v0.2.0-349-g4870b7f5
...

Create the KaaSCephCluster object:

Obtain kubeconfig of the newly created MOSK cluster:

KUBECONFIG=kubeconfig kubectl -n mosk-ns get secrets mosk-cluster-kubeconfig -o jsonpath='{.data.admin\.conf}' | base64 -d |  tee mosk.kubeconfig

Verify the status of the Ceph cluster in your MOSK cluster:

KUBECONFIG=mosk.kubeconfig kubectl -n rook-ceph exec -it $(KUBECONFIG=mosk.kubeconfig kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph -s

Example of system response:

cluster:
  id:     e75c6abd-c5d5-4ae8-af17-4711354ff8ef
  health: HEALTH_OK
services:
  mon: 3 daemons, quorum a,b,c (age 55m)
  mgr: a(active, since 55m)
  osd: 3 osds: 3 up (since 54m), 3 in (since 54m)
data:
  pools:   1 pools, 32 pgs
  objects: 273 objects, 555 MiB
  usage:   4.0 GiB used, 1.6 TiB / 1.6 TiB avail
  pgs:     32 active+clean
io:
  client:   51 KiB/s wr, 0 op/s rd, 4 op/s wr