Example of a complete L2 templates configuration for cluster creation¶

The following example contains all required objects of an advanced network and host configuration for a baremetal-based managed cluster.

The procedure below contains:

Various .yaml objects to be applied with a managed cluster kubeconfig
Useful comments inside the .yaml example files
Example hardware and configuration data, such as network, disk, auth, that must be updated accordingly to fit your cluster configuration
Example templates, such as l2template and baremetalhostprofline, that illustrate how to implement a specific configuration

Caution

The exemplary configuration described below is not production ready and is provided for illustration purposes only.

For illustration purposes, all files provided in this exemplary procedure are named by the Kubernetes object types:

managed-ns_BareMetalHost_cz7700-managed-cluster-control-noefi.yaml
managed-ns_BareMetalHost_cz7741-managed-cluster-control-noefi.yaml
managed-ns_BareMetalHost_cz7743-managed-cluster-control-noefi.yaml
managed-ns_BareMetalHost_cz812-managed-cluster-storage-worker-noefi.yaml
managed-ns_BareMetalHost_cz813-managed-cluster-storage-worker-noefi.yaml
managed-ns_BareMetalHost_cz814-managed-cluster-storage-worker-noefi.yaml
managed-ns_BareMetalHost_cz815-managed-cluster-worker-noefi.yaml
managed-ns_BareMetalHostProfile_bmhp-cluster-default.yaml
managed-ns_BareMetalHostProfile_worker-storage1.yaml
managed-ns_Cluster_managed-cluster.yaml
managed-ns_KaaSCephCluster_ceph-cluster-managed-cluster.yaml
managed-ns_L2Template_bm-1490-template-controls-netplan-cz7700-pxebond.yaml
managed-ns_L2Template_bm-1490-template-controls-netplan.yaml
managed-ns_L2Template_bm-1490-template-workers-netplan.yaml
managed-ns_Machine_cz7700-managed-cluster-control-noefi-.yaml
managed-ns_Machine_cz7741-managed-cluster-control-noefi-.yaml
managed-ns_Machine_cz7743-managed-cluster-control-noefi-.yaml
managed-ns_Machine_cz812-managed-cluster-storage-worker-noefi-.yaml
managed-ns_Machine_cz813-managed-cluster-storage-worker-noefi-.yaml
managed-ns_Machine_cz814-managed-cluster-storage-worker-noefi-.yaml
managed-ns_Machine_cz815-managed-cluster-worker-noefi-.yaml
managed-ns_PublicKey_managed-cluster-key.yaml
managed-ns_cz7700-cred.yaml
managed-ns_cz7741-cred.yaml
managed-ns_cz7743-cred.yaml
managed-ns_cz812-cred.yaml
managed-ns_cz813-cred.yaml
managed-ns_cz814-cred.yaml
managed-ns_cz815-cred.yaml
managed-ns_Subnet_lcm-nw.yaml
managed-ns_Subnet_metallb-public-for-managed.yaml
managed-ns_Subnet_metallb-public-for-extiface.yaml
managed-ns_MetalLBConfig-lb-managed.yaml
managed-ns_MetalLBConfigTemplate-lb-managed-template.yaml
managed-ns_Subnet_storage-backend.yaml
managed-ns_Subnet_storage-frontend.yaml
default_Namespace_managed-ns.yaml

Caution

The procedure below assumes that you apply each new .yaml file using kubectl create -f <file_name.yaml>.

To create an example configuration for a managed cluster creation:

Verify that you have configured the following items:
1. All bmh nodes for PXE boot as described in Add a bare metal host using CLI
2. All physical NICs of the bmh nodes
3. All required physical subnets and routing
Create an empty .yaml file with the namespace object:
```
apiVersion: v1
```
Select from the following options:
Since Container Cloud 2.21.0 and 2.21.1 for MOSK 22.5
Create the required number of .yaml files with the BareMetalHostCredential objects for each bmh node with the unique name and authentication data. The following example contains one BareMetalHostCredential object:

Note

The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.
managed-ns_cz815-cred.yaml
apiVersion: kaas.mirantis.com/v1alpha1 kind: BareMetalHostCredential metadata: name: cz815-cred namespace: managed-ns labels: kaas.mirantis.com/region: region-one spec: username: admin password: value: supersecret
Before Container Cloud 2.21.0 or MOSK 22.5
Create the required number of .yaml files with the Secret objects for each bmh node with the unique name and authentication data. The following example contains one Secret object:
managed-ns_cz815-cred.yaml
apiVersion: v1 data: password: YWRtaW4= username: ZW5naW5lZXI= kind: Secret metadata: labels: kaas.mirantis.com/credentials: 'true' kaas.mirantis.com/provider: baremetal kaas.mirantis.com/region: region-one name: cz815-cred namespace: managed-ns

Create a set of files with the bmh nodes configuration:

Since Container Cloud 2.21.0 and 2.21.1 for MOSK 22.5

Before Container Cloud 2.21.0 or MOSK 22.5

Verify that the inspecting phase has started:

KUBECONFIG=kubeconfig kubectl -n managed-ns get bmh -o wide

Example of system response:

NAME                                       STATUS STATE CONSUMER BMC           BOOTMODE ONLINE ERROR REGION
cz7700-managed-cluster-control-noefi       OK     inspecting     192.168.1.12  legacy   true         region-one
cz7741-managed-cluster-control-noefi       OK     inspecting     192.168.1.76  legacy   true         region-one
cz7743-managed-cluster-control-noefi       OK     inspecting     192.168.1.78  legacy   true         region-one
cz812-managed-cluster-storage-worker-noefi OK     inspecting     192.168.1.182 legacy   true         region-one

Wait for inspection to complete. Usually, it takes up to 15 minutes.

Collect the bmh hardware information to create the l2template and bmh objects:

KUBECONFIG=kubeconfig kubectl -n managed-ns get bmh -o wide

Example of system response:

NAME                                       STATUS STATE CONSUMER BMC           BOOTMODE ONLINE ERROR REGION
cz7700-managed-cluster-control-noefi       OK     ready          192.168.1.12  legacy   true         region-one
cz7741-managed-cluster-control-noefi       OK     ready          192.168.1.76  legacy   true         region-one
cz7743-managed-cluster-control-noefi       OK     ready          192.168.1.78  legacy   true         region-one
cz812-managed-cluster-storage-worker-noefi OK     ready          192.168.1.182 legacy   true         region-one

KUBECONFIG=kubeconfig kubectl -n managed-ns get bmh cz7700-managed-cluster-control-noefi -o yaml | less

Example of system response:

..
nics:
- ip: ""
  mac: 0c:c4:7a:1d:f4:a6
  model: 0x8086 0x10fb
  # discovered interfaces
  name: ens4f0
  pxe: false
  # temporary PXE address discovered from baremetal-mgmt
- ip: 172.16.170.30
  mac: 0c:c4:7a:34:52:04
  model: 0x8086 0x1521
  name: enp9s0f0
  pxe: true
  # duplicates temporary PXE address discovered from baremetal-mgmt
  # since we have fallback-bond configured on host
- ip: 172.16.170.33
  mac: 0c:c4:7a:34:52:05
  model: 0x8086 0x1521
  # discovered interfaces
  name: enp9s0f1
  pxe: false
...
storage:
- by_path: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
  model: Samsung SSD 850
  name: /dev/sda
  rotational: false
  sizeBytes: 500107862016
- by_path: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
  model: Samsung SSD 850
  name: /dev/sdb
  rotational: false
  sizeBytes: 500107862016
....

Create bare metal host profiles:

managed-ns_BareMetalHostProfile_bmhp-cluster-default.yaml

apiVersion: metal3.io/v1alpha1
kind: BareMetalHostProfile
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: managed-cluster
    # This label indicates that this profile will be default in
    # namespaces, so machines w\o exact profile selecting will use
    # this template
    kaas.mirantis.com/defaultBMHProfile: 'true'
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
  name: bmhp-cluster-default
  namespace: managed-ns
spec:
  devices:
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
      minSize: 120Gi
      wipe: true
    partitions:
    - name: bios_grub
      partflags:
      - bios_grub
      size: 4Mi
      wipe: true
    - name: uefi
      partflags:
      - esp
      size: 200Mi
      wipe: true
    - name: config-2
      size: 64Mi
      wipe: true
    - name: lvm_dummy_part
      size: 1Gi
      wipe: true
    - name: lvm_root_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
      minSize: 30Gi
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-3
      minSize: 30Gi
      wipe: true
    partitions:
    - name: lvm_lvp_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-4
      wipe: true
  fileSystems:
  - fileSystem: vfat
    partition: config-2
  - fileSystem: vfat
    mountPoint: /boot/efi
    partition: uefi
  - fileSystem: ext4
    logicalVolume: root
    mountPoint: /
  - fileSystem: ext4
    logicalVolume: lvp
    mountPoint: /mnt/local-volumes/
  grubConfig:
    defaultGrubOptions:
    - GRUB_DISABLE_RECOVERY="true"
    - GRUB_PRELOAD_MODULES=lvm
    - GRUB_TIMEOUT=30
  kernelParameters:
    modules:
    - content: 'options kvm_intel nested=1'
      filename: kvm_intel.conf
    sysctl:
    # For the list of options prohibited to change, refer to
    # https://docs.mirantis.com/mke/3.7/install/predeployment/set-up-kernel-default-protections.html
      fs.aio-max-nr: '1048576'
      fs.file-max: '9223372036854775807'
      fs.inotify.max_user_instances: '4096'
      kernel.core_uses_pid: '1'
      kernel.dmesg_restrict: '1'
      net.ipv4.conf.all.rp_filter: '0'
      net.ipv4.conf.default.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.m-pub.rp_filter: '0'
      vm.max_map_count: '262144'
  logicalVolumes:
  - name: root
    size: 0
    vg: lvm_root
  - name: lvp
    size: 0
    vg: lvm_lvp
  postDeployScript: |
    #!/bin/bash -ex
    # used for test-debug only!
    echo "root:r00tme" | sudo chpasswd
    echo 'ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"' > /etc/udev/rules.d/60-ssd-scheduler.rules
    echo $(date) 'post_deploy_script done' >> /root/post_deploy_done

  preDeployScript: |
    #!/bin/bash -ex
    echo "$(date) pre_deploy_script done" >> /root/pre_deploy_done
  volumeGroups:
  - devices:
    - partition: lvm_root_part
    name: lvm_root
  - devices:
    - partition: lvm_lvp_part
    name: lvm_lvp
  - devices:
    - partition: lvm_dummy_part
    # here we can create lvm, but do not mount or format it somewhere
    name: lvm_forawesomeapp

managed-ns_BareMetalHostProfile_worker-storage1.yaml

apiVersion: metal3.io/v1alpha1
kind: BareMetalHostProfile
metadata:
  labels:
    cluster.sigs.k8s.io/cluster-name: managed-cluster
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
  name: worker-storage1
  namespace: managed-ns
spec:
  devices:
  - device:
      minSize: 120Gi
      wipe: true
    partitions:
    - name: bios_grub
      partflags:
      - bios_grub
      size: 4Mi
      wipe: true
    - name: uefi
      partflags:
      - esp
      size: 200Mi
      wipe: true
    - name: config-2
      size: 64Mi
      wipe: true
    # Create dummy partition w\o mounting
    - name: lvm_dummy_part
      size: 1Gi
      wipe: true
    - name: lvm_root_part
      size: 0
      wipe: true
  - device:
      # Will be used for Ceph, so required to be wiped
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-1
      minSize: 30Gi
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-2
      minSize: 30Gi
      wipe: true
    partitions:
    - name: lvm_lvp_part
      size: 0
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-3
      wipe: true
  - device:
      byPath: /dev/disk/by-path/pci-0000:00:1f.2-ata-4
      minSize: 30Gi
      wipe: true
    partitions:
      - name: lvm_lvp_part_sdf
        wipe: true
        size: 0
  fileSystems:
  - fileSystem: vfat
    partition: config-2
  - fileSystem: vfat
    mountPoint: /boot/efi
    partition: uefi
  - fileSystem: ext4
    logicalVolume: root
    mountPoint: /
  - fileSystem: ext4
    logicalVolume: lvp
    mountPoint: /mnt/local-volumes/
  grubConfig:
    defaultGrubOptions:
    - GRUB_DISABLE_RECOVERY="true"
    - GRUB_PRELOAD_MODULES=lvm
    - GRUB_TIMEOUT=30
  kernelParameters:
    modules:
    - content: 'options kvm_intel nested=1'
      filename: kvm_intel.conf
    sysctl:
    # For the list of options prohibited to change, refer to
    # https://docs.mirantis.com/mke/3.6/install/predeployment/set-up-kernel-default-protections.html
      fs.aio-max-nr: '1048576'
      fs.file-max: '9223372036854775807'
      fs.inotify.max_user_instances: '4096'
      kernel.core_uses_pid: '1'
      kernel.dmesg_restrict: '1'
      net.ipv4.conf.all.rp_filter: '0'
      net.ipv4.conf.default.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.k8s-ext.rp_filter: '0'
      net.ipv4.conf.m-pub.rp_filter: '0'
      vm.max_map_count: '262144'
  logicalVolumes:
  - name: root
    size: 0
    vg: lvm_root
  - name: lvp
    size: 0
    vg: lvm_lvp
  postDeployScript: |

    #!/bin/bash -ex

    # used for test-debug only! That would allow operator to logic via TTY.
    echo "root:r00tme" | sudo chpasswd
    # Just an example for enforcing "ssd" disks to be switched to use "deadline" i\o scheduler.
    echo 'ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"' > /etc/udev/   rules.d/60-ssd-scheduler.rules
    echo $(date) 'post_deploy_script done' >> /root/post_deploy_done

  preDeployScript: |
    #!/bin/bash -ex
    echo "$(date) pre_deploy_script done" >> /root/pre_deploy_done

  volumeGroups:
  - devices:
    - partition: lvm_root_part
    name: lvm_root
  - devices:
    - partition: lvm_lvp_part
    - partition: lvm_lvp_part_sdf
    name: lvm_lvp
  - devices:
    - partition: lvm_dummy_part
    name: lvm_forawesomeapp

Create the L2Template objects:

managed-ns_L2Template_bm-1490-template-controls-netplan-cz7700-pxebond.yaml

apiVersion: ipam.mirantis.com/v1alpha1
kind: L2Template
metadata:
  labels:
    bm-1490-template-controls-netplan-cz7700-pxebond: anymagicstring
    cluster.sigs.k8s.io/cluster-name: managed-cluster
    kaas.mirantis.com/provider: baremetal
    kaas.mirantis.com/region: region-one
  name: bm-1490-template-controls-netplan-cz7700-pxebond
  namespace: managed-ns
spec:
  ifMapping:
  - enp9s0f0
  - enp9s0f1
  - eno1
  - ens3f1
  l3Layout:
  - scope: namespace
    subnetName: lcm-nw
  - scope: namespace
    subnetName: storage-frontend
  - scope: namespace
    subnetName: storage-backend
  - scope: namespace
    subnetName: metallb-public-for-extiface
  npTemplate: |-
    version: 2
    ethernets:
      {{nic 0}}:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 0}}
        set-name: {{nic 0}}
        mtu: 1500
      {{nic 1}}:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 1}}
        set-name: {{nic 1}}
        mtu: 1500
      {{nic 2}}:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 2}}
        set-name: {{nic 2}}
        mtu: 1500
      {{nic 3}}:
        dhcp4: false
        dhcp6: false
        match:
          macaddress: {{mac 3}}
        set-name: {{nic 3}}
        mtu: 1500
    bonds:
      bond0:
        parameters:
          mode: 802.3ad
          #transmit-hash-policy: layer3+4
          #mii-monitor-interval: 100
        interfaces:
          - {{ nic 0 }}
          - {{ nic 1 }}
      bond1:
        parameters:
          mode: 802.3ad
          #transmit-hash-policy: layer3+4
          #mii-monitor-interval: 100
        interfaces:
          - {{ nic 2 }}
          - {{ nic 3 }}
    vlans:
      stor-f:
        id: 1494
        link: bond1
        addresses:
          - {{ip "stor-f:storage-frontend"}}
      stor-b:
        id: 1489
        link: bond1
        addresses:
          - {{ip "stor-b:storage-backend"}}
      m-pub:
        id: 1491
        link: bond0
    bridges:
      k8s-ext:
        interfaces: [m-pub]
        addresses:
          - {{ ip "k8s-ext:metallb-public-for-extiface" }}
      k8s-lcm:
        dhcp4: false
        dhcp6: false
        gateway4: {{ gateway_from_subnet "lcm-nw" }}
        addresses:
          - {{ ip "k8s-lcm:lcm-nw" }}
        nameservers:
          addresses: [ 172.18.176.6 ]
        interfaces:
          - bond0

Create the Subnet objects:

Create MetalLB configuration objects:

Since Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0):

Before Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0):

Before Container Cloud 2.24.0 (Cluster release 14.0.0):

Create the PublicKey object for a managed cluster connection. For details, see Public key resources.

Create the Cluster object. For details, see Cluster resources.

Create the Machine objects linked to each bmh node. For details, see Machine resources.

Verify that the bmh nodes are in the provisioning state:

KUBECONFIG=kubectl kubectl -n managed-ns get bmh  -o wide

Example of system response:

NAME                                  STATUS STATE          CONSUMER                                    BMC          BOOTMODE   ONLINE  ERROR REGION
cz7700-managed-cluster-control-noefi  OK     provisioning   cz7700-managed-cluster-control-noefi-8bkqw  192.168.1.12  legacy     true          region-one
cz7741-managed-cluster-control-noefi  OK     provisioning   cz7741-managed-cluster-control-noefi-42tp2  192.168.1.76  legacy     true          region-one
cz7743-managed-cluster-control-noefi  OK     provisioning   cz7743-managed-cluster-control-noefi-8cwpw  192.168.1.78  legacy     true          region-one
...

Wait until all bmh nodes are in the provisioned state.

Verify that the lcmmachine phase has started:

KUBECONFIG=kubeconfig kubectl -n managed-ns get lcmmachines  -o wide

Example of system response:

NAME                                       CLUSTERNAME       TYPE      STATE   INTERNALIP     HOSTNAME                                         AGENTVERSION
cz7700-managed-cluster-control-noefi-8bkqw managed-cluster   control   Deploy  172.16.170.153 kaas-node-803721b4-227c-4675-acc5-15ff9d3cfde2   v0.2.0-349-g4870b7f5
cz7741-managed-cluster-control-noefi-42tp2 managed-cluster   control   Prepare 172.16.170.152 kaas-node-6b8f0d51-4c5e-43c5-ac53-a95988b1a526   v0.2.0-349-g4870b7f5
cz7743-managed-cluster-control-noefi-8cwpw managed-cluster   control   Prepare 172.16.170.151 kaas-node-e9b7447d-5010-439b-8c95-3598518f8e0a   v0.2.0-349-g4870b7f5
...

Verify that the lcmmachine phase is complete and the Kubernetes cluster is created:

KUBECONFIG=kubeconfig kubectl -n managed-ns get lcmmachines  -o wide

Example of system response:

NAME                                       CLUSTERNAME       TYPE     STATE  INTERNALIP      HOSTNAME                                        AGENTVERSION
cz7700-managed-cluster-control-noefi-8bkqw  managed-cluster  control  Ready  172.16.170.153  kaas-node-803721b4-227c-4675-acc5-15ff9d3cfde2  v0.2.0-349-g4870b7f5
cz7741-managed-cluster-control-noefi-42tp2  managed-cluster  control  Ready  172.16.170.152  kaas-node-6b8f0d51-4c5e-43c5-ac53-a95988b1a526  v0.2.0-349-g4870b7f5
cz7743-managed-cluster-control-noefi-8cwpw  managed-cluster  control  Ready  172.16.170.151  kaas-node-e9b7447d-5010-439b-8c95-3598518f8e0a  v0.2.0-349-g4870b7f5
...

Create the KaaSCephCluster object:

Note

The storageDevices[].fullPath field is available since Container Cloud 2.25.0. For the clusters running earlier product versions, define the /dev/disk/by-id symlinks using storageDevices[].name instead.

Obtain kubeconfig of the newly created managed cluster:

KUBECONFIG=kubeconfig kubectl -n managed-ns get secrets managed-cluster-kubeconfig -o jsonpath='{.data.admin\.conf}' | base64 -d |  tee managed.kubeconfig

Verify the status of the Ceph cluster in your managed cluster:

KUBECONFIG=managed.kubeconfig kubectl -n rook-ceph exec -it $(KUBECONFIG=managed.kubeconfig kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph -s

Example of system response:

cluster:
  id:     e75c6abd-c5d5-4ae8-af17-4711354ff8ef
  health: HEALTH_OK
services:
  mon: 3 daemons, quorum a,b,c (age 55m)
  mgr: a(active, since 55m)
  osd: 3 osds: 3 up (since 54m), 3 in (since 54m)
data:
  pools:   1 pools, 32 pgs
  objects: 273 objects, 555 MiB
  usage:   4.0 GiB used, 1.6 TiB / 1.6 TiB avail
  pgs:     32 active+clean
io:
  client:   51 KiB/s wr, 0 op/s rd, 4 op/s wr