Bare Metal#

Mirantis k0rdent Enterprise can deploy managed clusters on bare metal servers using the Metal3 infrastructure provider. This implementation is based on the bare metal Cluster API Provider, Metal3 CAPM3, and provides out-of-tree (OOT) bare metal provisioning capabilities.

CAPM3 works by enabling you to add a representation of each bare metal server as a Kubernetes object. Mirantis k0rdent Enterprise can then assemble these machine objects into a cluster.

Structure#

The bare metal infrastructure provider is represented as a set of Helm charts. It includes the following charts:

baremetal-operator installs the Bare Metal Operator
capm3-crds installs the CustomResourceDefinition objects for the Metal3 CAPM3 and IPAM components
cluster-api-provider-metal3 installs the Metal3 CAPM3 provider and Metal3 IP Address Manager
ironic installs OpenStack Ironic and accompanying components needed for management of bare metal machines: MariaDB, keepalived, HTTP server, DHCP server, TFTP server, NTP server, dynamic-IPXE controller, resource controller.

Prerequisites#

An installed Mirantis k0rdent Enterprise management cluster. Follow the instructions in Install Mirantis k0rdent Enterprise to create a management cluster with Mirantis k0rdent Enterprise running. Prepare this cluster according to the Metal3 host configuration guide.
Supported hardware as documented in the Metal3 hardware compatibility guide
You should still have Helm installed from your installation of Mirantis k0rdent Enterprise. If not, install it again.

Prepare Mirantis k0rdent Enterprise for Bare Metal clusters#

Follow these instructions to make Mirantis k0rdent Enterprise capable of deploying bare metal clusters:

1. Create the required objects for the OOT CAPM3 provider#

Create the necessary Kubernetes objects to install the out-of-tree CAPM3 provider. Just as with other providers, these include a HelmRepository, ProviderTemplate, and ClusterTemplate.

kubectl create -f - <<EOF
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: oot-capm3-repo
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/managed: "true"
spec:
  type: oci
  url: 'oci://registry.mirantis.com/k0rdent-bm/charts/'
  interval: 10m0s
---
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ProviderTemplate
metadata:
  name: cluster-api-provider-metal3-0-4-0
  annotations:
    helm.sh/resource-policy: keep
spec:
  helm:
    chartSpec:
      chart: cluster-api-provider-metal3
      version: 0.4.0
      interval: 10m0s
      sourceRef:
        kind: HelmRepository
        name: oot-capm3-repo
---
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterTemplate
metadata:
  annotations:
    helm.sh/resource-policy: keep
  labels:
    k0rdent.mirantis.com/component: kcm
  name: capm3-standalone-cp-0-4-0
  namespace: kcm-system
spec:
  helm:
    chartSpec:
      chart: capm3-standalone-cp
      version: 0.4.0
      interval: 10m0s
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: oot-capm3-repo
EOF

2. Verify the `ProviderTemplate` is valid#

Check that the ProviderTemplate has been created successfully:

kubectl get providertemplates cluster-api-provider-metal3-0-4-0

NAME                              VALID
cluster-api-provider-metal3-0-4-0 true

3. Configure the `Management` object#

Edit the Management object to add the CAPM3 provider configuration:

kubectl edit managements.k0rdent.mirantis.com

Add the following configuration to the providers section:

- name: cluster-api-provider-metal3
  template: cluster-api-provider-metal3-0-4-0
  config:
    global:
      ironic:
        enabled: true # networking configuration ("ironic.networking" section) should be defined prior to enabling ironic
    ironic:
      networking:
        dhcp: # used by DHCP server to assign IPs to hosts during PXE boot
          rangeBegin: <DHCP_RANGE_START>      # e.g., 10.0.1.51
          rangeEnd: <DHCP_RANGE_END>          # e.g., 10.0.1.55
          netmask: <DHCP_SUBNET_MASK>         # e.g., 255.255.255.192 (default is 255.255.255.0)
          options:                            # DHCP options, used during PXE boot and by IPA
            - "option:router,<ROUTER_IP>"     # e.g., 10.0.1.1. It's a mandatory option.
            - "option:dns-server,<DNS_IP[,DNS2_IP...]>" # can be set to KEEPALIVED_VIP (dnsmasq can serve as a DNS server with user-defined DNS records) or to IP of your preferred server. Optional.
            - "option:ntp-server,<NTP_IP>"    # can be set to KEEPALIVED_VIP (internal ntp server) or to IP of your preferred server. That ntp server will be used on PXE boot stage then. Optional.
        interface: <PROVISION_INTERFACE>      # e.g., bond0 - interface of the management cluster node connected to BM hosts provision network
        ipAddress: <KEEPALIVED_VIP>          # e.g., 10.0.1.50 - keepalived VIP for DHCP server and Ironic services. This VIP will be configured on the <PROVISION_INTERFACE>, it must be in the same L3 network as DHCP range if no dhcp-relay used between management cluster and child cluster hosts.
    # Default (optional) values
    # You can define custom target OS and IPA images here if needed by defining the resources:
      images_ipa:
      - checksum: ff8c3caad212bd1f9ac46616cb8d7a2646ed0da85da32d89e1f5fea5813265f8
        name: ironic-python-agent_x86_64.initramfs
        url: https://get.mirantis.com/k0rdent-enterprise/bare-metal/ipa/ipa-centos9-stream-2025-12-01-11-53-35-amd64.initramfs
      - name: ironic-python-agent_x86_64.kernel
        checksum: e28f0a2185b618efb44e8f24ea806ec775cfc2f2446816da215ffa97a811e9af
        url: https://get.mirantis.com/k0rdent-enterprise/bare-metal/ipa/ipa-centos9-stream-2025-12-01-11-53-35-amd64.kernel
      images_target:
      - checksum: f0a5da9499adaaca6249792df25032430f33f0130eddf39433782b5362057b99
        name: ubuntu-24.04-server-cloudimg-20251118-amd64.img
        url: https://get.mirantis.com/k0rdent-enterprise/bare-metal/targetimages/ubuntu-24.04-server-cloudimg-20251118-amd64.img

4. Wait for the `Management` object to be ready#

Monitor the Management object status:

kubectl get managements.k0rdent.mirantis.com -w

This process usually takes up to 5 minutes. If the Management object doesn't become Ready, refer to the Troubleshooting section.

5. Verify the `ClusterTemplate` is valid#

Check that the ClusterTemplate has been created successfully:

kubectl -n kcm-system get clustertemplates capm3-standalone-cp-0-4-0

NAME                        VALID
capm3-standalone-cp-0-4-0   true

6. Optional. Tune the DHCP server.#

Warning

Modification of this configuration should be done with special care. It's not recommended to change it during provisioning/deprovisioning of bare metal machines.

After CAPM3 provider is deployed, you can reconfigure DHCP server. Using dnsmasq object, you can change configuration of the DHCP server and monitor DHCP leases related to your bare metal machines.

Note

Modification of this configuration requires good knowledge of DHCP basics.

kubectl -n kcm-system edit dnsmasq

7. Optional. Use `MetalLB` to advertise services of a bare metal management cluster#

MetalLB is a popular load-balancer implementation for bare metal Kubernetes clusters, and you can use it on your bare metal management cluster. Cloud providers often have their own LB solutions. If you use a cloud provider to host your management cluster, check the compatibility with your cloud provider. Before installation, please ensure the requirements are met, and check the preparation instructions.

To install MetalLB on your bare metal management cluster, use one of the methods recommended in the documentation. For example, it can be installed using a Helm chart:

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb -n metallb-system --create-namespace

For advanced installation options, see the upstream documentation.

After that, you can observe the process of MetalLB installation:

KUBECONFIG=<child-cluster-config> kubectl -n metallb-system get all -w

After the MetalLB pods are created, you can create the CustomResource objects that contain the MetalLB configuration.

A very simple configuration using ARP advertisements can be defined using the following objects:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: my-address-pool
  namespace: metallb-system
spec:
  addresses:
  - <SERVICE_ADDRESS_RANGE> # e.g., 199.177.1.24-199.177.1.51
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertise-all
  namespace: metallb-system

where SERVICE_ADDRESS_RANGE is the range of IP addresses that are controlled by MetalLB and are assigned by MetalLB to load-balanced Kubernetes services as their external IPs.

Note

The above objects can be created on the management cluster only after MetalLB components were installed on that cluster.

Warning

Ensure that the IPs that you define in IPAddressPool.metallb.io objects are not overlapped with any host IPs of the cluster nodes, management cluster k8s API VIP, any IPs that are used or controlled (for example, DHCP ranges) by CAPM3 provider, other IPs that should be accessible from your management cluster (for example, externally accessible IPs of planned child clusters).

Please see MetalLB configuration for more information.

Enroll bare metal machines#

The next step is to create BareMetalHost objects to represent your bare metal machines so Mirantis k0rdent Enterprise can manage them. For each bare metal machine, create two objects: a Secret and a BareMetalHost. For detailed instructions, see the Metal3 BareMetalHost enrollment guide (just Enrolling, not Provisioning), or follow these instructions.

Note

You don't need to provision bare metal hosts at this stage. Provisioning should happen later as part of a cluster deployment.

Create credential Secret objects

You need to provide BMC credentials for every BareMetalHost using Secret objects. For example:
```
apiVersion: v1
kind: Secret
metadata:
  name: <BMH_NAME>-bmc-secret
  namespace: <NAMESPACE>
type: Opaque
data:
  username: <BASE64_ENCODED_BMC_USERNAME>
  password: <BASE64_ENCODED_BMC_PASSWORD>
```
Note

namespace of all the objects used to describe bare metal machines and corresponding cluster must be equal to the namespace of the ClusterTemplate object used for deployment of that cluster.

Create BareMetalHost objects

A BareMetalHost object represents the physical machine. It contains a reference to the Secret created above. For example:

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: <BMH_NAME>
  namespace: <NAMESPACE>
spec:
  online: true
  bmc:
    address: <BMC_ADDRESS>  # e.g., 192.168.1.100:623
    credentialsName: <BMH_NAME>-bmc-secret
    #disableCertificateVerification: true # only needed when using redfish protocol
  bootMACAddress: <MAC_ADDRESS> # MAC address that is used for booting. It’s a MAC address of an actual NIC of the host, not the BMC MAC address.
  #bootMode: legacy # UEFI or legacy BIOS. UEFI is the default and should be used unless there are serious reasons not to.

One of the two remote management protocols must be supported by a BMC to get it work with Mirantis k0rdent Enterprise bare metal infrastructure provider. See Metal3 documentation for details.

IPMI protocol.

IPMI is the oldest and by far the most widely available remote management protocol.

There are notes and examples of setting bmc.address in the table below.

BMC address format & examples	Notes
`ipmi://<host>:<port>`	Port is optional, defaults to 623.
`ipmi://1.2.3.4`
`<host>:<port>`	IPMI is the default protocol in Metal3.
`1.2.3.4:623`
`1.2.3.4`

Note

Only network boot over iPXE is supported for IPMI.

Redfish protocol.

Before using Redfish, please read the related Metal3 documentation carefully and ensure that your hardware is supported and has the required licences.

There are notes and examples of setting bmc.address for different vendors and boot methods in the table below.

Technology	Boot method	BMC address format & examples	Notes
Generic Redfish	iPXE	`redfish://<host>:<port>/<systemID>`
		`redfish://1.2.3.4/redfish/v1/Systems/1`
	Virtual media	`redfish-virtualmedia://<host>:<port>/<systemID>`	Must not be used for Dell machines.
		`redfish-virtualmedia://1.2.3.4/redfish/v1/Systems/1`
Dell iDRAC 8+	iPXE	`idrac-redfish://<host>:<port>/<systemID>`
		`idrac-redfish://1.2.3.4/redfish/v1/Systems/System.Embedded.1`
	Virtual media	`idrac-virtualmedia://<host>:<port>/<systemID>`	Requires firmware v6.10.30.00+ for iDRAC 9, v2.75.75.75+ for iDRAC 8.
		`idrac-virtualmedia://1.2.3.4/redfish/v1/Systems/System.Embedded.1`
HPE iLO 5 and 6	iPXE	`ilo5-redfish://<host>:<port>/<systemID>`	An alias of redfish for convenience. RAID management only on iLO 6.
		`ilo5-redfish://1.2.3.4/redfish/v1/Systems/1`
	Virtual media	`ilo5-virtualmedia://<host>:<port>/<systemID>`	An alias of redfish for convenience. RAID management only on iLO 6.
		`ilo5-virtualmedia://1.2.3.4/redfish/v1/Systems/1`

For information on Redfish interoperability please check the Metal3 Documentation.

bmc.disableCertificateVerification (useful with Redfish only) can be set to true to skip certificate validation for https connection between a BMC and the management cluster.

Note

Redfish support for Mirantis k0rdent Enterprise with bare metal infrastructure provider has been tested on a limited number of vendors and protocols. Please contact Mirantis for detailed information.

Note

Redfish/VirtualMedia support in DHCP-less mode has not been tested and is current not automated. It requires manual steps for every host described in upstream docs.

Provisioning states

During its lifetime, a BareMetalHost resource goes through a series of various states. status.provisioning.state is the current phase of the provisioning process, and it can be one of the following:

Creating: Newly created hosts get an empty provisioning state briefly before moving either to unmanaged or registering.
Unmanaged: An unmanaged host is missing both the BMC address and credentials secret name, and does not have any information to access the BMC for registration.
Registering: The host will stay in the registering state while the BMC access details are being validated.
Inspecting: After the host is registered, an IPA ramdisk will be booted on it. The agent collects information about the available hardware components and sends it back to Metal3. The host will stay in the inspecting state until this process is completed.
Preparing: When setting up RAID or changing firmware settings, the host will be in preparing state.
Available: A host in the available state is ready to be provisioned. It will move to the provisioning state once the image field is populated.
Provisioning: While an image is being copied to the host, and the host is configured to run the image, the host will be in the provisioning state.
Provisioned: After an image is copied to the host and the host is running the image, it will be in the provisioned state.
Deprovisioning: When the previously provisioned image is being removed from the host, it will be in the deprovisioning state.
Powering off before delete: When the host that is not currently unmanaged is marked to be deleted, it will be powered off first and will stay in the powering off before delete until it’s done or until the retry limit is reached.
Deleting: When the host is marked to be deleted and has been successfully powered off, it will move from its current state to deleting, at which point the resource record is deleted.

Wait for BareMetalHost objects to complete enrollment

Monitor your BareMetalHost objects until they are available:

kubectl get bmh -n <NAMESPACE>

NAME      STATE       CONSUMER   ONLINE   ERROR   AGE
child-1   available              true             4d17h
child-2   available              true             4d17h
child-3   available              true             4d17h

Optional. Specify root device on BareMetalHost objects

Bare-metal hosts often have more than one block device, and in many cases a user will want to specify which of them to use as the root device. Root device hints enable users to select one device or a group of devices to choose from. You can provide these hints via the spec.rootDeviceHints field on the BareMetalHost object. For example:
```
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: <BMH_NAME>
  namespace: <NAMESPACE>
spec:
  ...
  rootDeviceHints:
    wwn: "0x9876543210fedcba"
```
You can also specify rootDeviceHints before inspection if you already know block device identifiers. The information about storage devices is available in BareMetalHost.status after inspection.

Warning

Block device names are not guaranteed to be consistent across reboots. If possible, choose a more reliable hint, such as wwn or serialNumber.

For more information about rootDeviceHints, see https://book.metal3.io/bmo/root_device_hints.

Baremetal host OS update#

Mirantis k0rdent Enterprise relies on cluster-api machinery to manage baremetal hosts and related components, so any cluster-api peculiarities and limits regarding manipulation of software components running on managed hosts also apply to Mirantis k0rdent Enterprise. From the k0rdent/cluster-api point of view, baremetal hosts in a provisioned state are mostly immutable; k0rdent/cluster-api can upgrade/downgrade the k0s version on the host. Other changes that go though Kubernetes API can be performed as well via manipulation of Kubernetes objects, but these capabilities are not enough to manage all software/OS changes on the host. As such, the only possible way to fully manage software/OS on the host is to deprovision the host -- that is, to detach the host from the existing cluster and clean it in advance for future usage or allocation.

So to make any modification of software running on a provisioned host you need to deprovision it and re-provision it using another image containing the required software changes/updates. It is possible to reuse the existing image if your cloud-init uset-data scripts are capable enough to install all required changes/updates during the provisioning stage, ut either way you will need to perform the deprovision and provision circle on all required hosts to apply changes to their software.

There are both pros and cons to this process, including:

Pro: It is easy to understand, and it is predictable
Pro: It does not depend on machine-specific peculiarities such as the OS, bootstrap mechanism, and so on
Con: It is time-consuming; even a small update requires the deprovisioning/provisioning cycle
Con: During host deprovisioning, all block devices will be erased (not just the partition table but also filesystem related areas) so it will not be possible to do fast data recovery using the sdisk --dump /dev/disk | sdisk /dev/disk process
Con: It requires a custom image for each update (unless the user-data scripts are capable of doing the required update)

So from the Mirantis k0rdent Enterprise cluster admin perspective, the host OS/software update processes will look something like this:

Prepare a new host image with the updated software
Upload the image to a web server accessible from the host's PXE network
Edit the ClusterDeployment object:
```
kubectl edit ClusterDeployment subject-generic-0
```
You need to change the image value for the following keys: * spec.config.controlPlane.image and spec.config.controlPlane.checksum (to update control plane hosts) * spec.config.worker.image and spec.config.worker.checksum (to update worker hosts)
The above changes to the ClusterDeployment object will not affect the deployed cluster hosts. You need to initiate the redeployment of the target hosts manually. To do this, locate the machine object that corresponds to the host(s) you are going to update:
```
$ kubectl -n kcm-system get machine
NAME                               CLUSTER             NODENAME   PROVIDERID                                                            PHASE          AGE   VERSION
subject-generic-0-cp-templ-0       subject-generic-0              metal3://kcm-system/child-master-1/subject-generic-0-cp-templ-0       Provisioning   15h   v1.32.3+k0s.0
subject-generic-0-md-zvj4w-5q8jw   subject-generic-0              metal3://kcm-system/child-worker-3/subject-generic-0-md-zvj4w-5q8jw   Provisioning   13h   v1.32.3
subject-generic-0-md-zvj4w-d95cq   subject-generic-0              metal3://kcm-system/child-worker-2/subject-generic-0-md-zvj4w-d95cq   Provisioning   15h   v1.32.3
```
You can easily map the BareMetalHost object name to the Machine object using data in the "ProviderID" column. So if we need to update the child-worker-2 host, we need to remove the corresponding machine object, in this case the subject-generic-0-md-zvj4w-d95cq object.

So we need to do kubectl delete machine subject-generic-0-md-zvj4w-d95cq. This command deprovisions the child-worker-2 host, and Mirantis k0rdent Enterprise allocates and provisions a new BareMetalHost(possibly even the same physical machine, but not necessarily).

If we perform this deprovisioning process (that is, remove the machine object) on all/many hosts at once, we can interrupt workloads on the child cluster, because there will be not enough hosts to serve/manage them. In other words, you should keep in mind how many hosts will be missing from the child cluster at one time. Also, never remove the last/only control plane node. Instead, scale up the cluster and add an extra control node if you have only one.

Another way to perform host reprovisioning is to use the clusterctl utility, as in:

clusterctl alpha rollout restart machinedeployment/subject-generic-0-md

Alternatively, you can perform this OS/software update process on bare metal hosts in a completely different way. You can configure the host via SSH access using well-known existing software management tools such as Ansible, SaltStack, Chef, Puppet or any other tool. The initial SSH access can be configured inside ClusterDeployment object in the following way:

spec:
  config:
    controlPlane:
      preStartCommands:
        - chown ubuntu /home/ubuntu/.ssh/authorized_keys # allow "ubuntu" user with ssh key
      files:
        - path: /home/ubuntu/.ssh/authorized_keys # ssh key for "ubuntu" user
          permissions: "0600"
          content: "ssh-rsa <ssh public key data>"

The same configuration chunk can be used for worker nodes. It must be placed into spec.config.worker parameters.

To get more details about the host's OS update process, see the following documents:

https://cluster-api.sigs.k8s.io/clusterctl/commands/alpha-rollout - clusterctl rollout command
https://github.com/kubernetes-sigs/cluster-api/issues/12291 - inplace update proposal

Manage block device partitions and filesystems#

The only way Mirantis k0rdent Enterprise can operate with a target host's block devices is via the cloud-init tool (through k0smotron -> capi -> metal3 -> ironic). So Mirantis k0rdent Enterprise block device partitioning capabilities are equal to the cloud-init partitioning capabilities. Mirantis k0rdent Enterprise (and all other intermediate layers) are only responsible for passing a partitioning scheme from the Mirantis k0rdent Enterprise API objects into cloud-init.

The capm3-standalone-cp ClusterTemplate allows passing raw chunks of UserData:

controlPlane.customUserData: A raw UserData chunk added to the UserData used to configure control-plane hosts
worker.customUserData: A raw UserData chunk added to the UserData used to configure worker hosts

This feature can be used to configure block device partitioning. Below is the stub of a ClusterDeployment object that contains basic block device partitions management:

apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: partitioning-example
spec:
  template: capm3-standalone-cp-0-4-0
  credential: capm3-stub-credential
  dryRun: false
  config:
    ...: ...
    controlPlane:
      ...: ...
      customUserData: &userData | 
        disk_setup:
          /dev/sdb:
            layout: [ 10, 10, 30, 50 ]
            overwrite: true
            table_type: gpt
        fs_setup:
          - { device: /dev/sdb1, filesystem: swap, label: swap }
          - { device: /dev/sdb2, filesystem: ext4, label: fs1 }
          - { device: /dev/sdb3, filesystem: ext4, label: fs2 }
          - { device: /dev/sdb4, filesystem: ext4, label: fs3 }
        mounts:
          - [ /dev/sdb1, none, swap, sw, '0', '0' ]
          - [ /dev/sdb2, /mnt/fs1 ]
          - [ /dev/sdb3, /mnt/fs2 ]
          - [ /dev/sdb4, /mnt/fs3 ]
    controlPlaneNumber: 1
    worker:
      ...: ...
      customUserData: *userData
    workersNumber: 1

The set of filesystems supported by cloud-init depends on the content of the ironic-python-agent (IPA) image. This image must include mkfs.<filesystem> (mkfs.ext4 for example) tools in a directory listed in the $PATH environment variable. This approach enables the operator to add (in advance) all required filesystem management tools into the IPA image.

All details regarding allowed cloud-init options can be found in the cloud-init documentation.

Support for RAID disk arrays on baremetal hosts#

The Ironic python agent (IPA) provides limited functionality for software RAID arrays. These limits do not allow creating a RAID disk as rootfs. One limit is the requirement to use a whole block device as a RAID array part. During RAID initialization, IPA creates a single partition on target disks and uses it as part of the RAID array. For example, the RAID array will be assembled not from the raw block device, but from the full disk partition on this block device.

The disk layout looks something like this:

disk0 -> part0 -,
disk1 -> part0 -+-> RAID -> [ p0, p1, p2 ... ]

So UEFI/BIOS has access only to the "root" partition, and it cannot access the EFI/boot partition defined in the "nested" partition table--that is, the partition table created inside part0 (inside the RAID array). So if the RAID array is defined as the root device, the system will be unbootable.

Consider this example of a BareMetalHost object that declares the RAID1 array:

---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: child-master-1
spec:
  bmc:
    address: ipmi://10.0.1.1:6234
    credentialsName: child-master-1.ipmi-auth
  bootMACAddress: 52:54:1f:8b:19:15
  # bootMode: legacy
  online: true
  raid:
    softwareRAIDVolumes:
      - level: "1"
        physicalDisks:
          - deviceName: /dev/disk/by-path/pci-0000:00:07.0-scsi-0:0:0:1
          - deviceName: /dev/disk/by-path/pci-0000:00:07.0-scsi-0:0:0:2
  rootDeviceHints:
    deviceName: /dev/disk/by-path/pci-0000:00:07.0-scsi-0:0:0:0

The host has three hard disks. The first one will be used as the root device, and the second and third disks will be assembled into the RAID1 array. The rootDeviceHint in the BareMetalHost spec must be defined, because if it has a RAID definition and doesn't have the rootDeviceHint, the first RAID array will be marked as the root device automatically.

For more information about software RAID support in IPA, see the Ironic documentation.

Create a bare metal cluster#

You need to create several objects before Mirantis k0rdent Enterprise can create a bare metal cluster.

Create the credential objects

Since CAPM3 doesn't require cloud credentials, create dummy Secret and Credential objects to satisfy ClusterDeployment requirements:

apiVersion: v1
kind: Secret
metadata:
  name: capm3-cluster-secret
  namespace: <NAMESPACE>
  labels:
    k0rdent.mirantis.com/component: "kcm"
type: Opaque
---
apiVersion: k0rdent.mirantis.com/v1beta1
kind: Credential
metadata:
  name: capm3-stub-credential
  namespace: <NAMESPACE>
spec:
  description: CAPM3 Credentials
  identityRef:
    apiVersion: v1
    kind: Secret
    name: capm3-cluster-secret
    namespace: <NAMESPACE>

Create the ConfigMap resource-template

Create an empty ConfigMap resource-template:

apiVersion: v1
kind: ConfigMap
metadata:
  name: capm3-cluster-credential-resource-template
  namespace: <NAMESPACE>
  labels:
    k0rdent.mirantis.com/component: "kcm"
  annotations:
    projectsveltos.io/template: "true"

Deploy a test cluster

Create a ClusterDeployment template to deploy a cluster using your bare metal machines. Start with a capm3-example.yaml file. This one creates a cluster with 1 control node and 2 workers:

apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
metadata:
  name: capm3-example
  namespace: <NAMESPACE>
spec:
  template: capm3-standalone-cp-0-4-0
  credential: capm3-stub-credential
  dryRun: false
  config:
    clusterAnnotations: {}
    clusterLabels: {}
    clusterNetwork:
      pods:
        cidrBlocks:
          - 10.243.0.0/16
      services:
        cidrBlocks:
          - 10.95.0.0/16
    controlPlane:
      # Available since 0.4.0
      # Automated node cleaning mode: 'metadata' or 'disabled'
      automatedCleaningMode: metadata
      # Available since 0.4.0
      # Re-use the same BaremetalHosts during deprovisioning and provisioning
      nodeReuse: false
      # Default target OS image
      checksum: f0a5da9499adaaca6249792df25032430f33f0130eddf39433782b5362057b99
      image: http://<IRONIC_HTTP_ENDPOINT>:6180/images/ubuntu-24.04-server-cloudimg-20251118-amd64.img
      keepalived:
        authPass: <VRRP_PASSWORD> # optional, from 4 to 8 letters
        enabled: true
        virtualIP: <CLUSTER_API_VIP>/<SUBNET_PREFIX>  # e.g., 10.0.1.70/24. must match k0s.api.externalAddress
      preStartCommands:
        - sudo useradd -G sudo -s /bin/bash -d /home/user1 -p $(openssl passwd -1 myuserpass) user1 # define your user here. it can be used e.g. for debugging.
        - sudo apt update # for Ubuntu
        - sudo apt install jq -y # for Ubuntu
        #- sudo dnf makecache # for RedHat
        #- sudo dnf install jq -y # for RedHat
        # jq is used in K0sControlPlane object to parse cloud-init data that is required for Metal3 provider
      files:
        - path: /home/user1/.ssh/authorized_keys
          permissions: "0644"
          content: "<SSH_PUBLIC_KEY>"
    controlPlaneNumber: 1
    dataTemplate:
      metaData:
        ipAddressesFromIPPool:
          - key: provisioningIP
            name: pool-pxe
        objectNames:
          - key: name
            object: machine
          - key: local-hostname
            object: machine
          - key: local_hostname
            object: machine
        prefixesFromIPPool:
          - key: provisioningCIDR
            name: pool-pxe
      networkData:
        links:
          ethernets:
            - id: <INTERFACE_NAME>  # e.g., ens3
              type: phy
              macAddress:
                fromHostInterface: <INTERFACE_NAME>
        networks:
          ipv4:
            - id: pxe
              ipAddressFromIPPool: pool-pxe
              link: <INTERFACE_NAME>
              routes:
              - gateway:
                  fromIPPool: pool-pxe
                network: 0.0.0.0
                prefix: 0
        services:
          dns:
            - <DNS_SERVER_IP>   # e.g., 8.8.8.8
    ipPools:
      - name: pool-pxe
        pools:
          - end: <IP_POOL_END>      # e.g., 10.0.1.65
            gateway: <GATEWAY_IP>   # e.g., 10.0.1.1
            prefix: <SUBNET_PREFIX> # e.g, 24
            start: <IP_POOL_START>  # e.g., 10.0.1.61
    k0s:
      api:
        externalAddress: <CLUSTER_API_VIP>  # e.g., 10.0.1.70
      telemetry:
        enabled: false
      version: v1.32.6+k0s.0
    worker:
      # Available since 0.4.0
      # Automated node cleaning mode: 'metadata' or 'disabled'
      automatedCleaningMode: metadata
      # Available since 0.4.0
      # Re-use the same BaremetalHosts during deprovisioning and provisioning
      nodeReuse: false   
      # Default target OS image
      checksum: f0a5da9499adaaca6249792df25032430f33f0130eddf39433782b5362057b99
      image: http://<IRONIC_HTTP_ENDPOINT>:6180/images/ubuntu-24.04-server-cloudimg-20251118-amd64.img
      preStartCommands:
        - sudo useradd -G sudo -s /bin/bash -d /home/user1 -p $(openssl passwd -1 myuserpass) user1 # define your user here. it can be used e.g. for debugging.
        - sudo apt update # for Ubuntu
        - sudo apt install jq -y # for Ubuntu
        #- sudo dnf makecache # for RedHat
        #- sudo dnf install jq -y # for RedHat
        # jq is used in K0sControlPlane object to parse cloud-init data that is required for Metal3 provider
      files:
        - path: /home/user1/.ssh/authorized_keys
          permissions: "0644"
          content: "<SSH_PUBLIC_KEY>"
    workersNumber: 2

Create a ClusterDeployment object on your management cluster from the YAML file:

kubectl create -f capm3-example.yaml

Monitor the provisioning process

Watch the BareMetalHost objects as they transition through provisioning states:

kubectl -n <NAMESPACE> get bmh -w

You should see the hosts transition from available to provisioning to provisioned:

NAME      STATE         CONSUMER                     ONLINE   ERROR   AGE
child-1   available                                  true             16m
child-2   available                                  true             16m
child-3   available                                  true             16m
child-2   provisioning  capm3-example-cp-templ-0     true             16m
child-2   provisioned   capm3-example-cp-templ-0     true             18m
child-1   provisioning  capm3-example-md-txr9f-k8z9d true             18m
child-3   provisioning  capm3-example-md-txr9f-lkc5c true             18m
child-3   provisioned   capm3-example-md-txr9f-lkc5c true             21m
child-1   provisioned   capm3-example-md-txr9f-k8z9d true             21m

Also monitor the Metal3Machine objects that are part of the cluster:

kubectl -n <NAMESPACE> get metal3machine -w

NAME                           AGE     PROVIDERID                                               READY   CLUSTER        PHASE
capm3-example-cp-templ-0       5m17s   metal3://kcm-system/child-2/capm3-example-cp-templ-0     true    capm3-example
capm3-example-md-txr9f-k8z9d   2m40s   metal3://kcm-system/child-1/capm3-example-md-txr9f-k8z9d true    capm3-example
capm3-example-md-txr9f-lkc5c   2m40s   metal3://kcm-system/child-3/capm3-example-md-txr9f-lkc5c true    capm3-example

Access the deployed cluster

Once the first control plane machine is ready, retrieve the KUBECONFIG for your new cluster:

kubectl get secret -n <NAMESPACE> capm3-example-kubeconfig -o jsonpath='{.data.value}' | base64 -d > capm3-example-kubeconfig
KUBECONFIG="capm3-example-kubeconfig" kubectl get pods -A

Cleanup

To clean up bare metal resources, delete the child cluster by deleting the ClusterDeployment:

kubectl get clusterdeployments -A

NAMESPACE    NAME            READY   STATUS
bm-example   capm3-example   True    ClusterDeployment is ready

kubectl delete clusterdeployments capm3-example -n <NAMESPACE>

clusterdeployment.k0rdent.mirantis.com "capm3-example" deleted

Cluster deletion may take several minutes. Bare metal machines are deprovisioned at this time.

Watch the BareMetalHost objects as they transition through provisioning states:

kubectl -n <NAMESPACE> get bmh -w

You should see the hosts transition from provisioned to available:

NAME      STATE          CONSUMER                     ONLINE   ERROR   AGE
child-1   deprovisioning capm3-example-md-txr9f-k8z9d true             31m
child-2   provisioned    capm3-example-cp-templ-0     true             31m
child-3   deprovisioning capm3-example-md-txr9f-lkc5c true             31m
child-1   available                                   true             36m
child-3   available                                   true             37m
child-2   deprovisioning capm3-example-cp-templ-0     true             37m
child-2   available                                   true             43m

Then, the available bare metal machines can be used to deploy another cluster, using the same BareMetalHost objects.

Use MetalLB to advertise cluster services#

MetalLB is a popular load-balancer implementation for bare metal Kubernetes clusters. You can install it on a bare metal child cluster using the k0rdent Catalog. Before installation, please ensure the requirements are met, also check the preparation instructions.

To install MetalLB using the k0rdent Catalog, create the following objects in your management cluster:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  labels:
    k0rdent.mirantis.com/managed: "true"
  name: k0rdent-catalog
  namespace: kcm-system  
spec:
  provider: generic
  type: oci
  url: oci://ghcr.io/k0rdent/catalog/charts
---
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ServiceTemplate
metadata:
  name: metallb-0.15.2 # version can be updated. please check catalog.k0rdent.io/latest/apps/metallb for available versions.
  namespace: kcm-system
spec:
  helm:
    chartSpec:
      chart: metallb
      version: 0.15.2  # version can be updated. please check catalog.k0rdent.io/latest/apps/metallb for available versions.
      sourceRef:
        kind: HelmRepository
        name: k0rdent-catalog

and wait until the ServiceTemplate becomes valid:

kubectl -n kcm-system get servicetemplate metallb-0.15.2
NAME             VALID
metallb-0.15.2   true

You can also use other installation methods if you need more customized installation or the desired MetalLB version is not in the k0rdent Catalog.

The abovementioned objects can be added before or after creation of a ClusterDeployment. After the objects were created, the metallb ServiceTemplate can be used to install MetalLB in your child cluster. Add the metallb ServiceTemplate to the spec.config.serviceSpec.services list of your ClusterDeployment object:

spec:
  config:
    serviceSpec:
      services:
        - template: metallb-0.15.2
          name: metallb
          namespace: metallb-system

If your child cluster is already deployed, you can observe the process of MetalLB installation:

KUBECONFIG=<child-cluster-config> kubectl -n metallb-system get all -w

After theMetalLB pods are created, you can create the CustomResource objects that contain the MetalLB configuration on your child cluster. A very simple configuration using ARP advertisements can be defined using the following objects:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: my-address-pool
  namespace: metallb-system
spec:
  addresses:
  - <SERVICE_ADDRESS_RANGE> # e.g., 192.168.1.240-192.168.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertise-all
  namespace: metallb-system

where SERVICE_ADDRESS_RANGE is the range of IP addresses that are controlled by MetalLB and are assigned by MetalLB to load-balanced Kubernetes services as their external IPs.

Note

The above objects can be created on the child cluster only after MetalLB components were installed on that cluster.

Warning

Ensure that the IPs that you define in IPAddressPool.metallb.io objects are not crossed with any host IPs of your child cluster nodes, management cluster k8s API VIP, child cluster k8s API VIP, other IPs that should be accessible from your child cluster.

Please see MetalLB configuration for more information.

The OOT CAPM3 provider upgrade notes#

To upgrade the OOT CAPM3 provider from v0.1.x to v0.2.x, you need to proceed through the same steps as you do in case of installing the provider from scratch. Pay attention to the parameters that are defined in the management object:

the following parameters are new to v0.2.x and "option:router,<ROUTER_IP>" is a required one:

config:
  ironic:
    networking:
      dhcp: # used by DHCP server to assign IPs to hosts during PXE boot
        netmask: <DHCP_SUBNET_MASK>         # e.g., 255.255.255.192 (default is 255.255.255.0)
        options:                            # DHCP options, used during PXE boot and by IPA
          - "option:router,<ROUTER_IP>"     # e.g., 10.0.1.1. It's a mandatory option.
          - "option:dns-server,<DNS_IP[,DNS2_IP...]>" # can be set to KEEPALIVED_VIP (dnsmasq can serve as a DNS server with user-defined DNS records) or to IP of your preferred server. Optional.
          - "option:ntp-server,<NTP_IP>"    # can be set to KEEPALIVED_VIP (internal ntp server) or to IP of your preferred server. That ntp server will be used on PXE boot stage then. Optional.

if you changed the set of OS images used for provisioning of bare metal machines, ensure that the new format is used:

# v0.1.x
config:
  ironic:
    resources:
      static:
        images:
          ubuntu-noble-hwe-2025-05-15-15-22-56.qcow2:
            sha256sum: 581a672e494fcda3297cc8917a91d827157ddcfa3997ad552a914f207b3603c3
            url: https://get.mirantis.com/k0rdent-bm/targetimages/ubuntu-noble-hwe-2025-05-15-15-22-56.qcow2
# v0.2.x
config:
  ironic:
    resources:
      images_target:
        - name: ubuntu-noble-hwe-2025-05-15-15-22-56.qcow2
          url: https://get.mirantis.com/k0rdent-bm/targetimages/ubuntu-noble-hwe-2025-05-15-15-22-56.qcow2
          checksum: 581a672e494fcda3297cc8917a91d827157ddcfa3997ad552a914f207b3603c3
# v0.3.x
config:
  ironic:
    images_target:
      - name: ubuntu-noble-hwe-2025-05-15-15-22-56.qcow2
        url: https://get.mirantis.com/k0rdent-bm/targetimages/ubuntu-noble-hwe-2025-05-15-15-22-56.qcow2
        checksum: 581a672e494fcda3297cc8917a91d827157ddcfa3997ad552a914f207b3603c3

Warning

Starting from v0.3.1 the OOT CAPM3 provider is installed using the CAPI operator. To ensure switching to an operator based installation earlier versions of the OOT CAPM3 providers must be upgraded to v0.3.1 before upgrading to later versions.

Install and use CAPM3 provider in the air-gapped environment#

After Mirantis k0rdent Enterprise is installed in the air-gapped environment, CAPM3 provider can also be installed for Mirantis k0rdent Enterprise and used in this environment.

Install CAPM3 provider in the air-gapped environment#

Download, unpack and verify the CAPM3 airgap bundle.

wget get.mirantis.com/k0rdent-enterprise/bare-metal/0.4.0/airgap-bundle-0.4.0.tar.gz
wget get.mirantis.com/k0rdent-enterprise/bare-metal/0.4.0/airgap-bundle-0.4.0.tar.gz.sig
cosign verify-blob --key https://get.mirantis.com/k0rdent-enterprise/cosign.pub --signature airgap-bundle-0.4.0.tar.gz.sig airgap-bundle-0.4.0.tar.gz

mkdir airgap-bm
tar xf airgap-bundle-0.4.0.tar.gz -C airgap-bm

Upload the CAPM3 artifacts.

It's implied here that the same registry host is used for CAPM3 artifacts as for general Mirantis k0rdent Enterprise artifacts. So, the same registry.local alias will be used as in Install Mirantis k0rdent Enterprise in the airgapped environment.

export REGISTRY="registry.local"

Warning

Replace registry.local with your actual registry hostname.

Upload charts, images and OCI artifacts to the registry.

cd airgap-bm/bundle
for i in $(ls images); do ARTIFACT=$(echo "$i" | tr '@' ':' | tr '&' '/' | sed 's/\.tar//g'); skopeo --insecure-policy copy --dest-cert-dir ~/certs -a oci-archive:images/${i} docker://${REGISTRY}/${ARTIFACT}; done

cd oci-binaries
for i in $(ls); do IMG=$(echo "$i" | tr '@' ':' | tr '&' '/' | sed 's/\/ci//g' | sed 's/\.tar//g'); tar xf "$i"; cd "${i%.tar}" ; oras push ${REGISTRY}/${IMG} *; cd - ; done
cd -

The same HTTP server is used for CAPM3 binaries as for other Mirantis k0rdent Enterprise binaries. So, the same binary.local alias will be used as in Install Mirantis k0rdent Enterprise in the airgapped environment.

export BIN_PATH="binary.local"

Warning

Replace binary.local with the actual path used by your HTTP server.

Move the binary files so that they are accessible via the HTTP server.

for i in $(ls bins); do
  BIN=$(echo "$i" | tr '@' ':' | tr '&' '/' | sed 's/\.tar//g')
  mkdir -p ${BIN_PATH}/${BIN%/*}
  mv bins/${i} ${BIN_PATH}/${BIN}
done

Create the necessary Kubernetes CR objects to install the CAPM3 provider.

kubectl create -f - <<EOF
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: oot-capm3-repo
  namespace: kcm-system
  labels:
    k0rdent.mirantis.com/managed: "true"
spec:
  type: oci
  url: 'oci://registry.local/k0rdent-bm/charts/'
  interval: 10m0s
  certSecretRef:          # This is required only if you didn't add a reference to the registry cert secret
    name: <registry-cert> # in the "management" object: `spec.core.kcm.config.controller.registryCertSecret`
---
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ProviderTemplate
metadata:
  name: cluster-api-provider-metal3-0-4-0
  annotations:
    helm.sh/resource-policy: keep
spec:
  helm:
    chartSpec:
      chart: cluster-api-provider-metal3
      version: 0.4.0
      interval: 10m0s
      sourceRef:
        kind: HelmRepository
        name: oot-capm3-repo
---
apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterTemplate
metadata:
  annotations:
    helm.sh/resource-policy: keep
  labels:
    k0rdent.mirantis.com/component: kcm
  name: capm3-standalone-cp-0-4-0
  namespace: kcm-system
spec:
  helm:
    chartSpec:
      chart: capm3-standalone-cp
      version: 0.4.0
      interval: 10m0s
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: oot-capm3-repo
EOF

Warning

Replace registry.local with your actual registry hostname.

Verify the ProviderTemplate is valid

Check that the ProviderTemplate has been created successfully:

kubectl get providertemplates cluster-api-provider-metal3-0-4-0

NAME                              VALID
cluster-api-provider-metal3-0-4-0 true

Configure the Management object

Edit the Management object to add the CAPM3 provider configuration:

kubectl edit managements.k0rdent.mirantis.com

Add the following configuration to the providers section:

- name: cluster-api-provider-metal3
  template: cluster-api-provider-metal3-0-4-0
  config:
    baremetal-operator:
      global:
        bm_registry: registry.local
    global:
      bm_registry: registry.local
      ironic:
        enabled: true # networking configuration ("ironic.networking" section) should be defined prior to enabling ironic
    ironic:
      dnsmasq:
        ipxe_boot_server: http://binary.local/k0rdent-bm/ipa/pxe/
      global:
        bm_registry: registry.local   
      networking:
        dhcp: # used by DHCP server to assign IPs to hosts during PXE boot
          rangeBegin: <DHCP_RANGE_START>      # e.g., 10.0.1.51
          rangeEnd: <DHCP_RANGE_END>          # e.g., 10.0.1.55
          netmask: <DHCP_SUBNET_MASK>         # e.g., 255.255.255.192 (default is 255.255.255.0)
          options:                            # DHCP options, used during PXE boot and by IPA
            - "option:router,<ROUTER_IP>"     # e.g., 10.0.1.1. It's a mandatory option.
            - "option:dns-server,<DNS_IP[,DNS2_IP...]>" # can be set to KEEPALIVED_VIP (dnsmasq can serve as a DNS server with user-defined DNS records) or to IP of your preferred server. Optional.
            - "option:ntp-server,<NTP_IP>"    # can be set to KEEPALIVED_VIP (internal ntp server) or to IP of your preferred server. That ntp server will be used on PXE boot stage then. Optional.
        interface: <PROVISION_INTERFACE>      # e.g., bond0 - interface of the management cluster node connected to BM hosts provision network
        ipAddress: <KEEPALIVED_VIP>          # e.g., 10.0.1.50 - keepalived VIP for DHCP server and Ironic services. This VIP will be configured on the <PROVISION_INTERFACE>, it must be in the same L3 network as DHCP range if no dhcp-relay used between management cluster and child cluster hosts.
      images_ipa:
      - checksum: ff8c3caad212bd1f9ac46616cb8d7a2646ed0da85da32d89e1f5fea5813265f8
        name: ironic-python-agent_x86_64.initramfs
        url: http://binary.local/k0rdent-bm/ipa/ipa-centos9-stream-2025-12-01-11-53-35-amd64.initramfs
      - name: ironic-python-agent_x86_64.kernel
        checksum: e28f0a2185b618efb44e8f24ea806ec775cfc2f2446816da215ffa97a811e9af
        url: http://binary.local/k0rdent-bm/ipa/ipa-centos9-stream-2025-12-01-11-53-35-amd64.kernel
      images_target:
      - checksum: f0a5da9499adaaca6249792df25032430f33f0130eddf39433782b5362057b99
        name: ubuntu-24.04-server-cloudimg-20251118-amd64.img
        url: http://binary.local/k0rdent-bm/target/ubuntu-24.04-server-cloudimg-20251118-amd64.img

Warning

Replace registry.local with your actual registry hostname. Replace binary.local with the actual path used by your HTTP server.

Continue with steps 4,5,6 of the "Prepare Mirantis k0rdent Enterprise for Bare Metal clusters" section

Deploy a bare metal cluster in the air-gapped environment#

After the CAPM3 provider is deployed in the air-gapped environment, you can create a bare metal child cluster. General procedure is the same as described in "Create a bare metal cluster" section. Though, there are certain differences in parameters of ClusterDeployment object (created in step 3):

spec.config.k0s parameters section must include the following:

k0s:
  ...
  version: v1.32.6+k0s.0 # by default must match K0rdent Enterprise configuration
  arch: amd64 # mandatory
  images:
    metricsserver:
      image: "registry.local/k0rdent-enterprise/metrics-server/metrics-server"
    kubeproxy:
      image: "registry.local/k0rdent-enterprise/k0sproject/kube-proxy"
    coredns:
      image: "registry.local/k0rdent-enterprise/k0sproject/coredns"
    pause:
      image: "registry.local/k0rdent-enterprise/pause"
    calico:
      cni:
        image: "registry.local/k0rdent-enterprise/k0sproject/calico-cni"
      node:
        image: "registry.local/k0rdent-enterprise/k0sproject/calico-node"
      kubecontrollers:
        image: "registry.local/k0rdent-enterprise/k0sproject/calico-kube-controllers"

Warning

Replace registry.local with your actual registry hostname.

Warning

Ensure that the artifacts for your target k0s version are included in the Mirantis k0rdent Enterprise airgap bindle.

spec.config.controlPlane.preStartCommands and spec.config.worker.preStartCommands must not include commands that need access to external networks.

Warning

jq package is still required for the deployment of child cluster. Thus, ensure that it's included in your target OS image, or it's uploaded from your HTTP server and installed during cloud-init - for both control plane and worker nodes.

Provisioning of servers with different CPU architectures#

The bare metal infrastructure provider, starting v0.4.0, is capable of provisioning the servers with amd64 and arm64 CPU architectures with certain limitations.

Limitations#

BM provider v0.4.0 does not include the ClusterTemplate suitable for provisioning of both amd64 and arm64 workers in a single cluster, but this can be done using the procedure described below in this chapter.
BM provider v0.4.0 does not support provisioning of both amd64 and arm64 control plane nodes in a single cluster.

Configuration of BM provider for provisioning of `arm64` servers#

BM provider v0.4.0 includes IPA images and target OS images configuration for both amd64 and arm64 servers. So, it's not necessary to change IPA images configuration. Though, user may want to add specific target OS images for their servers if default ones are not suitable. This is done by editing the Management object during the BM provider installation:

- name: cluster-api-provider-metal3
  template: cluster-api-provider-metal3-0-4-0
  config:
    ...
    ironic:
      ...
      images_target: # a list of OS images for both amd64 & arm64 servers
        - name: <image-local-filename> # e.g., ubuntu-noble-arm64.qcow2
          url: <image-download-url> # e.g., https://get.somewhere.com/targetimages/ubuntu-noble-arm64.qcow2
          checksum: <image-sha256-sum> # e.g., 581a672e494fcda3297cc8917a91d827157ddcfa3997ad552a914f207b3603c3
        ...

Note

Default target OS images configuration will be substituted by (not merged with) one provided in the Management object.

Provisioning of a cluster with `arm64` servers#

BM provider v0.4.0 supports provisioning of the following combinations of amd64 and arm64 servers in a child cluster:

amd64 servers for control plane and workers;
arm64 servers for control plane and workers;
amd64 servers for control plane and arm64 servers for workers;
arm64 servers for control plane and amd64 servers for workers;
amd64 servers for control plane and mixed CPU arch. servers for workers - requires a customized ClusterTemplate.

In case of using servers with mixed CPU architecture in a single cluster, you always need to use specific labels in all related BareMetalHost objects so that bare metal servers can be correctly selected for Machine objects that await particular CPU architecture. For example, amd64 servers are targeted as control plane nodes, arm64 servers are targeted as workers.

Configuring the `BareMetalHost` and `ClusterDeployment` objects#

During the enrollment of your bare metal servers, define the label in the BareMetalHost objects to distinguish servers with different CPU architectures:

  apiVersion: metal3.io/v1alpha1
  kind: BareMetalHost
  metadata:
    labels:
      # userlabel/cpuarch is used as an example, define the labels according to your needs
      userlabel/cpuarch: <CPU_ARCH> # e.g., "amd64", "arm64"

All the workers have the same CPU architecture

In this case, the default ClusterTemplate can be sufficient. The following example demonstrates parameters change for the case where amd64 servers are used for a control plane and all cluster workers are arm64 servers.

During the creation of your cluster, set the image-related parameters and host selectors in the ClusterDeployment, also any other parameters you need:

apiVersion: k0rdent.mirantis.com/v1beta1
kind: ClusterDeployment
...
spec:
  config:
    controlPlane:
      # the amd64 compatible OS image
      checksum: <amd64-image-sha256-sum>
      image: http://<IRONIC_HTTP_SERVER_IP>:6180/images/<amd64-image-local-filename>
      hostSelector:
        matchLabels:
          userlabel/cpuarch: amd64
      ...
    worker:
      # the arm64 compatible OS image
      checksum: <arm64-image-sha256-sum>
      image: http://<IRONIC_HTTP_SERVER_IP>:6180/images/<arm64-image-local-filename>
      hostSelector:
        matchLabels:
          userlabel/cpuarch: arm64
      ...

Workers have different CPU architectures

In this case, you need to: * create the customized ClusterTemplate or * use the default ClusterTemplate and create additional objects after the cluster was created.

It's worth creating a customized template if you plan to provision such clusters regularly.

For testing purposes, you can the default ClusterTemplate with additional objects as described below.

First, create a ClusterDeployment the same way as it's shown in the above example. You can configure it to provision either amd64 or arm64 workers. To provision the remaining group of workers, you'll need to create additional objects. For example, you configure the ClusterDeployment to provision arm64 workers, and amd64 workers will be provisioned using the additional objects.

After the ClusterDeployment object was created, check the MachineDeployment object for your cluster:

kubectl get MachineDeployment -n <NAMESPACE> # same namespace as in the ClusterDeployment

NAME               CLUSTER         AVAILABLE   DESIRED   CURRENT   READY   AVAILABLE   UP-TO-DATE   PHASE     AGE     VERSION
capm3-example-md   capm3-example   True        2         2         2       2           2            Running   3h53m   v1.32.3

and extract spec.template.spec.bootstrap.configRef.name from it:

kubectl get MachineDeployment -n <NAMESPACE> <MACHINE_DEPLOYMENT_NAME> -o jsonpath='{.spec.template.spec.bootstrap.configRef.name}'

capm3-example-machine-config

Check the Metal3DataTemplate object for your cluster

kubectl get Metal3DataTemplate -n <NAMESPACE> # same namespace as in the ClusterDeployment

NAME               CLUSTER   AGE
capm3-example-dt             5h28m

Create the following objects:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
  name: <MACHINE_TEMPLATE_NAME> # e.g., capm3-example-worker-mt-amd. it must not overlap with existing Metal3MachineTemplate objects
  namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
spec:
  template:
    spec:
      automatedCleaningMode: metadata
      dataTemplate:
        name: <METAL3_DATA_TEMPLATE_NAME> # e.g., capm3-example-dt - must match your existing Metal3DataTemplate (obtained in the previous step)
      hostSelector:
        matchLabels: # can be more specific, e.g. to distinguish control-plane amd64 servers and worker amd64 servers
          userlabel/cpuarch: amd64
      image:
        checksum: <amd64-image-sha256-sum>
        checksumType: sha256
        format: qcow2
        url: http://<IRONIC_HTTP_SERVER_IP>:6180/images/<amd64-image-local-filename>
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
  name: <MACHINE_DEPLOYMENT_NAME> # e.g., capm3-example-md-amd. it must not overlap with existing MachineDeployment objects
  namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
spec:
  clusterName: <CLUSTER_NAME> # e.g., capm3-example
  replicas: <NUMBER_OF_MACHINES> # e.g., 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: <CLUSTER_NAME> # e.g., capm3-example
  strategy: # example. optional
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <CLUSTER_NAME> # e.g., capm3-example
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: K0sWorkerConfigTemplate
          name: <K0S_WORKER_CONFIG_TEMPLATE_NAME> # e.g., capm3-example-machine-config - must match your existing K0sWorkerConfigTemplate (obtained in the previous step)
          namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
      clusterName: <CLUSTER_NAME> # e.g., capm3-example
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: Metal3MachineTemplate
        name: <MACHINE_TEMPLATE_NAME> # e.g., capm3-example-worker-mt-amd - must match the name of the Metal3MachineTemplate object above
        namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
      version: <K0S_VERSION> # e.g., v1.32.3 - must match your cluster version

Optional. Create more pairs of Metal3MachineTemplate and MachineDeployment objects if more distinct groups of workers are needed (for example, to use specific servers with kubevirt).

Optional. Create your own K0sWorkerConfigTemplate and Metal3DataTemplate objects for every pair of Metal3MachineTemplate and MachineDeployment objects if you need to set some parameters separately for different groups of workers. For example: pass additional Kubelet flags (check Metal3 requirements) and specify custom network settings (the static IP address) for the particular node

apiVersion: ipam.metal3.io/v1alpha1
kind: IPPool
metadata:
  name: pool-ext-192.168.50.20 # must not overlap with existing IPPool objects
  namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
spec:
  clusterName: <CLUSTER_NAME> # e.g., capm3-example
  namePrefix: pool-ext
  pools:
  - end: 192.168.50.20
    gateway: 192.168.50.1
    prefix: 24
    start: 192.168.50.20
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: K0sWorkerConfigTemplate
metadata:
  name: capm3-example-machine-config-custom # must not overlap with existing K0sWorkerConfigTemplate objects
  namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
spec:
  template:
    spec:
      args: # apply custom Kubelet flags
      - --kubelet-extra-args="--node-ip={{ ds.meta_data.provisioningIP }}" # default behavior when using bundled ClusterTemplate
      - --labels="metal3.io/uuid={{ ds.meta_data.uuid }}" # mandatory when using capm3 provider
      - --hostname-override="{{ ds.meta_data.local_hostname }}" # use predictable hostname
      files:
      - content: ssh-rsa <key>
        path: /home/user1/.ssh/authorized_keys
        permissions: "0644"
      k0sInstallDir: /usr/local/bin
      preStartCommands:
      - sudo useradd -G sudo -s /bin/bash -d /home/user1 -p $(openssl passwd -1 myuserpass)
        user1
      useSystemHostname: false
      version: <K0S_VERSION> # e.g., v1.32.3 - must match your cluster version
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3DataTemplate
metadata:
  name: capm3-example-dt-node1 # must not overlap with existing Metal3DataTemplate objects
  namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
spec:
  clusterName: <CLUSTER_NAME> # e.g., capm3-example
  metaData:
    ipAddressesFromIPPool:
    - key: provisioningIP
      name: pool-pxe
    # Use the static IP address from the custom single-address pool
    - key: extIP
      name: pool-ext-192.168.50.20
    objectNames:
    - key: name
      object: machine
    - key: local-hostname
      object: machine
    - key: local_hostname
      object: machine
    prefixesFromIPPool:
    - key: provisioningCIDR
      name: pool-pxe
  networkData:
    links:
      ethernets:
      - id: enp1s0
        macAddress:
          fromHostInterface: enp1s0
        mtu: 1500
        type: phy
      - id: enp8s0
        macAddress:
          fromHostInterface: enp8s0
        mtu: 1500
        type: phy
    networks:
      ipv4:
      - id: pxe
        ipAddressFromIPPool: pool-pxe
        link: enp1s0
        routes:
        - gateway:
            fromIPPool: pool-pxe
          network: 0.0.0.0
          services: {}
      - id: ext
        ipAddressFromIPPool: pool-ext-192.168.50.20
        link: enp8s0
    services:
      dns:
      - 192.168.100.10
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3MachineTemplate
  name: capm3-example-node1-mt # must not overlap with existing Metal3MachineTemplate objects
  namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
spec:
  template:
    spec:
      automatedCleaningMode: metadata
      dataTemplate:
        name: capm3-example-dt-node1 # Metal3DataTemplate name (defined above)
      hostSelector:
        matchLabels: # target the particular node by a custom label
          custom/node-name: node-1
      image:
        checksum: <amd64-image-sha256-sum>
        checksumType: sha256
        format: qcow2
        url: http://<IRONIC_HTTP_SERVER_IP>:6180/images/<amd64-image-local-filename>
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
  name: capm3-example-md-node1 # must not overlap with existing MachineDeployment objects
  namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
spec:
  clusterName: <CLUSTER_NAME> # e.g., capm3-example
  replicas: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: <CLUSTER_NAME> # e.g., capm3-example
  strategy: # example. optional
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: <CLUSTER_NAME> # e.g., capm3-example
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: K0sWorkerConfigTemplate
          name: capm3-example-machine-config-custom # K0sWorkerConfigTemplate name (defined above)
          namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
      clusterName: <CLUSTER_NAME> # e.g., capm3-example
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: Metal3MachineTemplate
        name: capm3-example-node1-mt # Metal3MachineTemplate name (defined above)
        namespace: <NAMESPACE> # same namespace as in the ClusterDeployment
      version: <K0S_VERSION> # e.g., v1.32.3 - must match your cluster version

Troubleshooting#

If you run into difficulties, you might find the solution here.

`ProviderTemplate` is not valid#

If the ProviderTemplate shows as invalid, inspect the object for error messages:

kubectl get providertemplates cluster-api-provider-metal3-0-4-0 -oyaml

Common issues include incorrect credentials for accessing artifacts or connection problems.

`Management` object does not become ready#

If the Management object remains in a non-ready state, inspect it for error messages:

kubectl get managements.k0rdent.mirantis.com -oyaml

If you see Ironic-related errors, check the Ironic deployment:

kubectl -n kcm-system get deployment/cluster-api-provider-metal3-ironic

If Ironic is not ready, verify its configuration:

kubectl -n kcm-system get cm ironic-bmo-config -oyaml

Ensure the configuration matches your network environment, particularly the PROVISIONING_INTERFACE, PROVISIONING_IP, and DHCP_RANGE settings.

The `HelmRelease` does not become ready#

Check to see if the HelmRelease object is Ready, and if not, why:

kubectl -n kcm-system get helmrelease cluster-api-provider-metal3

NAME                          AGE    READY   STATUS
cluster-api-provider-metal3   164m   False   Helm install failed for release kcm-system/cluster-api-provider-metal3 with chart cluster-api-provider-metal3@0.2.1: context deadline exceeded

If you see this error, delete the HelmRelease:

kubectl -n kcm-system delete helmrelease cluster-api-provider-metal3

Flux will automatically reinstall the HelmRelease. If you see the same error again, set the defaultHelmTimeout value in the management object (default value is 5 minutes), wait for the management object to become Ready and delete the HelmRelease again so Flux will reinstall it.

kubectl edit managements.k0rdent.mirantis.com

spec:
  core:
    kcm:
      config:
        controller:
          defaultHelmTimeout: 30m

For more context, see the related issue.

`BareMetalHost` registration error#

Ironic might fail to register a BareMetalHost with the following error:

  MAC address 00:01:02:03:04:05 conflicts with existing node default~bmh1

Ironic fails to register a BareMetalHost when the MAC address is already associated with another node. This conflict might occur if a new BareMetalHost reuses the MAC address used by a previous BareMetalHost.

Warning

Before proceeding with any further command, ensure that there is indeed no BareMetalHost with a conflicting MAC address in the environment.

To resolve the issue, you can restart the Ironic pod:

  kubectl -n kcm-system rollout restart deploy cluster-api-provider-metal3-ironic

Without the Ironic restart, you can access the Ironic database using baremetal or openstack CLI as follows:

  kubectl port-forward -n kcm-system svc/cluster-api-provider-metal3-ironic <HOST_PORT>:<IRONIC_API_PORT>
  export OS_ENDPOINT=https://localhost:<HOST_PORT>
  export OS_AUTH_TYPE=none

  # get a list of nodes:
  baremetal node list
  # unregister baremetal node:
  baremetal node delete <node>

Alternatively, you can delete the information about the old BareMetalHost from the Ironic database with the following commands:

  kubectl -n kcm-system exec -it <IRONIC_POD_NAME> -c mariadb -- \
    mysql -uironic -p<PASSWORD> ironic -e "DELETE p, ni FROM nodes n LEFT JOIN ports p ON p.node_id=n.id \
    LEFT JOIN node_inventory ni ON ni.node_id=n.id WHERE n.name='<OLD_BMH_NAMESPACE>~<BMH_NAME>';”
  kubectl -n kcm-system exec -it <IRONIC_POD_NAME> -c mariadb -- \
    mysql -uironic -p<PASSWORD> ironic -e "DELETE FROM nodes WHERE name='<OLD_BMH_NAMESPACE>~<BMH_NAME>';”

Note

You can obtain the Ironic database password as follows: kubectl -n kcm-system get secret ironic-auth-config -o jsonpath={.data.password} | base64 -d

Useful resources#

For additional troubleshooting guidance, refer to the Metal3 troubleshooting documentation.

For more information about bare metal cluster configuration options, see:

Bare Metal#

Structure#

Prerequisites#

Prepare Mirantis k0rdent Enterprise for Bare Metal clusters#

1. Create the required objects for the OOT CAPM3 provider#

2. Verify the ProviderTemplate is valid#

3. Configure the Management object#

4. Wait for the Management object to be ready#

5. Verify the ClusterTemplate is valid#

6. Optional. Tune the DHCP server.#

7. Optional. Use MetalLB to advertise services of a bare metal management cluster#

Enroll bare metal machines#

Baremetal host OS update#

Manage block device partitions and filesystems#

Support for RAID disk arrays on baremetal hosts#

Create a bare metal cluster#

Use MetalLB to advertise cluster services#

The OOT CAPM3 provider upgrade notes#

Install and use CAPM3 provider in the air-gapped environment#

Install CAPM3 provider in the air-gapped environment#

Deploy a bare metal cluster in the air-gapped environment#

Provisioning of servers with different CPU architectures#

Limitations#

Configuration of BM provider for provisioning of arm64 servers#

Provisioning of a cluster with arm64 servers#

Configuring the BareMetalHost and ClusterDeployment objects#

Troubleshooting#

ProviderTemplate is not valid#

Management object does not become ready#

The HelmRelease does not become ready#

BareMetalHost registration error#

Useful resources#

2. Verify the `ProviderTemplate` is valid#

3. Configure the `Management` object#

4. Wait for the `Management` object to be ready#

5. Verify the `ClusterTemplate` is valid#

7. Optional. Use `MetalLB` to advertise services of a bare metal management cluster#

Configuration of BM provider for provisioning of `arm64` servers#

Provisioning of a cluster with `arm64` servers#

Configuring the `BareMetalHost` and `ClusterDeployment` objects#

`ProviderTemplate` is not valid#

`Management` object does not become ready#

The `HelmRelease` does not become ready#

`BareMetalHost` registration error#