After you complete the prerequisite steps described in Prerequisites,
proceed with bootstrapping your VMware vSphere-based Mirantis Container Cloud
management cluster.
To bootstrap a vSphere-based management cluster:
Log in to the bootstrap node running Ubuntu 20.04 that is configured
as described in Prerequisites.
Prepare the bootstrap script:
Download and run the Container Cloud bootstrap script:
IP address from the provided vSphere network for Kubernetes API load
balancer (Keepalived VIP).
SET_VSPHERE_DATASTORE
Name of the vSphere datastore. You can use different datastores
for vSphere Cluster API and vSphere Cloud Provider.
SET_VSPHERE_MACHINES_FOLDER
Path to a folder where the cluster machines metadata will be stored.
SET_VSPHERE_NETWORK_PATH
Path to a network for cluster machines.
SET_VSPHERE_RESOURCE_POOL_PATH
Path to a resource pool in which VMs will be created.
Note
To obtain the LB_HOST parameter for the selected vSphere
network, contact your vSphere administrator who provides you with the
IP ranges dedicated to your environment.
Modify other parameters if required. For example, add the corresponding
values for cidrBlocks in the spec::clusterNetwork::services
section.
Provide the following additional parameters for a proper network
setup on machines using embedded IP address management (IPAM)
in templates/vsphere/cluster.yaml.template:
Note
To obtain IPAM parameters for the selected vSphere network, contact
your vSphere administrator who provides you with IP ranges dedicated to your
environment only.
Enables IPAM. Recommended value is true for either DHCP or
non-DHCP networks.
SET_VSPHERE_NETWORK_CIDR
CIDR of the provided vSphere network. For example, 10.20.0.0/16.
SET_VSPHERE_NETWORK_GATEWAY
Gateway of the provided vSphere network.
SET_VSPHERE_CIDR_INCLUDE_RANGES
IP range for the cluster machines. Specify the range of the
provided CIDR. For example, 10.20.0.100-10.20.0.200.
If the DHCP network is used, this range must not intersect with
the DHCP range of the network.
SET_VSPHERE_CIDR_EXCLUDE_RANGES
Optional. IP ranges to be excluded from being assigned to
the cluster machines. The MetalLB range and SET_LB_HOST
should not intersect with the addresses for IPAM. For example,
10.20.0.150-10.20.0.170.
SET_VSPHERE_NETWORK_NAMESERVERS
List of nameservers for the provided vSphere network.
Add SET_VSPHERE_METALLB_RANGE that is the MetalLB range of IP
addresses to assign to load balancers for Kubernetes Services.
Note
To obtain the VSPHERE_METALLB_RANGE parameter for the
selected vSphere network, contact your vSphere administrator who
provides you with the IP ranges dedicated to your environment.
For RHEL deployments, fill out
templates/vsphere/rhellicenses.yaml.template.
RHEL license configuration
Use one of the following set of parameters for RHEL machines subscription:
The user name and password of your RedHat Customer Portal account
associated with your RHEL license for Virtual Datacenters.
Optionally, provide the subscription allocation pools to use for the RHEL
subscription activation. If not needed, remove the poolIDs field
for subscription-manager to automatically select the licenses for
machines.
The activation key and organization ID associated with your RedHat
account with RHEL license for Virtual Datacenters. The activation key
can be created by the organization administrator on the RedHat Customer
Portal.
If you use the RedHat Satellite server for management of your
RHEL infrastructure, you can provide a pre-generated activation key from
that server. In this case:
Provide the URL to the RedHat Satellite RPM for installation
of the CA certificate that belongs to that server.
Configure squid-proxy on the management or regional cluster to
allow access to your Satellite server. For details, see
Configure squid-proxy.
For RHEL 8.7, verify mirrors configuration for your
activation key. For more details, see RHEL 8 mirrors configuration.
Warning
Provide only one set of parameters. Mixing the parameters
from different activation methods will cause deployment failure.
For CentOS deployments, in templates/vsphere/rhellicenses.yaml.template,
remove all lines under items:.
Available since Container Cloud 2.24.0. Optional.
Technology Preview. Enable custom host names for cluster machines.
When enabled, any machine host name in a particular region matches the related
Machine object name. For example, instead of the default
kaas-node-<UID>, a machine host name will be master-0. The custom
naming format is more convenient and easier to operate with.
To enable the feature on the management and its future managed clusters:
Since 2.25.0
In templates/vsphere/cluster.yaml.template, find the
spec.providerSpec.value.kaas.regional section of the required
region.
In this section, find the required provider name under
helmReleases.
Under values.config, add customHostnamesEnabled:true.
For example, for the bare metal provider in region-one:
In bootstrap.env, add the KAAS_VSPHERE_ENABLED=true environment
variable that enables the vSphere provider deployment in Container Cloud.
Configure NTP server.
Before Container Cloud 2.23.0, optional if servers from the Ubuntu NTP pool
(*.ubuntu.pool.ntp.org) are accessible from the node where
the management cluster is being provisioned. Otherwise, configure the
regional NTP server parameters as described below.
Since Container Cloud 2.23.0, optionally disable NTP that is enabled by default. This option disables the
management of chrony configuration by Container Cloud to use your own
system for chrony management. Otherwise, configure the regional NTP server
parameters as described below.
NTP configuration
Configure the regional NTP server parameters to be applied to all machines
of regional and managed clusters in the specified region.
In templates/vsphere/cluster.yaml.template, add the ntp:servers section
with the list of required server names:
In templates/vsphere/machines.yaml.template, define the following
parameters:
rhelLicense
RHEL license name defined in rhellicenses.yaml.template, defaults to
kaas-mgmt-rhel-license. Remove or comment out this parameter for
CentOS and Ubuntu deployments.
diskGiB
Disk size in GiB for machines that must match the disk size of the VM
template. You can leave this parameter commented to use the disk size of
the VM template. The minimum requirement is 120 GiB.
template
Path to the VM template prepared in the previous step.
If you require all Internet access to go through a proxy server,
in bootstrap.env, add the following environment variables
to bootstrap the management and regional cluster using proxy:
In MITM proxy deployments, use the internal Red Hat Satellite
server to register RHEL machines so that a VM can access this server directly
without a MITM proxy.
Available since Container Cloud 2.24.0 and 2.24.2 for MOSK 23.2. Optional.
Technology Preview. Enable the Linux Audit daemon auditd
to monitor activity of cluster processes and prevent potential malicious
activity.
Configuration for auditd
In templates/vsphere/cluster.yaml.template, add the auditd parameters:
Boolean, default - false. Enables the auditd role to install the
auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.
enabledAtBoot
Boolean, default - false. Configures grub to audit processes that can
be audited even if they start up prior to auditd startup. CIS rule:
4.1.1.3.
backlogLimit
Integer, default - none. Configures the backlog to hold records. If during
boot audit=1 is configured, the backlog holds 64 records. If more than
64 records are created during boot, auditd records will be lost with a
potential malicious activity being undetected. CIS rule: 4.1.1.4.
maxLogFile
Integer, default - none. Configures the maximum size of the audit log file.
Once the log reaches the maximum size, it is rotated and a new log file is
created. CIS rule: 4.1.2.1.
maxLogFileAction
String, default - none. Defines handling of the audit log file reaching the
maximum file size. Allowed values:
keep_logs - rotate logs but never delete them
rotate - add a cron job to compress rotated log files and keep
maximum 5 compressed files.
compress - compress log files and keep them under the
/var/log/auditd/ directory. Requires
auditd_max_log_file_keep to be enabled.
CIS rule: 4.1.2.2.
maxLogFileKeep
Integer, default - 5. Defines the number of compressed log files to keep
under the /var/log/auditd/ directory. Requires
auditd_max_log_file_action=compress. CIS rules - none.
mayHaltSystem
Boolean, default - false. Halts the system when the audit logs are
full. Applies the following configuration:
space_left_action=email
action_mail_acct=root
admin_space_left_action=halt
CIS rule: 4.1.2.3.
customRules
String, default - none. Base64-encoded content of the 60-custom.rules
file for any architecture. CIS rules - none.
customRulesX32
String, default - none. Base64-encoded content of the 60-custom.rules
file for the i386 architecture. CIS rules - none.
customRulesX64
String, default - none. Base64-encoded content of the 60-custom.rules
file for the x86_64 architecture. CIS rules - none.
presetRules
String, default - none. Comma-separated list of the following built-in
preset rules:
access
actions
delete
docker
identity
immutable
logins
mac-policy
modules
mounts
perm-mod
privileged
scope
session
system-locale
time-change
You can use two keywords for these rules:
none - disables all built-in rules.
all - enables all built-in rules. With this key, you can add the
! prefix to a rule name to exclude some rules. You can use the
! prefix for rules only if you add the all keyword as the
first rule. Place a rule with the ! prefix only after
the all keyword.
Example configurations:
presetRules:none - disable all preset rules
presetRules:docker - enable only the docker rules
presetRules:access,actions,logins - enable only the
access, actions, and logins rules
presetRules:all - enable all preset rules
presetRules:all,!immutable,!sessions - enable all preset
rules except immutable and sessions
Optional. Enable infinite timeout for all bootstrap stages by exporting the
following environment variable or adding it to bootstrap.env:
exportKAAS_BOOTSTRAP_INFINITE_TIMEOUT=true
Infinite timeout prevents the bootstrap failure due to timeout. This option
is useful in the following cases:
The network speed is slow for artifacts downloading
An infrastructure configuration does not allow booting fast
A bare-metal node inspecting presupposes more than two HDDSATA disks
to attach to a machine
Optional. Available since Container Cloud 2.23.0. Customize the cluster and
region name by exporting the following environment variables or adding them
to bootstrap.env:
The Keycloak URL that the system outputs when the bootstrap completes.
The admin password for Keycloak is located in
kaas-bootstrap/passwords.yml along with other IAM passwords.
Note
The Container Cloud web UI and StackLight endpoints are available
through Transport Layer Security (TLS) and communicate with Keycloak
to authenticate users. Keycloak is exposed using HTTPS and
self-signed TLS certificates that are not trusted by web browsers.
Not all of Swarm and MCR addresses are usually in use. One Swarm Ingress
network is created by default and occupies the 10.0.0.0/24 address
block. Also, three MCR networks are created by default and occupy
three address blocks: 10.99.0.0/20, 10.99.16.0/20,
10.99.32.0/20.
To verify the actual networks state and addresses in use, run: