Introduction¶
This documentation provides information on how to deploy and operate a Mirantis Kubernetes Engine (MKE). The documentation is intended to help operators to understand the core concepts of the product. The documentation provides sufficient information to deploy and operate the solution.
The information provided in this documentation set is being constantly improved and amended based on the feedback and kind requests from the consumers of MKE.
Product Overview¶
Mirantis Kubernetes Engine (MKE, formerly Universal Control Plane or UCP) is the industry-leading container orchestration platform for developing and running modern applications at scale, on private clouds, public clouds, and on bare metal.
MKE delivers immediate value to your business by allowing you to adopt modern application development and delivery models that are cloud-first and cloud-ready. With MKE you get a centralized place with a graphical UI to manage and monitor your Kubernetes and/or Swarm cluster instance.
Your business benefits from using MKE as a container orchestration platform, especially in the following use cases:
- More than one container orchestrator
Whether your application requirements are complex and require medium to large clusters or simple ones that can be deployed quickly on development environments, MKE gives you a container orchestration choice. Deploy Kubernetes, Swarm, or both types of clusters and manage them on a single MKE instance or centrally manage your instance using Mirantis Container Cloud.
- Robust and scalable applications deployment
Monolithic applications are old school, microservices are the modern way to deploy an application at scale. Delivering applications through an automated CI/CD pipeline can dramatically improve time-to-market and service agility. Adopting microservices becomes a lot easier when using Kubernetes and/or Swarm clusters to deploy and test microservice-based applications.
- Multi-tenant software offerings
Containerizing existing monolithic SaaS applications enables quicker development cycles, automated continuous integration and deployment. But these applications need to allow multiple users to share a single instance of a software application. MKE can operate multi-tenant environments, isolate teams and organizations, separate cluster resources, and so on.
See also
Reference Architecture¶
The MKE Reference Architecture provides a technical overview of Mirantis Kubernetes Engine (MKE). It is your source for the product hardware and software specifications, standards, component information, and configuration detail.
Introduction to MKE¶
Mirantis Kubernetes Engine (MKE) allows you to adopt modern application development and delivery models that are cloud-first and cloud-ready. With MKE you get a centralized place with a graphical UI to manage and monitor your Kubernetes and/or Swarm cluster instance.
The core MKE components are:
ucp-cluster-agentReconciles the cluster-wide state, including Kubernetes addons such as Kubecompose and KubeDNS, managing replication configurations for the
etcdand RethinkDB clusters, and syncing the node inventories of SwarmKit and Swarm Classic. This component is a single-replica service that runs on any manager node in the cluster.
ucp-manager-agentReconciles the node-local state on manager nodes, including the configuration of the local Docker daemon, local date volumes, certificates, and local container components. Each manager node in the cluster runs a task from this service.
ucp-worker-agentPerforms the same reconciliation operations as
ucp-manager-agentbut on worker nodes. This component runs a task on each worker node.
The following MKE component names differ based on the node’s operating system:
Component name on Linux |
Component name on Windows |
|---|---|
|
|
|
|
|
|
No equivalent |
|
|
|
MKE hardware requirements¶
Take careful note of the minimum and recommended hardware requirements for MKE manager and worker nodes prior to deployment.
Note
High availability (HA) installations require transferring files between hosts.
On manager nodes, MKE only supports the workloads it requires to run.
Windows container images are typically larger than Linux container images. As such, provision more local storage for Windows nodes and for any MSR repositories that store Windows container images.
Manager nodes |
Worker nodes |
|
|---|---|---|
Minimum hardware requirements |
|
|
Recommended hardware requirements |
|
Recommendations vary depending on the workloads. |
MKE software requirements¶
Prior to MKE deployment, consider the following software requirements:
Run the same MCR version (20.10.0 or later) on all nodes.
Run Linux kernel 3.10 or higher on all nodes.
For debugging purposes, the host OS kernel versions should match as closely as possible.
Use a static IP address for each node in the cluster.
Manager nodes¶
Manager nodes manage a swarm and persist the swarm state. Using several
containers per node, the ucp-manager-agent automatically deploys all
MKE components on manager nodes, including the MKE web UI and the data stores
that MKE uses.
Note
Some Kubernetes components are run as Swarm services because the MKE control plane is itself a Docker Swarm cluster.
The following tables detail the MKE services that run on manager nodes:
MKE component |
Description |
|---|---|
|
The centralized service for identity and authentication used by MKE and MSR. |
|
A container that stores authentication configurations and data for users, organizations, and teams. |
|
A container that performs scheduled LDAP synchronizations and cleans authentication and authorization data. |
|
A certificate authority to sign client bundles. |
|
The agent that monitors the cluster-wide MKE components. Runs on only one manager node. |
|
A certificate authority used for TLS communication between MKE components. |
|
The MKE web server. |
|
A container for collecting disk/hardware information about the host. |
|
A container that monitors Swarm workloads configured to use layer 7 routing. Only runs when you enable layer 7 routing. |
|
A service that manages Interlock configuration. |
|
A service that verifies the run status of the Interlock extension. |
|
A service that provides load balancing and proxying for Swarm workloads. Runs only when layer 7 routing is enabled. |
|
A master component that serves the Kubernetes API. It persists its state
in |
|
A master component that manages the desired state of controllers and other Kubernetes objects. It monitors the API server and performs background tasks when needed. |
|
The Kubernetes node agent running on every node, which is responsible for running Kubernetes pods, reporting the health of the node, and monitoring resource usage. |
|
The networking proxy running on every node, which enables pods to contact Kubernetes services and other pods by way of cluster IP addresses. |
|
A master component that manages Pod scheduling, which communicates with the API server only to obtain workloads that need to be scheduled. |
|
A container used to store the MKE configurations. Do not use it in your applications, as it is for internal use only. Also used by Kubernetes components. |
|
The agent that monitors the manager node and ensures that the right MKE services are running. |
|
A TLS proxy that allows secure access from the local Mirantis Container Runtime to MKE components. |
|
A Swarm service that sends notifications to Salesforce when alerts are configured by OpsCare, and later when they are triggered. |
|
A container used to provide backward compatibility with Docker Swarm. |
MKE component |
Description |
|---|---|
|
An MKE service that accounts for the removal of dockershim from Kubernetes as of version 1.24, thus enabling MKE to continue using Docker as the container runtime. |
|
A cluster-scoped Kubernetes controller used to coordinate Calico networking. Runs on one manager node only. |
|
The Calico node agent, which coordinates networking fabric according
to the cluster-wide Calico configuration. Part of the |
|
An init container for Calico controller that sets the StrictAffinity in Calico networking according to the configured boolean value. |
|
An init container for |
|
A container in which the Calico CNI plugin binaries are installed and configured on each host. Part of the calico-node DaemonSet. Runs on all nodes. |
|
The CoreDNS plugin, which provides service discovery for Kubernetes services and Pods. |
|
The Gatekeeper manager controller for Kubernetes that provides policy enforcement. Only runs when OPA Gatekeeper is enabled in MKE. |
|
The audit controller for Kubernetes that provides audit functionality of OPA Gatekeeper. Only runs when OPA Gatekeeper is enabled in MKE. |
|
A custom Kubernetes resource component that translates Compose files into Kubernetes constructs. Part of the Compose deployment. Runs on one manager node only. |
|
The API server for Kube Compose, which is part of the compose deployment. Runs on one manager node only. |
|
The Ingress controller for Kubernetes, which provides layer 7 routing for Kubernertes services. Only runs with Ingress for Kubernetes enabled. |
|
A container that generates the inventory targets for Prometheus server. Part of the Kubernetes Prometheus Metrics plugin. |
|
A container used to collect and process metrics for a node. Part of the Kubernetes Prometheus Metrics plugin. |
|
A container that runs a proxy for the metrics server. Part of the Kubernetes Prometheus Metrics plugin. |
|
A container that provides node feature discovery labels for Kubernetes nodes. |
|
A container that provides node feature discovery labels for Kubernetes nodes. |
|
A container that provides support for Multi Instance GPU (MIG) on NVIDIA GPUs. |
|
A container that provides a per-node service that manages the encryption state of the data plane. |
|
A container that provides the key management process that configures and periodically rotates the encryption keys. |
MKE component |
Description |
|---|---|
|
The pause container for the |
|
The pause container for the
|
|
The pause container for the |
|
The pause container for
|
|
The pause container for the ucp-coredns Pod. |
|
The pause container for
|
|
The pause container for
|
|
The pause container for
|
|
The pause container for the ucp-metrics. |
|
The pause container for the node feature discovery labels on Kubernetes nodes. |
|
A pause container for
|
|
A pause container for
|
Worker nodes¶
Worker nodes are instances of MCR that participate in a swarm for the purpose of executing containers. Such nodes receive and execute tasks dispatched from manager nodes. Worker nodes must have at least one manager node, as they do not participate in the Raft distributed state, perform scheduling, or serve the swarm mode HTTP API.
Note
Some Kubernetes components are run as Swarm services because the MKE control plane is itself a Docker Swarm cluster.
The following tables detail the MKE services that run on worker nodes.
MKE component |
Description |
|---|---|
|
A container for collecting host information regarding disks and hardware. |
|
A service that manages Interlock configuration. |
|
A helper service that reconfigures the |
|
A service that provides load balancing and proxying for swarm workloads. Only runs when you enable layer 7 routing. |
|
The networking proxy running on every node, which enables Pods to
contact Kubernetes services and other Pods through cluster IP
addresses. Named |
|
The Kubernetes node agent running on every node, which is responsible
for running Kubernetes Pods, reporting the health of the node, and
monitoring resource usage. Named |
|
A service that removes all the Kubernetes Pods that remain once Kubernetes components are removed from Windows nodes. Runs only on Windows nodes. |
|
A TLS proxy that allows secure access from the local Mirantis Container Runtime to MKE components. |
|
The Calico node agent that coordinates networking fabric for Windows nodes according to the cluster-wide Calico configuration. Runs on Windows nodes when Kubernetes is set as the orchestrator. |
|
A Calico component that runs on every machine that provides endpoints. Runs on Windows nodes when Kubernetes is set as the orchestrator. |
|
A service that monitors the worker node and ensures that the correct MKE
services are running. The |
MKE component |
Description |
|---|---|
|
An MKE service that accounts for the removal of dockershim from Kubernetes as of version 1.24, thus enabling MKE to continue using Docker as the container runtime. |
|
The Calico node agent that coordinates networking fabric according to the cluster-wide Calico configuration. Part of the calico-node DaemonSet. Runs on all nodes. |
|
An init container for |
|
A container that installs the Calico CNI plugin
binaries and configuration on each host. Part of the |
|
A container that provides node feature discovery labels for Kubernetes nodes. |
|
A container that provides node feature discovery labels for Kubernetes nodes. |
|
A container that provides supports for Multi Instance GPU (MIG) on NVIDIA GPUs. |
|
A container that provides a per-node service that manages the encryption state of the data plane. |
MKE component |
Description |
|---|---|
|
The pause container for the Calico-node Pod. This container is hidden by default, but you can see it by running the following command: docker ps -a |
|
The pause container for the node feature discovery labels on Kubernetes nodes. |
|
The pause container for
|
|
The pause container for
|
Admission controllers¶
Admission controllers are plugins that govern and enforce cluster usage. There are two types of admission controllers: default and custom. The tables below list the available admission controllers. For more information, see Kubernetes documentation: Using Admission Controllers.
Note
You cannot enable or disable custom admission controllers.
Name |
Description |
|---|---|
|
Adds a default storage class to |
|
Sets the pod default forgiveness toleration to tolerate the
|
|
Ensures that incoming requests do not violate the constraints in a
namespace |
|
Calls any mutating webhooks that match the request. |
|
Ensures that users cannot create new objects in namespaces undergoing
termination and that MKE rejects requests in nonexistent namespaces.
It also prevents users from deleting the reserved |
|
Limits the |
|
Attaches region or zone labels automatically to |
|
Limits which node selectors can be used within a namespace by reading a namespace annotation and a global configuration. |
|
Determines whether a new or modified pod should be admitted based on the requested security context and the available Pod Security Policies. |
|
Observes incoming requests and ensures they do not violate any of the
constraints in a namespace |
|
Implements automation for |
|
Calls any validating webhooks that match the request. |
Name |
Description |
|---|---|
|
|
|
Enforces MKE Docker Content Trust policy which, if enabled, requires that all pods use container images that have been digitally signed by trusted and authorized users, which are members of one or more teams in MKE. |
|
Adds a |
Pause containers¶
Every Kubernetes Pod includes an empty pause container, which bootstraps the Pod to establish all of the cgroups, reservations, and namespaces before its individual containers are created. The pause container image is always present, so the pod resource allocation happens instantaneously as containers are created.
To display pause containers:
When using the client bundle, pause containers are hidden by default.
To display pause containers when using the client bundle:
docker ps -a | grep -I pause
To display pause containers when not using the client bundle:
Log in to a manager or worker node.
Display pause containers:
docker ps | grep -I pause
Example output on a manager node:
5aeeafb80e8f mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_calico-kube-controllers-86565cb444-rwlrd_kube-system_fdd491cc-94e4-4510-a080-396454f2798c_0 ea4a1263398d mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_ucp-node-feature-discovery-59btp_node-feature-discovery_ef7a6f29-e3d4-4430-9c75-22940208f616_0 951f6622f8de d50ea4c05222 "/pause" 2 hours ago Up 2 hours k8s_ucp-pause_ucp-nvidia-device-partitioner-77qq5_kube-system_59d95409-721e-48f3-9524-97f1d30e63a4_0 f99ab238282e mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_ucp-nvidia-device-partitioner-77qq5_kube-system_59d95409-721e-48f3-9524-97f1d30e63a4_0 eec3d297e7a2 mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_ucp-metrics-6sf2z_kube-system_de4f67d3-99cc-4d00-a4f1-ccad66c31ebc_0 5a40fdc669b1 mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_compose-api-cb58448cc-xfb5g_kube-system_d1c8c8d2-9b81-475f-9cd4-9f486d3ace97_0 8e5897a13cd6 mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_coredns-9d5479b97-gmwct_kube-system_9c89d798-ff47-4194-b5d3-1ba3368698bc_0 d308274689a4 mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_coredns-9d5479b97-sxnb2_kube-system_74a70909-771c-4dce-9518-d49129e3645c_0 c45bf83d032a mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_compose-69d4dc8c69-f56ql_kube-system_64646ec3-f9e8-4cce-aeb1-37636e1858ce_0 c32ea1407b28 mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_calico-node-j9fmw_kube-system_0939bae3-0659-4608-8547-0b4095d99cc5_0
Example output on a worker node:
c5e836c38435 mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_ucp-node-feature-discovery-wztkl_node-feature-discovery_efe87dc1-349e-47f2-a98f-67d4675f6d9b_0 0f66550f654e d50ea4c05222 "/pause" 2 hours ago Up 2 hours k8s_ucp-pause_ucp-nvidia-device-partitioner-bq5th_kube-system_873f6045-8f61-4a55-9d00-d55b27e8f2c9_0 753efca985ef mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_calico-node-xx28v_kube-system_6c024ae0-8d27-4d89-a327-8ca635b37f79_0 7f2eda992ea6 mirantis/ucp-pause:3.6.0 "/pause" 2 hours ago Up 2 hours k8s_POD_ucp-nvidia-device-partitioner-bq5th_kube-system_873f6045-8f61-4a55-9d00-d55b27e8f2c9_0
See also
Volumes¶
MKE uses named volumes to persist data on all nodes on which it runs.
Volume name |
Contents |
|---|---|
|
Certificate and keys for the authentication and authorization service. |
|
Certificate and keys for the authentication and authorization store. |
|
Data of the authentication and authorization store, replicated across managers. |
|
Certificate and keys for authentication worker. |
|
Data of the authentication worker. |
|
Root key material for the MKE root CA that issues client certificates. |
|
Root key material for the MKE root CA that issues certificates for swarm members. |
|
Certificate and keys that the MKE web server uses to communicate with other MKE components. |
|
Certificate and keys for the MKE web server running in the node. |
|
MKE configuration data, replicated across managers. |
|
Certificates and keys for the key-value store. |
|
Monitoring data that MKE gathers. |
|
Configuration file that the |
|
Certificate and keys for node communication. |
|
Backup artifacts that are created while processing a backup. The artifacts persist on the volume for the duration of the backup and are cleaned up when the backup completes, though the volume itself remains. |
|
Symlinks to MKE component log files, created by |
|
Audit logs streamed by |
Volume name |
Contents |
|---|---|
|
Certificate and keys for node communication. |
|
Symlinks to MKE component log files, created by |
You can customize the volume driver for the volumes by creating the volumes prior to installing MKE. During installation, MKE determines which volumes do not yet exist on the node and creates those volumes using the default volume driver.
By default, MKE stores the data for these volumes at
/var/lib/docker/volumes/<volume-name>/_data.
Configuration¶
The table below presents the configuration files in use by MKE:
Configuration file name |
Description |
|---|---|
|
Configuration of the Interlock extension service that monitors and configures the proxy service |
|
Configuration of the service that handles and routes user requests |
|
MKE license |
|
Configuration of the core Interlock service |
Web UI and CLI¶
You can interact with MKE either through the web UI or the CLI.
With the MKE web UI you can manage your swarm, grant and revoke user permissions, deploy, configure, manage, and monitor your applications.
In addition, MKE exposes the standard Docker API, so you can continue using such existing tools as the Docker CLI client. As MKE secures your cluster with RBAC, you must configure your Docker CLI client and other client tools to authenticate your requests using client certificates that you can download from your MKE profile page.
Role-based access control¶
MKE allows administrators to authorize users to view, edit, and use cluster resources by granting role-based permissions for specific resource sets.
To authorize access to cluster resources across your organization, high-level actions that MKE administrators can take include the following:
Add and configure subjects (users, teams, organizations, and service accounts).
Define custom roles (or use defaults) by adding permitted operations per resource type.
Group cluster resources into resource sets of Swarm collections or Kubernetes namespaces.
Create grants by combining subject, role, and resource set.
Note
Only administrators can manage Role-based access control (RBAC).
The following table describes the core elements used in RBAC:
Element |
Description |
|---|---|
Subjects |
Subjects are granted roles that define the permitted operations for one or more resource sets and include:
|
Roles |
Roles define what operations can be done by whom. A role is a set of permitted operations for a type of resource, such as a container or volume. It is assigned to a user or a team with a grant. For example, the built-in Most organizations use multiple roles to fine-tune the appropriate access for different subjects. Users and teams may have different roles for the different resources they access. |
Resource sets |
Users can group resources into two types of resource sets to control user access: Docker Swarm collections and Kubernetes namespaces.
|
Grants |
Grants consist of a subject, role, and resource set, and define how specific users can access specific resources. All the grants of an organization taken together constitute an access control list (ACL), which is a comprehensive access policy for the organization. |
For complete information on how to configure and use role-based access control in MKE, refer to Authorize role-based access.
MKE limitations¶
MKE does not support user namespaces for nodes.
Due to a Kubernetes limitation, MKE containers do not run in Hyper-V isolation mode on Windows.
For more information on Hyper-V isolation and Kubernetes, refer to:
See also
Installation Guide¶
The MKE Installation Guide provides everything you need to install and configure Mirantis Kubernetes Engine (MKE). The guide offers detailed information, procedures, and examples that are specifically designed to help DevOps engineers and administrators install and configure the MKE container orchestration platform.
Plan the deployment¶
Default install directories¶
The following table details the default MKE install directories:
Path |
Description |
|---|---|
|
Docker data root directory |
|
kubelet data root directory (created with |
|
containerd data root directory (created with |
Host name strategy¶
Before installing MKE, plan a single host name strategy to use consistently throughout the cluster, keeping in mind that MKE and MCR both use host names.
There are two general strategies for creating host names: short host names and fully qualified domain names (FQDN). Consider the following examples:
Short host name:
engine01Fully qualified domain name:
node01.company.example.com
MCR considerations¶
A number of MCR considerations must be taken into account when deploying any MKE cluster.
default-address-pools¶
MCR uses three separate IP ranges for the docker0, docker_gwbridge, and
ucp-bridge interfaces. By default, MCR assigns the first available subnet
in default-address-pools (172.17.0.0/16) to docker0, the second
(172.18.0.0/16) to docker_gwbridge, and the third (172.19.0.0/16)
to ucp-bridge.
Note
The ucp-bridge bridge network specifically supports MKE component
containers.
You can reassign the docker0, docker_gwbridge, and ucp-bridge
subnets in default-address-pools. To do so, replace the relevant values in
default-address-pools in the /etc/docker/daemon.json file, making sure
that the setting includes at least three IP pools. Be aware that you must
restart the docker.service to activate your daemon.json file edits.
By default, default-address-pools contains the following values:
{
"default-address-pools": [
{"base":"172.17.0.0/16","size":16}, <-- docker0
{"base":"172.18.0.0/16","size":16}, <-- docker_gwbridge
{"base":"172.19.0.0/16","size":16}, <-- ucp-bridge
{"base":"172.20.0.0/16","size":16},
{"base":"172.21.0.0/16","size":16},
{"base":"172.22.0.0/16","size":16},
{"base":"172.23.0.0/16","size":16},
{"base":"172.24.0.0/16","size":16},
{"base":"172.25.0.0/16","size":16},
{"base":"172.26.0.0/16","size":16},
{"base":"172.27.0.0/16","size":16},
{"base":"172.28.0.0/16","size":16},
{"base":"172.29.0.0/16","size":16},
{"base":"172.30.0.0/16","size":16},
{"base":"192.168.0.0/16","size":20}
]
}
Parameter |
Description |
|---|---|
|
The list of CIDR ranges used to allocate subnets for local bridge networks. |
|
The CIDR range allocated for bridge networks in each IP address pool. |
|
The CIDR netmask that determines the subnet size to allocate from the
base pool. If the |
For example, {"base":"192.168.0.0/16","size":20} allocates
/20 subnets from 192.168.0.0/16, including the following subnets for
bridge networks:
192.168.0.0/20 (192.168.0.1 - 192.168.15.255)
192.168.16.0/20 (192.168.16.1 - 192.168.31.255)
192.168.32.0/20 (192.168.32.1 - 192.168.47.255)
192.168.48.0/20 (192.168.48.1 - 192.168.63.255)
192.168.64.0/20 (192.168.64.1 - 192.168.79.255)
…
192.168.240.0/20 (192.168.240.1 - 192.168.255.255)
docker0¶
MCR creates and configures the host system with the docker0 virtual network
interface, an ethernet bridge through which all traffic between MCR
and the container moves. MCR uses docker0 to handle all container
routing. You can specify an alternative network interface when you start the
container.
MCR allocates IP addresses from the docker0 configurable IP range to the
containers that connect to docker0. The default IP range, or subnet, for
docker0 is 172.17.0.0/16.
You can change the docker0 subnet in /etc/docker/daemon.json using the
settings in the following table. Be aware that you must restart the
docker.service to activate your daemon.json file edits.
Parameter |
Description |
|---|---|
|
Modify the first pool in Caution By default, MCR assigns the second pool to {
"default-address-pools": [
{"base":"172.17.0.0/16","size":16}, <-- Modify this value
{"base":"172.18.0.0/16","size":16},
{"base":"172.19.0.0/16","size":16},
{"base":"172.20.0.0/16","size":16},
{"base":"172.21.0.0/16","size":16},
{"base":"172.22.0.0/16","size":16},
{"base":"172.23.0.0/16","size":16},
{"base":"172.24.0.0/16","size":16},
{"base":"172.25.0.0/16","size":16},
{"base":"172.26.0.0/16","size":16},
{"base":"172.27.0.0/16","size":16},
{"base":"172.28.0.0/16","size":16},
{"base":"172.29.0.0/16","size":16},
{"base":"172.30.0.0/16","size":16},
{"base":"192.168.0.0/16","size":20}
]
}
|
|
Configures a CIDR range. Customize the subnet for {
"fixed-cidr": "172.17.0.0/16",
}
|
|
Configures a gateway IP address and CIDR netmask of the Customize the subnet for {
"bip": "172.17.0.0/16",
}
|
docker_gwbridge¶
The docker_gwbridge is a virtual network interface that connects
overlay networks (including ingress) to individual MCR container networks.
Initializing a Docker swarm or joining a Docker host to a swarm automatically
creates docker_gwbridge in the kernel of the Docker host. The default
docker_gwbridge subnet (172.18.0.0/16) is the second available subnet
in default-address-pools.
To change the docker_gwbridge subnet, open daemon.json and modify the
second pool in default-address-pools:
{
"default-address-pools": [
{"base":"172.17.0.0/16","size":16},
{"base":"172.18.0.0/16","size":16}, <-- Modify this value
{"base":"172.19.0.0/16","size":16},
{"base":"172.20.0.0/16","size":16},
{"base":"172.21.0.0/16","size":16},
{"base":"172.22.0.0/16","size":16},
{"base":"172.23.0.0/16","size":16},
{"base":"172.24.0.0/16","size":16},
{"base":"172.25.0.0/16","size":16},
{"base":"172.26.0.0/16","size":16},
{"base":"172.27.0.0/16","size":16},
{"base":"172.28.0.0/16","size":16},
{"base":"172.29.0.0/16","size":16},
{"base":"172.30.0.0/16","size":16},
{"base":"192.168.0.0/16","size":20}
]
}
Caution
Modifying the first pool to customize the
docker0subnet can affect the defaultdocker_gwbridgesubnet. Refer to docker0 for more information.You can only customize the
docker_gwbridgesettings before you join the host to the swarm or after temporarily removing it.
Docker swarm¶
The default address pool that Docker Swarm uses for its overlay network is
10.0.0.0/8. If this pool conflicts with your current network
implementation, you must use a custom IP address pool. Prior to installing MKE,
specify your custom address pool using the --default-addr-pool
option when initializing swarm.
Note
The Swarm default-addr-pool and MCR default-address-pools settings
define two separate IP address ranges used for different purposes.
A node.Status.Addr of 0.0.0.0 can cause unexpected problems. To
prevent any such issues, add the --advertise-addr flag to the
docker swarm join command.
To resolve the 0.0.0.0 situation, initiate the following workaround:
Stop the docker daemon that has
.Status.Addr 0.0.0.0.In the
/var/lib/docker/swarm/docker-state.jsonfile, apply the correct node IP toAdvertiseAddrandLocalAddr.Start the docker daemon.
Example result:
`{"LocalAddr":"","RemoteAddr":"10.200.200.10:2377","ListenAddr":"0.0.0.0:2377","AdvertiseAddr":"","DataPathAddr":"","DefaultAddressPool":null,"SubnetSize":0,"DataPathPort":0,"JoinInProgress":false,"FIPS":false}`
to
`{"LocalAddr":"10.200.200.13","RemoteAddr":"","ListenAddr":"0.0.0.0:2377","AdvertiseAddr":"10.200.200.13:2377","DataPathAddr":"","DefaultAddressPool":null,"SubnetSize":0,"DataPathPort":0,"JoinInProgress":false,"FIPS":false}
Kubernetes¶
Kubernetes uses two internal IP ranges, either of which can overlap and conflict with the underlying infrastructure, thus requiring custom IP ranges.
- The pod network
Either Calico or Azure IPAM services gives each Kubernetes pod an IP address in the default
192.168.0.0/16range. To customize this range, during MKE installation, use the--pod-cidrflag with the ucp install command.- The services network
You can access Kubernetes services with a VIP in the default
10.96.0.0/16Cluster IP range. To customize this range, during MKE installation, use the--service-cluster-ip-rangeflag with the ucp install command.
See also
docker data-root¶
The storage path for such persisted data as images, volumes, and cluster state
is docker data root (data-root in /etc/docker/daemon.json).
MKE clusters require that all nodes have the same docker data-root for the
Kubernetes network to function correctly. In addition, if the data-root is
changed on all nodes you must recreate the Kubernetes network configuration in
MKE by running the following commands:
kubectl -n kube-system delete configmap/calico-config
kubectl -n kube-system delete ds/calico-node deploy/calico-kube-controllers
See also
no-new-privileges¶
The no-new-privileges setting prevents the container application processes
from gaining new privileges during the execution process.
For most Linux distributions, MKE supports setting no-new-privileges to
true in the /etc/docker/daemon.json file. The parameter is not,
however, supported on RHEL 7.9, CentOS 7.9, Oracle Linux 7.8, and Oracle Linux
7.9.
This option is not supported on Windows. It is a Linux kernel feature.
Device Mapper storage driver¶
MCR hosts that run the devicemapper storage driver use the loop-lvm
configuration mode by default. This mode uses sparse files to build
the thin pool used by image and container snapshots and is designed to work
without any additional configuration.
Note
Mirantis recommends that you use direct-lvm mode in production
environments in lieu of loop-lvm mode. direct-lvm mode is more
efficient in its use of system resources than loop-lvm mode, and you can
scale it as necessary.
For information on how to configure direct-lvm mode, refer to the Docker
documentation, Use the Device Mapper storage driver.
Memory metrics reporting¶
To report accurate memory metrics, MCR requires that you enable specific kernel settings that are often disabled on Ubuntu and Debian systems. For detailed instructions on how to do this, refer to the Docker documentation, Your kernel does not support cgroup swap limit capabilities.
Perform pre-deployment configuration¶
Configure networking¶
A well-configured network is essential for the proper functioning of your MKE deployment. Pay particular attention to such key factors as IP address provisioning, port management, and traffic enablement.
IP considerations¶
Before installing MKE, adopt the following practices when assigning IP addresses:
Ensure that your network and nodes support using a static IPv4 address and assign one to every node.
Avoid IP range conflicts. The following table lists the recommended addresses you can use to avoid IP range conflicts:
Component
Subnet
Range
Recommended IP address
MCR
default-address-poolsCIDR range for interface and bridge networks
172.17.0.0/16 - 172.30.0.0/16, 192.168.0.0/16
Swarm
default-addr-poolCIDR range for Swarm overlay networks
10.0.0.0/8
Kubernetes
pod-cidrCIDR range for Kubernetes pods
192.168.0.0/16
Kubernetes
service-cluster-ip-rangeCIDR range for Kubernetes services
10.96.0.0/16
Minimum: 10.96.0.0/24
See also
Open ports to incoming traffic¶
When installing MKE on a host, you need to open specific ports to incoming traffic. Each port listens for incoming traffic from a particular set of hosts, known as the port scope.
MKE uses the following scopes:
Scope |
Description |
|---|---|
External |
Traffic arrives from outside the cluster through end-user interaction. |
Internal |
Traffic arrives from other hosts in the same cluster. |
Self |
Traffic arrives to Self ports only from processes on the same host. These ports, however, do not need to be open to outside traffic. |
Open the following ports for incoming traffic on each host type:
Hosts |
Port |
Scope |
Purpose |
|---|---|---|---|
Managers, workers |
TCP 179 |
Internal |
BGP peers, used for Kubernetes networking |
Managers |
TCP 443 (configurable) |
External, internal |
MKE web UI and API |
Managers |
TCP 2376 (configurable) |
Internal |
Docker swarm manager, used for backwards compatibility |
Managers |
TCP 2377 (configurable) |
Internal |
Control communication between swarm nodes |
Managers, workers |
UDP 4789 |
Internal |
Overlay networking |
Managers |
TCP 6443 (configurable) |
External, internal |
Kubernetes API server endpoint |
Managers, workers |
TCP 6444 |
Self |
Kubernetes API reverse proxy |
Managers, workers |
TCP, UDP 7946 |
Internal |
Gossip-based clustering |
Managers, workers |
TCP 9091 |
Internal |
Felix Prometheus |
Managers |
TCP 9094 |
Self |
Felix Prometheus |
Managers, workers |
TCP 9099 |
Self |
Calico health check |
Managers, workers |
TCP 10248 |
Self |
Kubelet health check |
Managers, workers |
TCP 10250 |
Internal |
Kubelet |
Managers, workers |
TCP 12376 |
Internal |
TLS authentication proxy that provides access to MCR |
Managers, workers |
TCP 12378 |
Self |
etcd reverse proxy |
Managers |
TCP 12379 |
Internal |
etcd Control API |
Managers |
TCP 12380 |
Internal |
etcd Peer API |
Managers |
TCP 12381 |
Internal |
MKE cluster certificate authority |
Managers |
TCP 12382 |
Internal |
MKE client certificate authority |
Managers |
TCP 12383 |
Internal |
Authentication storage backend |
Managers |
TCP 12384 |
Internal |
Authentication storage backend for replication across managers |
Managers |
TCP 12385 |
Internal |
Authentication service API |
Managers |
TCP 12386 |
Internal |
Authentication worker |
Managers |
TCP 12387 |
Internal |
Prometheus server |
Managers |
TCP 12388 |
Internal |
Kubernetes API server |
Managers, workers |
TCP 12389 |
Self |
Hardware Discovery API |
Cluster and service networking options¶
MKE supports the following cluster and service networking options:
Kube-proxy with iptables proxier, and either the managed CNI or an unmanaged alternative
Kube-proxy with ipvs proxier, and either the managed CNI or an unmanaged alternative
eBPF mode with either the managed CNI or an unmanaged alternative
You can configure cluster and service networking options at install time or in existing clusters. For detail on reconfiguring existing clusters, refer to Configure cluster and service networking in an existing cluster in the MKE Operations Guide.
Caution
Swarm workloads that require the use of encrypted overlay networks must use iptables proxier with either the managed CNI or an unmanaged alternative. Be aware that the other networking options detailed here automatically disable Docker Swarm encrypted overlay networks.
Solution component |
Develop and maintain |
Test and integrate with MKE |
First line support |
Product support |
|---|---|---|---|---|
Calico Open Source |
Community |
Mirantis |
Mirantis |
Tigera for Linux, Mirantis for Windows |
Calico Enterprise |
Tigera |
Tigera, for every major MKE release |
Mirantis |
Tigera, with customers paying for additional features |
Cilium Open Source |
Community |
Planned |
Mirantis |
Community or Isovalent |
Cilium Enterprise |
Isovalent |
Isovalent |
Mirantis |
Isovalent |
To enable kube-proxy with iptables proxier while using the managed CNI:
Using default option kube-proxy with iptables proxier is the
equivalent of specifying --kube-proxy-mode=iptables at install time. To
verify that the option is operational, confirm the presence of the following
line in the ucp-kube-proxy container logs:
I1027 05:35:27.798469 1 server_others.go:212] Using iptables Proxier.
To enable kube-proxy with ipvs proxier while using the managed CNI:
Prior to MKE installation, verify that the following kernel modules are available on all Linux manager and worker nodes:
ipvsip_vs_rrip_vs_wrrip_vs_shnf_conntrack_ipv4
Specify
--kube-proxy-mode=ipvsat install time.Optional. Once installation is complete, configure the following ipvs-related parameters in the MKE configuration file (otherwise, MKE will use the Kubernetes default parameter settings):
ipvs_exclude_cidrs = ""ipvs_min_sync_period = ""ipvs_scheduler = ""ipvs_strict_arp = falseipvs_sync_period = ""ipvs_tcp_timeout = ""ipvs_tcpfin_timeout = ""ipvs_udp_timeout = ""
For more information on using these parameters, refer to kube-proxy in the Kubernetes documentation.
Note
The ipvs-related parameters have no install time counterparts and therefore must only be configured once MKE installation is complete.
Verify that kube-proxy with ipvs proxier is operational by confirming the presence of the following lines in the
ucp-kube-proxycontainer logs:I1027 05:14:50.868486 1 server_others.go:274] Using ipvs Proxier. W1027 05:14:50.868822 1 proxier.go:445] IPVS scheduler not specified, use rr by default
To enable eBPF mode while using the managed CNI:
Verify that the prerequisites for eBPF use have been met, including kernel compatibility, for all Linux manager and worker nodes. Refer to the Calico documentation Enable the eBPF dataplane for more information.
Specify
--calico-ebpf-enabledat install time.Verify that eBPF mode is operational by confirming the presence of the following lines in the
ucp-kube-proxycontainer logs:KUBE_PROXY_MODE (disabled) CLEANUP_ON_START_DISABLED true "Sleeping forever...."
To enable kube-proxy with iptables proxier while using an unmanaged CNI.
Specify
--unmanaged-cniat install time.Verify that kube-proxy with iptables proxier is operational by confirming the presence of the following line in the
ucp-kube-proxycontainer logs:I1027 05:35:27.798469 1 server_others.go:212] Using iptables Proxier.
To enable kube-proxy with ipvs proxier while using an unmanaged CNI:
Specify the following parameters at install time:
--unmanaged-cni--kube-proxy-mode=ipvs
Verify that kube-proxy with ipvs proxier is operational by confirming the presence of the following lines in the
ucp-kube-proxycontainer logs:I1027 05:14:50.868486 1 server_others.go:274] Using ipvs Proxier. W1027 05:14:50.868822 1 proxier.go:445] IPVS scheduler not specified, use rr by default
To enable eBPF mode while using an unmanaged CNI:
Verify that the prerequisites for eBPF use have been met, including kernel compatibility, for all Linux manager and worker nodes. Refer to the Calico documentation Enable the eBPF dataplane for more information.
Specify the following parameters at install time:
--unmanaged-cni--kube-proxy-mode=disabled--kube-default-drop-masq-bits
Verify that eBPF mode is operational by confirming the presence of the following lines in
ucp-kube-proxycontainer logs:KUBE_PROXY_MODE (disabled) CLEANUP_ON_START_DISABLED true "Sleeping forever...."
Calico networking¶
Calico is the default networking plugin for MKE. The default Calico encapsulation setting for MKE is VXLAN, however the plugin also supports IP-in-IP encapsulation. Refer to the Calico documentation on Overlay networking for more information.
Important
NetworkManager can impair the Calico agent routing function. To resolve
this issue, you must create a file called
/etc/NetworkManager/conf.d/calico.conf with the following content:
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:wireguard.cali
Enable ESP traffic¶
For overlay networks with encryption to function, you must allow IP protocol 50 Encapsulating Security Payload (ESP) traffic.
If you are running RHEL 8.x, Rocky Linux 8.x, or CentOS 8, install kernel
module xt_u32:
sudo dnf install kernel-modules-extra
Avoid firewall conflicts¶
Avoid firewall conflicts in the following Linux distributions:
Linux distribution |
Procedure |
|---|---|
SUSE Linux Enterprise Server 12 SP2 |
Installations have the To turn off the
|
SUSE Linux Enterprise Server 12 SP3 |
No change is required, as installations have the |
DNS entry in hosts file¶
MKE adds the proxy.local DNS entry to the following files at install
time:
- Linux
/etc/hosts- Windows
c:\Windows\System32\Drivers\etc\hosts
To configure MCR to connect to the Internet using HTTP_PROXY you
must set the value of proxy.local to NOPROXY.
Preconfigure an SLES installation¶
Before performing SUSE Linux Enterprise Server (SLES) installations, consider the following prerequisite steps:
For SLES 15 installations, disable
CLOUD_NETCONFIG_MANAGEprior to installing MKE:Set
CLOUD_NETCONFIG_MANAGE="no"in the/etc/sysconfig/network/ifcfg-eth0network interface configuration file.Run the service network restart command.
By default, SLES disables connection tracking. To allow Kubernetes controllers in Calico to reach the Kubernetes API server, enable connection tracking on the loopback interface for SLES by running the following commands for each node in the cluster:
sudo mkdir -p /etc/sysconfig/SuSEfirewall2.d/defaults echo FW_LO_NOTRACK=no | sudo tee \ /etc/sysconfig/SuSEfirewall2.d/defaults/99-docker.cfg sudo SuSEfirewall2 start
See also
Verify the timeout settings¶
Confirm that MKE components have the time they require to effectively communicate.
Component |
Timeout (ms) |
Configurable |
|---|---|---|
Raft consensus between manager nodes |
3000 |
no |
Gossip protocol for overlay networking |
5000 |
no |
etcd |
500 |
yes |
RethinkDB |
10000 |
no |
Stand-alone cluster |
90000 |
no |
Network lag of more than two seconds between MKE manager nodes can cause problems in your MKE cluster. For example, such a lag can indicate to MKE components that the other nodes are down, resulting in unnecessary leadership elections that will result in temporary outages and reduced performance. To resolve the issue, decrease the latency of the MKE node communication network.
Configure time synchronization¶
Configure all containers in an MKE cluster to regularly synchronize with a Network Time Protocol (NTP) server, to ensure consistency between all containers in the cluster and to circumvent unexpected behavior that can lead to poor performance.
Install NTP on every machine in your cluster:
sudo apt-get update && sudo apt-get install ntp ntpdate
sudo yum install ntp ntpdate sudo systemctl start ntpd sudo systemctl enable ntpd sudo systemctl status ntpd sudo ntpdate -u -s 0.centos.pool.ntp.org sudo systemctl restart ntpd
sudo zypper ref && zypper install ntp
In addition to installing NTP, the command sequence starts ntpd, a daemon that periodically syncs the machine clock to a central server.
Sync the machine clocks:
sudo ntpdate pool.ntp.org
Verify that the time of each machine is in sync with the NTP servers:
sudo ntpq -p
Example output, which illustrates how much the machine clock is out of sync with the NTP servers:
remote refid st t when poll reach delay offset jitter ============================================================================== 45.35.50.61 139.78.97.128 2 u 24 64 1 60.391 4623378 0.004 time-a.timefreq .ACTS. 1 u 23 64 1 51.849 4623377 0.004 helium.constant 128.59.0.245 2 u 22 64 1 71.946 4623379 0.004 tock.usshc.com .GPS. 1 u 21 64 1 59.576 4623379 0.004 golem.canonical 17.253.34.253 2 u 20 64 1 145.356 4623378 0.004
Configure a load balancer¶
Though MKE does not include a load balancer, you can configure your own to balance user requests across all manager nodes. Before that, decide whether you will add nodes to the load balancer using their IP address or their fully qualified domain name (FQDN), and then use that strategy consistently throughout the cluster. Take note of all IP addresses or FQDNs before you start the installation.
If you plan to deploy both MKE and MSR, your load balancer must be able to differentiate between the two: either by IP address or port number. Because both MKE and MSR use port 443 by default, your options are as follows:
Configure your load balancer to expose either MKE or MSR on a port other than 443.
Configure your load balancer to listen on port 443 with separate virtual IP addresses for MKE and MSR.
Configure separate load balancers for MKE and MSR, both listening on port 443.
If you want to install MKE in a high-availability configuration with a load
balancer in front of your MKE controllers, include the
appropriate IP address and FQDN for the load balancer VIP. To do so, use one
or more --san flags either with the ucp install command or in
interactive mode when MKE requests additional SANs.
Configure IPVS¶
MKE supports the setting of values for all IPVS related parameters that are exposed by kube-proxy.
Kube-proxy runs on each cluster node, its role being to load-balance traffic whose destination is services (via cluster IPs and node ports) to the correct backend pods. Of the modes in which kube-proxy can run, IPVS (IP Virtual Server) offers the widest choice of load balancing algorithms and superior scalability.
Refer to the Calico documentation, Comparing kube-proxy modes: iptables or IPVS? for detailed information on IPVS.
Caution
You can only enable IPVS for MKE at installation, and it persists throughout the life of the cluster. Thus, you cannot switch to iptables at a later stage or switch over existing MKE clusters to use IPVS proxier.
MKE supports setting values for all IPVS-related parameters. For full parameter details, refer to the Kubernetes documentation for kube-proxy.
Use the kube-proxy-mode parameter at install time to enable IPVS proxier. The
two valid values are iptables (default) and ipvs.
You can specify the following ipvs parameters for kube-proxy:
ipvs_exclude_cidrsipvs_min_sync_periodipvs_scheduleripvs_strict_arp = falseipvs_sync_periodipvs_tcp_timeoutipvs_tcpfin_timeoutipvs_udp_timeout
To set these values at the time of bootstrap/installation:
Add the required values under [cluster_config] in a TOML file (for example,
config.toml).Create a config named
com.docker.ucp.configfrom this TOML file:docker config create com.docker.ucp.config config.toml
Use the
--existing-configparameter when installing MKE. You can also change these values post-install using theMKE-s ucp/config-tomlendpoint.
Caution
If you are using MKE 3.3.x with IPVS proxier and plan to upgrade to MKE 3.4.x, you must upgrade to MKE 3.4.3 or later as earlier versions of MKE 3.4.x do not support IPVS proxier.
Customize named volumes¶
Note
Skip this step if you want to use the default named volumes.
MKE uses named volumes to persist data. If you want to customize the drivers that manage such volumes, create the volumes before installing MKE. During the installation process, the installer will automatically detect the existing volumes and start using them. Otherwise, MKE will create the default named volumes.
Configure kernel parameters¶
MKE uses a number of kernel parameters in its deployment.
Note
The MKE parameter values are not set by MKE, but by either MCR or
an upstream component.
kernel.<subtree>¶
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets the number of seconds the kernel waits to reboot following a panic. Note The |
|
|
Sets whether the kernel should panic on an oops rather than continuing to attempt operations. Note The |
|
|
Sets the maximum number of keys that the root user ( Note The |
|
|
Sets the maximum number of bytes of data that the root user ( Allocate 25 bytes per key multiplied by the number of kernel/keys/root_maxkeys. Note The |
|
|
Sets the number of open PTYs. |
net.bridge.bridge-nf-<subtree>¶
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets whether |
|
|
Sets whether |
|
|
Sets whether |
|
|
Sets whether netfilter rules apply to bridged PPPOE network traffic. If the bridge module is not loaded, and thus no bridges are present, this key is not present. |
|
|
Sets whether netfilter rules apply to bridged VLAN network traffic. If the bridge module is not loaded, and thus no bridges are present, this key is not present. |
|
|
Sets whether netfilter strips the incoming VLAN interface name from bridged traffic. If the bridge module is not loaded, and thus no bridges are present, this key is not present. |
net.fan.<subtree>¶
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets the version of the VXLAN module on older kernels, not present on kernel version 5.x. If the VXLAN module is not loaded this key is not present. |
net.ipv4.<subtree>¶
Note
The
*.vs.*default values persist, changing only because theipvskernel module was not previously loaded. For more information, refer to the Linux kernel documentation.
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets whether ICMP redirects are permitted. This key affects all interfaces. |
|
|
Sets whether network traffic is forwarded. This key affects all interfaces. |
|
|
Sets |
|
|
Sets |
|
|
Sets forwarding for localhost traffic. |
|
|
Sets whether traffic forwards between interfaces. For Kubernetes to run,
this parameter must be set to |
|
|
Sets the always mode drop rate used in mode 3 of the |
|
|
Sets the available memory threshold in pages, which is used in the
automatic modes of defense. When there is not enough available memory,
this enables the strategy and the variable is set to |
|
|
Sets whether the director function is disabled while the server is in back-up mode, to avoid packet loops for DR/TUN methods. |
|
|
Sets whether packets forward directly to the original destination when
no cache server is available and the destination address is not local
( |
|
|
Sets how IPVS handles connections detected on port reuse. It is a bitmap with the following values:
|
|
|
Sets whether connection-tracking entries are maintained for connections
handled by IPVS. Enable if connections handled by IPVS
are to be subject to stateful firewall rules. That is, |
|
|
Sets whether entries are randomly dropped in the connection hash table,
to collect memory back for new connections. In the current
code, the The valid values of |
|
|
Sets whether rate packets are dropped prior to being forwarded to real servers. Rate 1 drops all incoming packets. The value definition is the same as that for |
|
|
Sets whether the load balancer silently drops packets when its destination server is not available. This can be useful when the user-space monitoring program deletes the destination server (due to server overload or wrong detection) and later adds the server back, and the connections to the server can continue. If this feature is enabled, the load balancer terminates the connection immediately whenever a packet arrives and its destination server is not available, after which the client program will be notified that the connection is closed. This is equivalent to the feature that is sometimes required to flush connections when the destination is not available. |
|
|
Sets whether IPVS configures the |
|
|
Sets whether ICMP error messages ( |
|
|
Sets whether all DF packets that exceed the PMTU are rejected with
|
|
|
Sets whether scheduling ICMP packets in IPVS is enabled. |
|
|
Sets the use of a more complicated TCP state transition table.
For VS/NAT, the |
|
|
Sets whether IPVS is permitted to create a connection state on any packet, rather than an SCTP INIT only. |
|
|
Sets whether IPVS is permitted to create a connection state on any packet, rather than a TCP SYN only. |
|
|
Sets whether the route of SNATed packets is recalculated from real servers as if they originate from the director. If disabled, SNATed packets are routed as if they have been forwarded by the director. If policy routing is in effect, then it is possible that the route of a packet originating from a director is routed differently to a packet being forwarded by the director. If policy routing is not in effect, then the recalculated route will always be the same as the original route. It is an optimization to disable snat_reroute and avoid the recalculation. |
|
|
Sets the synchronization of connections when using persistence. The possible values are defined as follows:
|
|
|
Sets the number of threads that the master and back-up servers can use
for sync traffic. Every thread uses a single UDP port, thread 0 uses the
default port |
|
|
Sets a hard limit for queued sync messages that are not yet sent. It defaults to 1/32 of the memory pages but actually represents number of messages. It will protect us from allocating large parts of memory when the sending rate is lower than the queuing rate. |
|
|
Sets (in seconds) the difference in the reported connection timer that triggers new sync messages. It can be used to avoid sync messages for the specified period (or half of the connection timeout if it is lower) if the connection state has not changed since last sync. This is useful for normal connections with high traffic, to reduce
sync rate. Additionally, retry |
|
|
Sets sync retries with period of |
|
|
Sets the configuration of SNDBUF (master) or RCVBUF (slave) socket limit. Default value is 0 (preserve system defaults). |
|
|
Sets the synchronization threshold, which is the minimum number of
incoming packets that a connection must receive before the
connection is synchronized. A connection will be synchronized every time
the number of its incoming packets modulus |
|
|
Sets the version of the synchronization protocol to use when sending synchronization messages. The possible values are:
Kernels with this |
net.netfilter.nf_conntrack_<subtree>¶
Note
The
net.netfilter.nf_conntrack_<subtree>default values persist, changing only when thenf_conntrackkernel module has not been previously loaded. For more information, refer to the Linux kernel documentation.
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets whether connection-tracking flow accounting is enabled. Adds 64-bit byte and packet counter per flow. |
|
|
Sets the size of the hash table. If not specified during module loading, the default size is calculated by dividing total memory by 16384 to determine the number of buckets. The hash table will never have fewer than 1024 and never more than 262144 buckets. This sysctl is only writeable in the initial net namespace. |
|
|
Sets whether the checksum of incoming packets is verified. Packets with bad checksums are in an invalid state. If this is enabled, such packets are not considered for connection tracking. |
|
|
Sets whether picking up already established connections for Datagram Congestion Control Protocol (DCCP) is permitted. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
Sets whether the connection tracking code provides userspace with connection-tracking events through ctnetlink. |
|
|
Sets the maximum size of the expectation table. The default value is nf_conntrack_buckets / 256. The minimum is 1. |
|
|
Sets the maximum memory used to reassemble IPv6 fragments. When
|
|
|
See |
|
|
Sets the time to keep an IPv6 fragment in memory. |
|
|
Sets the default for a generic timeout. This refers to layer 4 unknown and unsupported protocols. |
|
|
Set the GRE timeout from the conntrack table. |
|
|
Sets the GRE timeout for streamed connections. This extended timeout is used when a GRE stream is detected. |
|
|
Sets whether the automatic conntrack helper assignment is enabled.
If disabled, you must set up |
|
|
Sets the default for ICMP timeout. |
|
|
Sets the default for ICMP6 timeout. |
|
|
Sets whether invalid packets of a type specified by value are logged. |
|
|
Sets the maximum number of allowed connection tracking entries. This
value is set to Connection-tracking entries are added to the table twice, once for the original direction and once for the reply direction (that is, with the reversed address). Thus, with default settings a maxed-out table will have an average hash chain length of 2, not 1. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
Sets whether only out of window RST segments are marked as |
|
|
Sets whether already established connections are picked up. |
|
|
Sets the maximum number of packets that can be retransmitted without receiving an acceptable ACK from the destination. If this number is reached, a shorter timer is started. Timeout for unanswered. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
The parameter description is not yet available in the Linux kernel documentation. |
|
|
Sets whether connection-tracking flow timestamping is enabled. |
|
|
Sets the UDP timeout. |
|
|
Sets the extended timeout that is used whenever a UDP stream is detected. |
net.nf_conntrack_<subtree>¶
Note
The
net.nf_conntrack_<subtree>default values persist, changing only when thenf_conntrackkernel module has not been previously loaded. For more information, refer to the Linux kernel documentation.
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets the maximum number of connections to track. The size of this parameter is calculated based on system memory. |
vm.overcommit_<subtree>¶
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets whether the kernel permits memory overcommitment from Note The |
vm.panic_<subtree>¶
Parameter |
Values |
Description |
|---|---|---|
|
|
Sets whether the kernel should panic on an out-of-memory, rather than continuing to attempt operations. When set to Note The |
Set up kernel default protections¶
To protect kernel parameters from being overridden by kublet, you can either
invoke the --kube-protect-kernel-defaults command option at the time of MKE
install, or following MKE install you can adjust the cluster_config |
kube_protect_kernel_defaults parameter in the MKE configuration file.
Important
When enabled, kubelet can fail to start if the kernel parameters on the nodes are not properly set. You must set those kernel parameters on the nodes before you install MKE or before adding a new node to an existing cluster.
- Create a configuration file called
/etc/sysctl.d/90-kubelet.confand add the following snipped to it:
vm.panic_on_oom=0 vm.overcommit_memory=1 kernel.panic=10 kernel.panic_on_oops=1 kernel.keys.root_maxkeys=1000000 kernel.keys.root_maxbytes=25000000
Run sysctl -p /etc/sysctl.d/90-kubelet.conf
Install the MKE image¶
To install MKE:
Log in to the target host using Secure Shell (SSH).
Pull the latest version of MKE:
docker image pull mirantis/ucp:3.6.19
Install MKE:
docker container run --rm -it --name ucp \ -v /var/run/docker.sock:/var/run/docker.sock \ mirantis/ucp:3.6.19 install \ --host-address <node-ip-address> \ --interactive
The ucp install command runs in interactive mode, prompting you for the necessary configuration values. For more information about the ucp install command, including how to install MKE on a system with SELinux enabled, refer to the MKE Operations Guide: mirantis/ucp install.
Note
MKE installs Project Calico for Kubernetes container-to-container communication. However, you may install an alternative CNI plugin, such as Cilium, Weave, or Flannel. For more information, refer to the MKE Operations Guide: Installing an unmanaged CNI plugin.
See also
Obtain the license¶
After you Install the MKE image, proceed with downloading your MKE license as described below. This section also contains steps to apply your new license using the MKE web UI.
Warning
Users are not authorized to run MKE without a valid license. For more information, refer to Mirantis Agreements and Terms.
To download your MKE license:
Open an email from Mirantis Support with the subject Welcome to Mirantis’ CloudCare Portal and follow the instructions for logging in.
If you did not receive the CloudCare Portal email, it is likely that you have not yet been added as a Designated Contact. To remedy this, contact your Designated Administrator.
In the top navigation bar, click Environments.
Click the Cloud Name associated with the license you want to download.
Scroll down to License Information and click the License File URL. A new tab opens in your browser.
Click View file to download your license file.
To update your license settings in the MKE web UI:
Log in to your MKE instance using an administrator account.
In the left navigation, click Settings.
On the General tab, click Apply new license. A file browser dialog displays.
Navigate to where you saved the license key (
.lic) file, select it, and click Open. MKE automatically updates with the new settings.
Note
Though MKE is generally a subscription-only service, Mirantis offers a free trial license by request. Use our contact form to request a free trial license.
Install MKE on AWS¶
This section describes how to customize your MKE installation on AWS. It is for those deploying Kubernetes workloads while leveraging the AWS Kubernetes cloud provider, which provides dynamic volume and loadbalancer provisioning.
Note
You may skip this topic if you plan to install MKE on AWS with no customizations or if you will only deploy Docker Swarm workloads. Refer to Install the MKE image for the appropriate installation instruction.
Prerequisites¶
Complete the following prerequisites prior to installing MKE on AWS.
Log in to the AWS Management Console.
Assign a host name to your instance. To determine the host name, run the following curl command within the EC2 instance:
curl http://169.254.169.254/latest/meta-data/hostnameTag your instance, VPC, security-groups, and subnets by specifying
kubernetes.io/cluster/<unique-cluster-id>in the Key field and<cluster-type>in the Value field. Possible<cluster-type>values are as follows:owned, if the cluster owns and manages the resources that it createsshared, if the cluster shares its resources between multiple clusters
For example, Key:
kubernetes.io/cluster/1729543642a6and Value:owned.To enable introspection and resource provisioning, specify an instance profile with appropriate policies for manager nodes. The following is an example of a very permissive instance profile:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "elasticloadbalancing:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": [ "route53:*" ], "Resource": [ "*" ] }, { "Effect": "Allow", "Action": "s3:*", "Resource": [ "arn:aws:s3:::kubernetes-*" ] } ] }
To enable access to dynamically provisioned resources, specify an instance profile with appropriate policies for worker nodes. The following is an example of a very permissive instance profile:
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": "s3:*", "Resource": ["arn:aws:s3:::kubernetes-*"] }, { "Effect": "Allow", "Action": "ec2:Describe*", "Resource": "*" }, { "Effect": "Allow", "Action": "ec2:AttachVolume", "Resource": "*" }, { "Effect": "Allow", "Action": "ec2:DetachVolume", "Resource": "*" }, { "Effect": "Allow", "Action": ["route53:*"], "Resource": ["*"] } ] }
Install MKE¶
After you perform the steps described in Prerequisites, run the
following command to install MKE on a master node. Substitute <ucp-ip> with
the private IP address of the master node.
Warning
If your cluster includes Kubernetes Windows worker nodes, you must omit the
--cloud-provider aws flag from the following command, as its inclusion
causes the Kubernetes Windows worker nodes never to enter a healthy state.
docker container run --rm -it \
--name ucp \
--volume /var/run/docker.sock:/var/run/docker.sock \
mirantis/ucp:3.6.19 install \
--host-address <ucp-ip> \
--cloud-provider aws \
--interactive
Install MKE on Azure¶
Mirantis Kubernetes Engine (MKE) closely integrates with Microsoft Azure for its Kubernetes Networking and Persistent Storage feature set. MKE deploys the Calico CNI provider. In Azure, the Calico CNI leverages the Azure networking infrastructure for data path networking and the Azure IPAM for IP address management.
Prerequisites¶
To avoid significant issues during the installation process, you must meet the following infrastructure prerequisites to successfully deploy MKE on Azure.
Deploy all MKE nodes (managers and workers) into the same Azure resource group. You can deploy the Azure networking components (virtual network, subnets, security groups) in a second Azure resource group.
Size the Azure virtual network and subnet appropriately for your environment, because addresses from this pool will be consumed by Kubernetes Pods.
Attach all MKE worker and manager nodes to the same Azure subnet.
Set internal IP addresses for all nodes to
Staticrather than theDynamicdefault.Match the Azure virtual machine object name to the Azure virtual machine computer name and the node operating system hostname that is the FQDN of the host (including domain names). All characters in the names must be in lowercase.
Ensure the presence of an Azure Service Principal with
Contributoraccess to the Azure resource group hosting the MKE nodes. Kubernetes uses this Service Principal to communicate with the Azure API. The Service Principal ID and Secret Key are MKE prerequisites.If you are using a separate resource group for the networking components, the same Service Principal must have
Network Contributoraccess to this resource group.Ensure that an open NSG between all IPs on the Azure subnet passes into MKE during installation. Kubernetes Pods integrate into the underlying Azure networking stack, from an IPAM and routing perspective with the Azure CNI IPAM module. As such, Azure network security groups (NSG) impact pod-to-pod communication. End users may expose containerized services on a range of underlying ports, resulting in a manual process to open an NSG port every time a new containerized service deploys on the platform, affecting only workloads that deploy on the Kubernetes orchestrator.
To limit exposure, restrict the use of the Azure subnet to container host VMs and Kubernetes Pods. Additionally, you can leverage Kubernetes Network Policies to provide micro segmentation for containerized applications and services.
The MKE installation requires the following information:
subscriptionIdAzure Subscription ID in which to deploy the MKE objects
tenantIdAzure Active Directory Tenant ID in which to deploy the MKE objects
aadClientIdAzure Service Principal ID
aadClientSecretAzure Service Principal Secret Key
Networking¶
MKE configures the Azure IPAM module for Kubernetes so that it can allocate IP addresses for Kubernetes Pods. Per Azure IPAM module requirements, the configuration of each Azure VM that is part of the Kubernetes cluster must include a pool of IP addresses.
You can use automatic or manual IPs provisioning for the Kubernetes cluster on Azure.
- Automatic provisioning
Allows for IP pool configuration and maintenance for standalone Azure virtual machines (VMs). This service runs within the
calico-nodedaemonset and provisions 128 IP addresses for each node by default.Note
If you are using a VXLAN data plane, MKE automatically uses Calico IPAM. It is not necessary to do anything specific for Azure IPAM.
New MKE installations use Calico VXLAN as the default data plane (the MKE configuration
calico_vxlanis set totrue). MKE does not use Calico VXLAN if the MKE version is lower than 3.3.0 or if you upgrade MKE from lower than 3.3.0 to 3.3.0 or higher.
- Manual provisioning
Manual provisioning of additional IP address for each Azure VM can be done through the Azure Portal, the Azure CLI az network nic ip-config create, or an ARM template.
Azure configuration file¶
For MKE to integrate with Microsoft Azure, the azure.json configuration
file must be identical across all manager and worker nodes in your cluster. For
Linux nodes, place the file in /etc/kubernetes on each host. For Windows
nodes, place the file in C:\k on each host. Because root owns the
configuration file, set its permissions to 0644 to ensure that the
container user has read access.
The following is an example template for azure.json.
{
"cloud":"AzurePublicCloud",
"tenantId": "<parameter_value>",
"subscriptionId": "<parameter_value>",
"aadClientId": "<parameter_value>",
"aadClientSecret": "<parameter_value>",
"resourceGroup": "<parameter_value>",
"location": "<parameter_value>",
"subnetName": "<parameter_value>",
"securityGroupName": "<parameter_value>",
"vnetName": "<parameter_value>",
"useInstanceMetadata": true
}
Optional parameters are available for Azure deployments:
primaryAvailabilitySetNameWorker nodes availability set
vnetResourceGroupVirtual network resource group if your Azure network objects live in a separate resource group
routeTableNameApplicable if you have defined multiple route tables within an Azure subnet
Guidelines for IPAM configuration¶
Warning
To avoid significant issue during the installation process, follow these guidelines to either use the appropriate size network in Azure or take the necessary actions to fit within the subnet.
Configure the subnet and the virtual network associated with the primary interface of the Azure VMs with an adequate address prefix/range. The number of required IP addresses depends on the workload and the number of nodes in the cluster.
For example, for a cluster of 256 nodes, make sure that the address space of the subnet and the virtual network can allocate at least 128 * 256 IP addresses, in order to run a maximum of 128 pods concurrently on a node. This is in addition to initial IP allocations to VM network interface card (NICs) during Azure resource creation.
Accounting for the allocation of IP addresses to NICs that occur during VM
bring-up, set the address space of the subnet and virtual network to
10.0.0.0/16. This ensures that the network can dynamically allocate
at least 32768 addresses, plus a buffer for initial allocations for
primary IP addresses.
Note
The Azure IPAM module queries the metadata of an Azure VM to obtain a list
of the IP addresses that are assigned to the VM NICs. The IPAM module
allocates these IP addresses to Kubernetes pods. You configure the IP
addresses as ipConfigurations in the NICs associated with a VM or
scale set member, so that Azure IPAM can provide the addresses to Kubernetes
on request.
Manually provision IP address pools as part of an Azure VM scale set¶
Configure IP Pools for each member of the VM scale set during
provisioning by associating multiple ipConfigurations with the scale
set’s networkInterfaceConfigurations.
The following example networkProfile configuration for an ARM template
configures pools of 32 IP addresses for each VM in the VM scale set.
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "[variables('nicName')]",
"properties": {
"ipConfigurations": [
{
"name": "[variables('ipConfigName1')]",
"properties": {
"primary": "true",
"subnet": {
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
},
"loadBalancerBackendAddressPools": [
{
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/backendAddressPools/', variables('bePoolName'))]"
}
],
"loadBalancerInboundNatPools": [
{
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/inboundNatPools/', variables('natPoolName'))]"
}
]
}
},
{
"name": "[variables('ipConfigName2')]",
"properties": {
"subnet": {
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
}
}
}
.
.
.
{
"name": "[variables('ipConfigName32')]",
"properties": {
"subnet": {
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
}
}
}
],
"primary": "true"
}
}
]
}
Adjust the IP count value¶
During an MKE installation, you can alter the number of Azure IP addresses that MKE automatically provisions for pods.
By default, MKE will provision 128 addresses, from the same Azure subnet as the
hosts, for each VM in the cluster. If, however, you have manually attached
additional IP addresses to the VMs (by way of an ARM Template, Azure CLI or
Azure Portal) or you are deploying in to small Azure subnet (less than /16),
you can use an --azure-ip-count flag at install time.
Note
Do not set the --azure-ip-count variable to a value of less than 6 if
you have not manually provisioned additional IP addresses for each VM. The
MKE installation needs at least 6 IP addresses to allocate to the core MKE
components that run as Kubernetes pods (in addition to the VM’s private IP
address).
Below are several example scenarios that require the defining of the
--azure-ip-count variable.
Scenario 1: Manually provisioned addresses
If you have manually provisioned additional IP addresses for each VM and want
to disable MKE from dynamically provisioning more IP addresses, you must
pass --azure-ip-count 0 into the MKE installation command.
Scenario 2: Reducing the number of provisioned addresses
Pass --azure-ip-count <custom_value> into the MKE installation command
to reduce the number of IP addresses dynamically allocated from 128 to a
custom value due to:
Primary use of the Swarm Orchestrator
Deployment of MKE on a small Azure subnet (for example, /24)
Plans to run a small number of Kubernetes pods on each node
To adjust this value post-installation, refer to the instructions on how to download the MKE configuration file, change the value, and update the configuration via the API.
Note
If you reduce the value post-installation, existing VMs will not reconcile and you will need to manually edit the IP count in Azure.
Run the following command to install MKE on a manager node.
docker container run --rm -it \
--name ucp \
--volume /var/run/docker.sock:/var/run/docker.sock \
mirantis/ucp:3.6.19 install \
--host-address <ucp-ip> \
--pod-cidr <ip-address-range> \
--cloud-provider Azure \
--interactive
The
--pod-cidroption maps to the IP address range that you configured for the Azure subnet.Note
The
pod-cidrrange must match the Azure virtual network’s subnet attached to the hosts. For example, if the Azure virtual network had the range172.0.0.0/16with VMs provisioned on an Azure subnet of172.0.1.0/24, then the Pod CIDR should also be172.0.1.0/24.This requirement applies only when MKE does not use the VXLAN data plane. If MKE uses the VXLAN data plane, the
pod-cidrrange must be different than the node IP subnet.The
--host-addressmaps to the private IP address of the master node.The
--azure-ip-countserves to adjust the amount of IP addresses provisioned to each VM.
Azure custom roles¶
You can create your own Azure custom roles for use with MKE. You can assign these roles to users, groups, and service principals at management group (in preview only), subscription, and resource group scopes.
Deploy an MKE cluster into a single resource group¶
A resource group is a container that holds resources for an Azure solution. These resources are the virtual machines (VMs), networks, and storage accounts that are associated with the swarm.
To create a custom all-in-one role with permissions to deploy an MKE cluster into a single resource group:
Create the role permissions JSON file.
For example:
{ "Name": "Docker Platform All-in-One", "IsCustom": true, "Description": "Can install and manage Docker platform.", "Actions": [ "Microsoft.Authorization/*/read", "Microsoft.Authorization/roleAssignments/write", "Microsoft.Compute/availabilitySets/read", "Microsoft.Compute/availabilitySets/write", "Microsoft.Compute/disks/read", "Microsoft.Compute/disks/write", "Microsoft.Compute/virtualMachines/extensions/read", "Microsoft.Compute/virtualMachines/extensions/write", "Microsoft.Compute/virtualMachines/read", "Microsoft.Compute/virtualMachines/write", "Microsoft.Network/loadBalancers/read", "Microsoft.Network/loadBalancers/write", "Microsoft.Network/loadBalancers/backendAddressPools/join/action", "Microsoft.Network/networkInterfaces/read", "Microsoft.Network/networkInterfaces/write", "Microsoft.Network/networkInterfaces/join/action", "Microsoft.Network/networkSecurityGroups/read", "Microsoft.Network/networkSecurityGroups/write", "Microsoft.Network/networkSecurityGroups/join/action", "Microsoft.Network/networkSecurityGroups/securityRules/read", "Microsoft.Network/networkSecurityGroups/securityRules/write", "Microsoft.Network/publicIPAddresses/read", "Microsoft.Network/publicIPAddresses/write", "Microsoft.Network/publicIPAddresses/join/action", "Microsoft.Network/virtualNetworks/read", "Microsoft.Network/virtualNetworks/write", "Microsoft.Network/virtualNetworks/subnets/read", "Microsoft.Network/virtualNetworks/subnets/write", "Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.Resources/subscriptions/resourcegroups/read", "Microsoft.Resources/subscriptions/resourcegroups/write", "Microsoft.Security/advancedThreatProtectionSettings/read", "Microsoft.Security/advancedThreatProtectionSettings/write", "Microsoft.Storage/*/read", "Microsoft.Storage/storageAccounts/listKeys/action", "Microsoft.Storage/storageAccounts/write" ], "NotActions": [], "AssignableScopes": [ "/subscriptions/6096d756-3192-4c1f-ac62-35f1c823085d" ] }
Create the Azure RBAC role.
az role definition create --role-definition all-in-one-role.json
Deploy MKE compute resources¶
Compute resources act as servers for running containers.
To create a custom role to deploy MKE compute resources only:
Create the role permissions JSON file.
For example:
{ "Name": "Docker Platform", "IsCustom": true, "Description": "Can install and run Docker platform.", "Actions": [ "Microsoft.Authorization/*/read", "Microsoft.Authorization/roleAssignments/write", "Microsoft.Compute/availabilitySets/read", "Microsoft.Compute/availabilitySets/write", "Microsoft.Compute/disks/read", "Microsoft.Compute/disks/write", "Microsoft.Compute/virtualMachines/extensions/read", "Microsoft.Compute/virtualMachines/extensions/write", "Microsoft.Compute/virtualMachines/read", "Microsoft.Compute/virtualMachines/write", "Microsoft.Network/loadBalancers/read", "Microsoft.Network/loadBalancers/write", "Microsoft.Network/networkInterfaces/read", "Microsoft.Network/networkInterfaces/write", "Microsoft.Network/networkInterfaces/join/action", "Microsoft.Network/publicIPAddresses/read", "Microsoft.Network/virtualNetworks/read", "Microsoft.Network/virtualNetworks/subnets/read", "Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.Resources/subscriptions/resourcegroups/read", "Microsoft.Resources/subscriptions/resourcegroups/write", "Microsoft.Security/advancedThreatProtectionSettings/read", "Microsoft.Security/advancedThreatProtectionSettings/write", "Microsoft.Storage/storageAccounts/read", "Microsoft.Storage/storageAccounts/listKeys/action", "Microsoft.Storage/storageAccounts/write" ], "NotActions": [], "AssignableScopes": [ "/subscriptions/6096d756-3192-4c1f-ac62-35f1c823085d" ] }
Create the Docker Platform RBAC role.
az role definition create --role-definition platform-role.json
Deploy MKE network resources¶
Network resources are services inside your cluster. These resources can include virtual networks, security groups, address pools, and gateways.
To create a custom role to deploy MKE network resources only:
Create the role permissions JSON file.
For example:
{ "Name": "Docker Networking", "IsCustom": true, "Description": "Can install and manage Docker platform networking.", "Actions": [ "Microsoft.Authorization/*/read", "Microsoft.Network/loadBalancers/read", "Microsoft.Network/loadBalancers/write", "Microsoft.Network/loadBalancers/backendAddressPools/join/action", "Microsoft.Network/networkInterfaces/read", "Microsoft.Network/networkInterfaces/write", "Microsoft.Network/networkInterfaces/join/action", "Microsoft.Network/networkSecurityGroups/read", "Microsoft.Network/networkSecurityGroups/write", "Microsoft.Network/networkSecurityGroups/join/action", "Microsoft.Network/networkSecurityGroups/securityRules/read", "Microsoft.Network/networkSecurityGroups/securityRules/write", "Microsoft.Network/publicIPAddresses/read", "Microsoft.Network/publicIPAddresses/write", "Microsoft.Network/publicIPAddresses/join/action", "Microsoft.Network/virtualNetworks/read", "Microsoft.Network/virtualNetworks/write", "Microsoft.Network/virtualNetworks/subnets/read", "Microsoft.Network/virtualNetworks/subnets/write", "Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.Resources/subscriptions/resourcegroups/read", "Microsoft.Resources/subscriptions/resourcegroups/write" ], "NotActions": [], "AssignableScopes": [ "/subscriptions/6096d756-3192-4c1f-ac62-35f1c823085d" ] }
Create the Docker Networking RBAC role.
az role definition create --role-definition networking-role.json
See also
See also
Install MKE on Google Cloud Platform¶
MKE includes support for installing and running MKE on Google Cloud Platform (GCP). You will learn in this section how to prepare your system for MKE installation on GCP, how to perform the installation, and some limitations with the support for GCP on MKE.
To learn how to deploy MKE on GCP using Launchpad, see Bootstrapping MKE cluster on GCP.
Prerequisites¶
Verify the following prerequisites before you install MKE on GCP:
MTU (maximum transmission unit) is set to at least
1500on the VPC where you want to create your instances. For more information, refer to Google Cloud official documentation: Change the MTU setting of a VPC network.All MKE instances have the necessary authorization for managing cloud resources.
GCP defines authorization through the use of service accounts, roles, and access scopes. For information on how to best configure the authorization required for your MKE instances, refer to Google Cloud official documentation: Service accounts.
An example of a permissible role for a service account is
roles/owner, and an example of an access scope that provides access to most Google services ishttps://www.googleapis.com/auth/cloud-platform. As a best practice, define a broad access scope such as this to an instance and then restrict access using roles.Refer to Google Identity official documentation: OAuth 2.0 Scopes for Google APIs for a list of available scopes, and to Google Cloud official documentation: Understanding roles for a list of available roles.
All of your MKE instances include the same prefix.
Each instance is tagged with the prefix of its associated instance names. For example, if the instance names are
testcluster-m1andtestcluster-m2, tag the associated instance withtestcluster.
Install MKE¶
To install MKE on GCP, run the following command:
docker container run --rm -it \
--name ucp \
--volume /var/run/docker.sock:/var/run/docker.sock \
mirantis/ucp:3.6.19 install \
--host-address <ucp-ip> \
--cloud-provider gce \
--interactive
Note
Do not use the --cloud-provider gce flag if you do not require cloud
provider integration.
Google Cloud Platform support limitations¶
Be aware of the following limitations in the MKE support for GCP:
[MKE-8914] Windows Server Core with Containers images incompatible with GCP
[MKE-8814] Mismatched MTU values cause Swarm overlay network issues on GCP
[MKE-8914] Windows Server Core with Containers images incompatible with GCP¶
The use of Windows ServerCore with Containers images will prevent kubelet
from starting up, as these images are not compatible with GCP.
As a workaround, use Windows Server or Windows Server Core images.
[MKE-8814] Mismatched MTU values cause Swarm overlay network issues on GCP¶
Communication between GCP VPCs and Docker networks that use Swarm overlay networks will fail if their MTU values are not manually aligned. By default, the MTU value for GCP VPCs is 1460, while the default MTU value for Docker networks is 1500.
Workaround:
Select from the following options:
Create a new VPC and set the MTU value to 1500.
Set the MTU value of the existing VPC to 1500.
For more information, refer to the Google Cloud Platform documentation, Change the MTU setting of a VPC network.
Install MKE offline¶
To install MKE on an offline host, you must first use a separate computer with an Internet connection to download a single package with all the images and then copy that package to the host where you will install MKE. Once the package is on the host and loaded, you can install MKE offline as described in Install the MKE image.
Note
During the offline installation, both manager and worker nodes must be offline.
To install MKE offline:
Download the required MKE package.
Using one of the following links:
Using the following command with the package URL:
wget <mke-package-url> -O ucp.tar.gz
Copy the MKE package to the host machine:
scp ucp.tar.gz <user>@<host>
Use SSH to log in to the host where you transferred the package.
Load the MKE images from the
.tar.gzfile:docker load -i ucp.tar.gz
See also
Uninstall MKE¶
This topic describes how to uninstall MKE from your cluster. After uninstalling MKE, your instances of MCR will continue running in swarm mode and your applications will run normally. You will not, however, be able to do the following unless you reinstall MKE:
Enforce role-based access control (RBAC) to the cluster.
Monitor and manage the cluster from a central place.
Join new nodes using
docker swarm join.Note
You cannot join new nodes to your cluster after uninstalling MKE because your cluster will be in swarm mode, and swarm mode relies on MKE to provide the CA certificates that allow nodes to communicate with each other. After the certificates expire, the nodes will not be able to communicate at all. Either reinstall MKE before the certificates expire, or disable swarm mode by running
docker swarm leave --forceon every node.
To uninstall MKE:
Note
If SELinux is enabled, you must temporarily disable it prior to running the uninstall-ucp command.
Log in to a manager node using SSH.
Run the uninstall-ucp command in interactive mode, thus prompting you for the necessary configuration values:
docker container run --rm -it \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/log:/var/log \ --name ucp \ mirantis/ucp:3.6.19 uninstall-ucp --interactive
Note
The uninstall-ucp command completely removes MKE from every node in the cluster. You do not need to run the command from multiple nodes.
If the uninstall-ucp command fails, manually uninstall MKE.
On any manager node, remove the remaining MKE services:
docker service rm $(docker service ls -f name=ucp- -q)
On each manager node, remove the remaining MKE containers:
docker container rm -f $(docker container ps -a -f name=ucp- -f name=k8s_ -q)
On each manager node, remove the remaining MKE volumes:
docker volume rm $(docker volume ls -f name=ucp -q)
Note
For more information about the uninstall-ucp failure, refer to the logs in
/var/logon any manager node. Be aware that you will not be able to access the logs if the volume/var/log:/var/logis not mounted while running theucpcontainer.Optional. Delete the MKE configuration:
docker container run --rm -it \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/log:/var/log \ --name ucp \ mirantis/ucp:3.6.19 uninstall-ucp \ --purge-config --interactive
MKE keeps the configuration by default in case you want to reinstall MKE later with the same configuration. For all available uninstall-ucp options, refer to mirantis/ucp uninstall-ucp.
Optional. Restore the host IP tables to their pre-MKE installation values by restarting the node.
Note
The Calico network plugin changed the host IP tables from their original values during MKE installation.
See also
Deploy Swarm-only mode¶
Swarm-only mode is an MKE configuration that supports only Swarm orchestration. Lacking Kubernetes and its operational and health-check dependencies, the resulting highly-stable application is smaller than a typical mixed-orchestration MKE installation.
You can only enable or disable Swarm-only mode at the time of MKE installation. MKE preserves the Swarm-only setting through upgrades, backups, and system restoration. Installing MKE in Swarm-only mode pulls only the images required to run MKE in this configuration. Refer to Swarm-only images for more information.
Note
Installing MKE in Swarm-only mode removes all Kubernetes options from the MKE web UI.
To install MKE in Swarm-only mode:
Complete the steps and recommendations in Plan the deployment and Perform pre-deployment configuration.
Add the
--swarm-onlyflag to the install command in Install the MKE image:docker container run --rm -it --name ucp \ -v /var/run/docker.sock:/var/run/docker.sock \ mirantis/ucp:3.6.19 install \ --host-address <node-ip-address> \ --interactive \ --swarm-only
Note
In addition, MKE includes the --swarm-only flag with the bootstrapper
images command, which you can use to pull or to check the
required images on manager nodes.
Caution
To restore Swarm-only clusters, invoke the ucp restore command
with the --swarm-only option.
Swarm-only images¶
Installing MKE in Swarm-only mode pulls the following set of images, which is smaller than that of a typical MKE installation:
ucp-agent(ucp-agent-winon Windows)ucp-auth-storeucp-authucp-azure-ip-allocatorucp-cfsslucp-composeucp-containerd-shim-process(ucp-containerd-shim-process-winon Windows)ucp-controllerucp-dsinfo(ucp-dsinfo-winon Windows)ucp-etcducp-interlock-configucp-interlock-extensionucp-interlock-proxyucp-interlockucp-metricsucp-sf-notifierucp-swarm
Prometheus¶
In Swarm-only mode, MKE runs the Prometheus server and the authenticating proxy
in a single container on each manager node. Thus, unlike in conventional MKE
installations, you cannot configure Prometheus server placement. Prometheus
does not collect Kubernetes metrics in Swarm-only mode, and it requires an
additional reserved port on manager nodes: 12387.
See also
See also
Operations Guide¶
The MKE Operations Guide provides the comprehensive information you need to run the MKE container orchestration platform. The guide is intended for anyone who needs to effectively develop and securely administer applications at scale, on private clouds, public clouds, and on bare metal.
Access an MKE cluster¶
You can access an MKE cluster in a variety of ways including through the MKE web UI, Docker CLI, and kubectl (the Kubernetes CLI). To use the Docker CLI and kubectl with MKE, first download a client certificate bundle. This topic describes the MKE web UI, how to download and configure the client bundle, and how to configure kubectl with MKE.
Access the MKE web UI¶
MKE allows you to control your cluster visually using the web UI. Role-based access control (RBAC) gives administrators and non-administrators access to the following web UI features:
Administrators:
Manage cluster configurations.
View and edit all cluster images, networks, volumes, and containers.
Manage the permissions of users, teams, and organizations.
Grant node-specific task scheduling permissions to users.
Non-administrators:
View and edit all cluster images, networks, volumes, and containers. Requires administrator to grant access.
To access the MKE web UI:
Open a browser and navigate to
https://<ip-address>(substituting<ip-address>with the IP address of the machine that ran docker run).Enter the user name and password that you set up when installing the MKE image.
Note
To set up two-factor authentication for logging in to the MKE web UI, see Use two-factor authentication.
Download and configure the client bundle¶
Download and configure the MKE client certificate bundle to use MKE with Docker CLI and kubectl. The bundle includes:
A private and public key pair for authorizing your requests using MKE
Utility scripts for configuring Docker CLI and kubectl with your MKE deployment
Note
MKE issues different certificates for each user type:
- User certificate bundles
Allow running docker commands only through MKE manager nodes.
- Administrator certificate bundles
Allow running docker commands through all node types.
Download the client bundle¶
This section explains how to download the client certificate bundle using either the MKE web UI or the MKE API.
To download the client certificate bundle using the MKE web UI:
Navigate to My Profile.
Click Client Bundles > New Client Bundle.
To download the client certificate bundle using the MKE API on Linux:
Create an environment variable with the user security token:
AUTHTOKEN=$(curl -sk -d \ '{"username":"<username>","password":"<password>"}' \ https://<mke-ip>/auth/login | jq -r .auth_token)
Download the client certificate bundle:
curl -k -H "Authorization: Bearer $AUTHTOKEN" \ https://<mke-ip>/api/clientbundle -o bundle.zip
To download the client certificate bundle using the MKE API on Windows Server 2016:
Open an elevated PowerShell prompt.
Create an environment variable with the user security token:
$AUTHTOKEN=((Invoke-WebRequest -Body '{"username":"<username>", \ "password":"<password>"}' -Uri https://`<mke-ip`>/auth/login \ -Method POST).Content)|ConvertFrom-Json|select auth_token \ -ExpandProperty auth_token
Download the client certificate bundle:
[io.file]::WriteAllBytes("ucp-bundle.zip", \ ((Invoke-WebRequest -Uri https://`<mke-ip`>/api/clientbundle \ -Headers @{"Authorization"="Bearer $AUTHTOKEN"}).Content))
Configure the client bundle¶
This section explains how to configure the client certificate bundle to authenticate your requests with MKE using the Docker CLI and kubectl.
To configure the client certificate bundle:
Extract the client bundle
.zipfile into a directory, and use the appropriate utility script for your system:For Linux:
cd client-bundle && eval "$(<env.sh)"
For Windows (from an elevated PowerShell prompt):
cd client-bundle && env.cmd
The utility scripts do the following:
Update
DOCKER_HOSTto make the client tools communicate with your MKE deployment.Update
DOCKER_CERT_PATHto use the certificates included in the client bundle.Configure kubectl with the kubectl config command.
Note
The
kubeconfigfile is namedkube.yamland is located in the unzipped client bundle directory.
Verify that your client tools communicate with MKE:
docker version --format '{{.Server.Version}}' kubectl config current-context
The expected Docker CLI server version starts with
ucp/, and the expected kubectl context name starts withucp_.Optional. Change your context directly using the client certificate bundle
.zipfiles. In the directory where you downloaded the user bundle, add the new context:cd client-bundle && docker context \ import myucp ucp-docker-bundle.zip
Note
If you use the client certificate bundle with buildkit, make sure that builds are not accidentally scheduled on manager nodes. For more information, refer to Manage services node deployment.
Configure kubectl with MKE¶
MKE installations include Kubernetes. Users can deploy, manage, and monitor Kubernetes using either the MKE web UI or kubectl.
To install and use kubectl:
Identify which version of Kubernetes you are running by using the MKE web UI, the MKE API
versionendpoint, or the Docker CLI docker version command with the client bundle.Caution
Kubernetes requires that kubectl and Kubernetes be within one minor version of each other.
Refer to Kubernetes: Install Tools to download and install the appropriate kubectl binary.
Refer to Configure the client bundle to configure kubectl with MKE using the certificates and keys contained in the client bundle.
Optional. Install Helm, the Kubernetes package manager, and Tiller, the Helm server.
Caution
Helm requires MKE 3.1.x or higher.
To use Helm and Tiller with MKE, grant the default service account within the
kube-systemnamespace the necessary roles:kubectl create rolebinding default-view --clusterrole=view \ --serviceaccount=kube-system:default --namespace=kube-system kubectl create clusterrolebinding add-on-cluster-admin \ --clusterrole=cluster-admin --serviceaccount=kube-system:default
Note
Helm recommends that you specify a Role and RoleBinding to limit the scope of Tiller to a particular namespace. Refer to the official Helm documentation for more information.
See also
Administer an MKE cluster¶
Add labels to cluster nodes¶
With MKE, you can add labels to your nodes. Labels are metadata that describe the node, such as:
node role (development, QA, production)
node region (US, EU, APAC)
disk type (HDD, SSD)
Once you apply a label to a node, you can specify constraints when deploying a service to ensure that the service only runs on nodes that meet particular criteria.
Hint
Use resource sets (MKE collections or Kubernetes namespaces) to organize access to your cluster, rather than creating labels for authorization and permissions to resources.
Apply labels to a node¶
The following example procedure applies the ssd label to a node.
Log in to the MKE web UI with administrator credentials.
Click Shared Resources in the navigation menu to expand the selections.
Click Nodes. The details pane will display the full list of nodes.
Click the node on the list that you want to attach labels to. The details pane will transition, presenting the Overview information for the selected node.
Click the settings icon in the upper-right corner to open the Edit Node page.
Navigate to the Labels section and click Add Label.
Add a label, entering
diskinto the Key field andssdinto the Value field.Click Save to dismiss the Edit Node page and return to the node Overview.
Hint
You can use the CLI to apply a label to a node:
docker node update --label-add <key>=<value> <node-id>
Deploy a service with constraints¶
The following example procedure deploys a service with a constraint that
ensures that the service only runs on nodes with SSD storage
node.labels.disk == ssd.
To deploy an application stack with service constraints:
Log in to the MKE web UI with administrator credentials.
Verify that the target node orchestrator is set to Swarm.
Click Shared Resources in the left-side navigation panel to expand the selections.
Click Stacks. The details pane will display the full list of stacks.
Click the Create Stack button to open the Create Application page.
Under 1. Configure Application, enter “wordpress” into the Name field .
Under ORCHESTRATOR NODE, select Swarm Services.
Under 2. Add Application File, paste the following stack file in the docker-compose.yml editor:
version: "3.1" services: db: image: mysql:5.7 deploy: placement: constraints: - node.labels.disk == ssd restart_policy: condition: on-failure networks: - wordpress-net environment: MYSQL_ROOT_PASSWORD: wordpress MYSQL_DATABASE: wordpress MYSQL_USER: wordpress MYSQL_PASSWORD: wordpress wordpress: depends_on: - db image: wordpress:latest deploy: replicas: 1 placement: constraints: - node.labels.disk == ssd restart_policy: condition: on-failure max_attempts: 3 networks: - wordpress-net ports: - "8000:80" environment: WORDPRESS_DB_HOST: db:3306 WORDPRESS_DB_PASSWORD: wordpress networks: wordpress-net:
Click Create to deploy the stack.
Click Done once the stack deployment completes to return to the stacks list which now features your newly created stack.
To verify service tasks deployed to labeled node:
In the left-side navigation panel, navigate to Shared Resources > Nodes. The details pane will display the full list of nodes.
Click the node with the
disklabel.In the details pane, click the Metrics tab to verify that WordPress containers are scheduled on the node.
In the left-side navigation panel, navigate to Shared Resources > Nodes.
Click any node that does not have the
disklabel.In the details pane, click the Metrics tab to verify that there are no WordPress containers scheduled on the node.
Add Swarm placement constraints¶
If a node is set to use Kubernetes as its orchestrator while simultaneously running Swarm services, you must deploy placement constraints to prevent those services from being scheduled on the node.
The necessary service constraints will be automatically adopted by any new MKE-created Swarm services, as well as by older Swarm services that you have updated. MKE does not automatically add placement constraints, however, to Swarm services that were created using older versions of MKE, as to do so would restart the service tasks.
To add placement constraints to older Swarm services:
Identify the Swarm services that do not have placement constraints:
services=$(docker service ls -q) for service in $services; do if docker service inspect $service --format '{{.Spec.TaskTemplate.Placement.Constraints}}' | grep -q -v 'node.labels.com.docker.ucp.orchestrator.swarm==true'; then name=$(docker service inspect $service --format '{{.Spec.Name}}') if [ $name = "ucp-agent" ] || [ $name = "ucp-agent-win" ] || [ $name = "ucp-agent-s390x" ]; then continue fi echo "Service $name (ID: $service) is missing the node.labels.com.docker.ucp.orchestrator.swarm=true placement constraint" fi done
Add placement constraints to the Swarm services you identified:
Note
All service tasks will restart, thus causing some amount of service downtime.
services=$(docker service ls -q) for service in $services; do if docker service inspect $service --format '{{.Spec.TaskTemplate.Placement.Constraints}}' | grep -q -v 'node.labels.com.docker.ucp.orchestrator.swarm=true'; then name=$(docker service inspect $service --format '{{.Spec.Name}}') if [ $name = "ucp-agent" ] || [ $name = "ucp-agent-win" ]; then continue fi echo "Updating service $name (ID: $service)" docker service update --detach=true --constraint-add node.labels.com.docker.ucp.orchestrator.swarm==true $service fi done
Add or remove a service constraint using the MKE web UI¶
You can declare the deployment constraints in your docker-compose.yml
file or when you create a stack. Also, you can apply constraints when
you create a service.
To add or remove a service constraint:
Verify whether a service has deployment constraints:
Navigate to the Services page and select that service.
In the details pane, click Constraints to list the constraint labels.
Edit the constraints on the service:
Click Configure and select Details to open the Update Service page.
Click Scheduling to view the constraints.
Add or remove deployment constraints.
Add SANs to cluster certificates¶
A SAN (Subject Alternative Name) is a structured means for associating various values (such as domain names, IP addresses, email addresses, URIs, and so on) with a security certificate.
MKE always runs with HTTPS enabled. As such, whenever you connect to MKE, you must ensure that the MKE certificates recognize the host name in use. For example, if MKE is behind a load balancer that forwards traffic to your MKE instance, your requests will not be for the MKE host name or IP address but for the host name of the load balancer. Thus, MKE will reject the requests, unless you include the address of the load balancer as a SAN in the MKE certificates.
Note
To use your own TLS certificates, confirm first that these certificates have the correct SAN values.
To use the self-signed certificate that MKE offers out-of-the-box, you can use the
--sanargument to set up the SANs during MKE deployment.
To add new SANs using the MKE web UI:
Log in to the MKE web UI using administrator credentials.
Navigate to the Nodes page.
Click on a manager node to display the details pane for that node.
Click Configure and select Details.
In the SANs section, click Add SAN and enter one or more SANs for the cluster.
Click Save.
Repeat for every existing manager node in the cluster.
Note
Thereafter, the SANs are automatically applied to any new manager nodes that join the cluster.
To add new SANs using the MKE CLI:
Get the current set of SANs for the given manager node:
docker node inspect --format '{{ index .Spec.Labels "com.docker.ucp.SANs" }}' <node-id>
Example of system response:
default-cs,127.0.0.1,172.17.0.1
Append the desired SAN to the list (for example,
default-cs,127.0.0.1,172.17.0.1,example.com) and run:docker node update --label-add com.docker.ucp.SANs=<SANs-list> <node-id>
Note
<SANs-list>is the comma-separated list of SANs with your new SAN appended at the end.Repeat the command sequence for each manager node.
Collect MKE cluster metrics with Prometheus¶
Prometheus is an open-source systems monitoring and alerting toolkit to which you can configure MKE as a target.
Prometheus runs as a Kubernetes deployment that, by default, is a DaemonSet that runs on every manager node. A key benefit of this is that you can set the DaemonSet to not schedule on any nodes, which effectively disables Prometheus if you do not use the MKE web interface.
Along with events and logs, metrics are data sources that provide a view into your cluster, presenting numerical data values that have a time-series component. There are several sources from which you can derive metrics, each providing different meanings for a business and its applications.
As the metrics data is stored locally on disk for each Prometheus server, it does not replicate on new managers or if you schedule Prometheus to run on a new node. The metrics are kept no longer than 24 hours.
MKE metrics types¶
MKE provides a base set of metrics that gets you into production without having to rely on external or third-party tools. Mirantis strongly encourages, though, the use of additional monitoring to provide more comprehensive visibility into your specific MKE environment.
Metric type |
Description |
|---|---|
Business |
High-level aggregate metrics that typically combine technical, financial, and organizational data to create IT infrastructure information for business leaders. Examples of business metrics include:
|
Application |
Metrics on APM tools domains (such as AppDynamics and DynaTrace) that supply information on the state or performance of the application itself.
|
Service |
Metrics on the state of services that are running on the container platform. Such metrics have very low cardinality, meaning the values are typically from a small fixed set of possibilities (commonly binary).
Note Web UI disk usage (including free space) reflects only the MKE
managed portion of the file system: |
See also
Metrics labels¶
The metrics that MKE exposes in Prometheus have standardized labels, depending on the target resource.
Label name |
Value |
|---|---|
|
The collection ID of the collection the container is in, if any. |
|
The ID of the container. |
|
The name of the container image. |
|
Set to |
|
The container name. |
|
The pod name, if the container is part of a Kubernetes Pod. |
|
The pod namespace, if the container is part of a Kubernetes Pod namespace. |
|
The container name in the pod spec, if the container is part of a Kubernetes pod. |
|
The service ID, if the container is part of a Swarm service. |
|
The stack name, if the container is part of a Docker Compose stack. |
Label name |
Value |
|---|---|
|
The collection ID of the collection the container is in, if any. |
|
The ID of the container. |
|
The name of the container image. |
|
Set to |
|
The container name. |
|
The ID of the network. |
|
The pod name, if the container is part of a Kubernetes pod. |
|
The pod namespace, if the container is part of a Kubernetes pod namespace. |
|
The container name in the pod spec, if the container is part of a Kubernetes pod. |
|
The service ID, if the container is part of a Swarm service. |
|
The stack name, if the container is part of a Docker Compose stack. |
Note
The container networking labels are the same as the Container labels, with
the addition of network.
Label name |
Value |
|---|---|
|
Set to |
See also
Core MKE metrics¶
MKE exports metrics on every node and also exports additional metrics from every controller.
Node-sourced MKE metrics¶
The metrics that MKE exports from nodes are specific to those nodes (for example, the total memory on that node).
The tables below offer detail on the node-sourced metrics that MKE
exposes in Prometheus with the ucp_ label.
Units |
Percentage |
|---|---|
Description |
Percentage of CPU time in use by the container |
Labels |
Container |
Units |
Nanoseconds |
|---|---|
Description |
Total CPU time used by the container |
Labels |
Container |
Units |
Bytes |
|---|---|
Description |
Total container disk size |
Labels |
Container |
Units |
0.0 or 1.0 |
|---|---|
Description |
The container health, according to its healthcheck. The |
Labels |
Container |
Units |
Bytes |
|---|---|
Description |
Maximum memory in use by the container in bytes |
Labels |
Container |
Units |
Bytes |
|---|---|
Description |
Current memory in use by the container in bytes |
Labels |
Container |
Units |
Percentage |
|---|---|
Description |
Percentage of total node memory currently in use by the container |
Labels |
Container |
Units |
Bytes |
|---|---|
Description |
Number of bytes received by the container over the network in the last sample |
Labels |
Container networking |
Units |
Number of packets |
|---|---|
Description |
Number of packets bound for the container over the network that were dropped in the last sample |
Labels |
Container networking |
Units |
Number of errors |
|---|---|
Description |
Number of received network errors for the container over the network in the last sample |
Labels |
Container networking |
Units |
Number of packets |
|---|---|
Description |
Number of packets received by the container over the network in the last sample |
Labels |
Container networking |
Units |
Bytes |
|---|---|
Description |
Number of bytes sent by the container over the network in the last sample |
Labels |
Container networking |
Units |
Number of packets |
|---|---|
Description |
Number of packets sent from the container over the network that were dropped in the last sample |
Labels |
Container networking |
Units |
Number of errors |
|---|---|
Description |
Number of sent network errors for the container on the network in the last sample |
Labels |
Container networking |
Units |
Number of packets |
|---|---|
Description |
Number of sent packets for the container over the network in the last sample |
Labels |
Container networking |
Units |
0.0 or 1.0 |
|---|---|
Description |
Indicates whether the container is healthy, according to its healthcheck. The |
Labels |
Container |
Units |
Number of containers |
|---|---|
Description |
Total number of containers on the node |
Labels |
Node |
Units |
Nanoseconds |
|---|---|
Description |
System CPU time used by the container |
Labels |
Container |
Units |
Bytes |
|---|---|
Description |
Free disk space on the Docker root directory on the node, in bytes. This metric is not available to Windows nodes |
Labels |
Node |
Units |
Bytes |
|---|---|
Description |
Total disk space on the Docker root directory on this node in bytes.
Note that the |
Labels |
Node |
Units |
Number of images |
|---|---|
Description |
Total number of images on the node |
Labels |
Node |
Units |
Bytes |
|---|---|
Description |
Total amount of memory on the node |
Labels |
Node |
Units |
Number of networks |
|---|---|
Description |
Total number of networks on the node |
Labels |
Node |
Units |
Number of cores |
|---|---|
Description |
Number of CPU cores on the node |
Labels |
Node |
Units |
Number of volumes |
|---|---|
Description |
Total number of volumes on the node |
Labels |
Node |
Controller-sourced MKE metrics¶
The metrics that MKE exports from controllers are cluster-scoped (for example, the total number of Swarm services).
The tables below offer detail on the controller-sourced metrics that MKE
exposes in Prometheus with the ucp_ label.
Units |
Number of services |
|---|---|
Description |
Total number of Swarm services |
Labels |
Not applicable |
Units |
0.0 or 1.0 |
|---|---|
Description |
Health status of the node, as determined by MKE |
Labels |
nodeName: node name, nodeAddr: node IP address |
Units |
0.0 or 1.0 |
|---|---|
Description |
Readiness of the container in a Kubernetes pod, as determined by its readiness probe |
Labels |
Pod |
Units |
0.0 or 1.0 |
|---|---|
Description |
Readiness of the container in a Kubernetes pod, as determined by its readiness probe |
Labels |
Pod |
See also
MKE cAdvsior metrics¶
Once you have enabled cAdvisor and generated an auth token, you can issue the following command to access the cAdvisor metrics:
curl -sk -H
"Authorization: Bearer $AUTHTOKEN"
"$<mke_url>/metricsservice/query?query=$<mke_specific_metric>\[$<time_duration>\]"
The Prometheus container metrics exposed by cAdvisor are presented below:
cadvisor_version_info¶
Units |
N/A |
|---|---|
Description |
A metric with a constant |
Labels |
|
container_blkio_device_usage_total¶
Units |
bytes |
|---|---|
Description |
The Block I/O (blkio) device bytes usage. |
Labels |
|
container_cpu_system_seconds_total¶
Value |
seconds |
|---|---|
Description |
Cumulative system CPU time consumed. |
Labels |
|
container_cpu_usage_seconds_total¶
Value |
seconds |
|---|---|
Description |
Cumulative CPU time consumed. |
Labels |
|
container_cpu_user_seconds_total¶
Value |
seconds |
|---|---|
Description |
Cumulative user CPU time consumed. |
Labels |
|
container_fs_reads_bytes_total¶
Units |
bytes |
|---|---|
Description |
Cumulative count of bytes read. |
Labels |
|
container_fs_reads_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of reads completed. |
Labels |
|
container_fs_writes_bytes_total¶
Units |
bytes |
|---|---|
Description |
Cumulative count of bytes written. |
Labels |
|
container_fs_writes_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of writes completed. |
Labels |
|
container_last_seen¶
Units |
timestamp |
|---|---|
Description |
Last time a container was seen by the exporter. |
Labels |
|
container_memory_cache¶
Units |
bytes |
|---|---|
Description |
Total page cache memory. |
Labels |
|
container_memory_failcnt¶
Value |
integer |
|---|---|
Description |
Number of memory usage hits limits. |
Labels |
|
container_memory_failures_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of memory allocation failures. |
Labels |
|
container_memory_mapped_file¶
Units |
bytes |
|---|---|
Description |
Size of memory mapped files. |
Labels |
|
container_memory_max_usage_bytes¶
Units |
bytes |
|---|---|
Description |
Maximum memory usage recorded. |
Labels |
|
container_memory_rss¶
Units |
bytes |
|---|---|
Description |
Size of RSS. |
Labels |
|
container_memory_swap¶
Units |
bytes |
|---|---|
Description |
Container swap usage. |
Labels |
|
container_memory_usage_bytes¶
Units |
bytes |
|---|---|
Description |
Current memory usage, including all memory regardless of when it was accessed. |
Labels |
|
container_memory_working_set_bytes¶
Units |
bytes |
|---|---|
Description |
Current working set. |
Labels |
|
container_network_receive_bytes_total¶
Units |
bytes |
|---|---|
Description |
Cumulative count of bytes received. |
Labels |
|
container_network_receive_errors_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of errors encountered while receiving. |
Labels |
|
container_network_receive_packets_dropped_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of packets dropped while receiving. |
Labels |
|
container_network_receive_packets_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of packets received. |
Labels |
|
container_network_transmit_bytes_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of bytes transmitted. |
Labels |
|
container_network_transmit_errors_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of errors encountered while transmitting. |
Labels |
|
container_network_transmit_packets_dropped_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of packets dropped while transmitting. |
Labels |
|
container_network_transmit_packets_total¶
Value |
integer |
|---|---|
Description |
Cumulative count of packets transmitted. |
Labels |
|
container_scrape_error¶
Units |
N/A |
|---|---|
Description |
|
Labels |
|
container_spec_cpu_period¶
Units |
N/A |
|---|---|
Description |
CPU period of the container. |
Labels |
|
container_spec_memory_limit_bytes¶
Units |
bytes |
|---|---|
Description |
Memory limit for the container. |
Labels |
|
container_spec_memory_reservation_limit_bytes¶
Units |
bytes |
|---|---|
Description |
Memory reservation limit for the container. |
Labels |
|
container_spec_memory_swap_limit_bytes¶
Units |
bytes |
|---|---|
Description |
Memory swap limit for the container. |
Labels |
|
container_start_time_seconds¶
Value |
seconds |
|---|---|
Description |
Start time of the container since unix epoch. |
Labels |
|
machine_cpu_cores¶
Value |
integer |
|---|---|
Description |
Number of logical CPU cores. |
Labels |
|
machine_cpu_physical_cores¶
Value |
integer |
|---|---|
Description |
Number of physical CPU cores. |
Labels |
|
machine_cpu_sockets¶
Value |
integer |
|---|---|
Description |
Number of CPU sockets. |
Labels |
|
machine_memory_bytes¶
Units |
bytes |
|---|---|
Description |
Amount of memory installed on the machine. |
Labels |
|
machine_nvm_avg_power_budget_watts¶
Units |
watts |
|---|---|
Description |
NVM power budget. |
Labels |
|
machine_nvm_capacity¶
Units |
bytes |
|---|---|
Description |
NVM capacity value, labeled by NVM mode (memory mode or app direct mode). |
Labels |
|
machine_scrape_error¶
Value |
integer |
|---|---|
Description |
|
Labels |
|
Deploy Prometheus on worker nodes¶
MKE deploys Prometheus by default on the manager nodes to provide a built-in metrics backend. For cluster sizes over 100 nodes, or if you need to scrape metrics from Prometheus instances, Mirantis recommends that you deploy Prometheus on dedicated worker nodes in the cluster.
To deploy Prometheus on worker nodes:
Source an admin bundle.
Verify that
ucp-metricspods are running on all managers:$ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide NAME READY STATUS RESTARTS AGE IP NODE ucp-metrics-hvkr7 3/3 Running 0 4h 192.168.80.66 3a724a-0
Add a Kubernetes node label to one or more workers. For example, a label with key
ucp-metricsand value""to a node with name3a724a-1.$ kubectl label node 3a724a-1 ucp-metrics= node "test-3a724a-1" labeled
SELinux Prometheus Deployment
If you use SELinux, label your
ucp-node-certsdirectories properly on the worker nodes before you move theucp-metricsworkload to them. To runucp-metricson a worker node, update theucp-node-certslabel by running:sudo chcon -R system_u:object_r:container_file_t:s0 /var/lib/docker/volumes/ucp-node-certs/_data.Patch the
ucp-metricsDaemonSet’s nodeSelector with the same key and value in use for the node label. This example shows the keyucp-metricsand the value"".$ kubectl -n kube-system patch daemonset ucp-metrics --type json -p '[{"op": "replace", "path": "/spec/template/spec/nodeSelector", "value": {"ucp-metrics": ""}}]' daemonset "ucp-metrics" patched
Confirm that
ucp-metricspods are running only on the labeled workers.$ kubectl -n kube-system get pods -l k8s-app=ucp-metrics -o wide NAME READY STATUS RESTARTS AGE IP NODE ucp-metrics-88lzx 3/3 Running 0 12s 192.168.83.1 3a724a-1 ucp-metrics-hvkr7 3/3 Terminating 0 4h 192.168.80.66 3a724a-0
See also
Configure external Prometheus to scrape metrics from MKE¶
To configure your external Prometheus server to scrape metrics from Prometheus in MKE:
Source an admin bundle.
Create a Kubernetes secret that contains your bundle TLS material.
(cd $DOCKER_CERT_PATH && kubectl create secret generic prometheus --from-file=ca.pem --from-file=cert.pem --from-file=key.pem)
Create a Prometheus deployment and
ClusterIPservice using YAML.Important
To run Prometheus external to MKE, change the domain for the inventory container in the Prometheus deployment from
ucp-controller.kube-system.svc.cluster.localto an external domain, to access MKE from the Prometheus node.kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: prometheus data: prometheus.yaml: | global: scrape_interval: 10s scrape_configs: - job_name: 'ucp' tls_config: ca_file: /bundle/ca.pem cert_file: /bundle/cert.pem key_file: /bundle/key.pem server_name: proxy.local scheme: https file_sd_configs: - files: - /inventory/inventory.json --- apiVersion: apps/v1 kind: Deployment metadata: name: prometheus spec: replicas: 2 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: nodeSelector: kubernetes.io/os: linux containers: - name: inventory image: alpine command: ["sh", "-c"] args: - apk add --no-cache curl && while :; do curl -Ss --cacert /bundle/ca.pem --cert /bundle/cert.pem --key /bundle/key.pem --output /inventory/inventory.json https://ucp-controller.kube-system.svc.cluster.local/metricsdiscovery; sleep 15; done volumeMounts: - name: bundle mountPath: /bundle - name: inventory mountPath: /inventory - name: prometheus image: prom/prometheus command: ["/bin/prometheus"] args: - --config.file=/config/prometheus.yaml - --storage.tsdb.path=/prometheus - --web.console.libraries=/etc/prometheus/console_libraries - --web.console.templates=/etc/prometheus/consoles volumeMounts: - name: bundle mountPath: /bundle - name: config mountPath: /config - name: inventory mountPath: /inventory volumes: - name: bundle secret: secretName: prometheus - name: config configMap: name: prometheus - name: inventory emptyDir: medium: Memory --- apiVersion: v1 kind: Service metadata: name: prometheus spec: ports: - port: 9090 targetPort: 9090 selector: app: prometheus sessionAffinity: ClientIP EOF
Determine the service
ClusterIP:$ kubectl get service prometheus NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus ClusterIP 10.96.254.107 <none> 9090/TCP 1h
Forward port 9090 on the local host to the
ClusterIP. The tunnel you create does not need to be kept alive as its only purpose is to expose the Prometheus UI.ssh -L 9090:10.96.254.107:9090 ANY_NODE
Visit
http://127.0.0.1:9090to explore the MKE metrics that Prometheus is collecting.
See also
Set up Grafana with MKE Prometheus¶
Important
The information offered herein on how to set up a Grafana instance connected to MKE Prometheus is derived from the official Deploy Grafana on Kubernetes documentation and modified to work with MKE. As it deploys Grafana with default credentials, Mirantis strongly recommends that you adjust the configuration detail to meet your specific needs prior to deploying Grafana with MKE in a production environment.
Create the
monitoringnamespace on which you will deploy Grafana:kubectl create namespace monitoring
Obtain the UCP cluster ID:
CLUSTER_ID=$(docker info --format '{{json .Swarm.Cluster.ID}}')
Apply the following YAML file to deploy Grafana in the monitoring namespace and to automatically configure MKE Prometheus as a data source:
kubectl apply -f - <<EOF --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: grafana name: grafana namespace: monitoring spec: selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: securityContext: runAsUser: 0 containers: - name: grafana image: grafana/grafana:9.1.0-ubuntu imagePullPolicy: IfNotPresent ports: - containerPort: 3000 name: http-grafana protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /robots.txt port: 3000 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 2 livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3000 timeoutSeconds: 1 resources: requests: cpu: 250m memory: 750Mi volumeMounts: - mountPath: /etc/grafana/ name: grafana-config-volume - mountPath: /etc/ssl name: ucp-node-certs volumes: - name: grafana-config-volume configMap: name: grafana-config items: - key: grafana.ini path: grafana.ini - key: dashboard.json path: dashboard.json - key: datasource.yml path: provisioning/datasources/datasource.yml - name: ucp-node-certs hostPath: path: /var/lib/docker/volumes/ucp-node-certs/_data nodeSelector: node-role.kubernetes.io/master: "" --- apiVersion: v1 kind: Service metadata: name: grafana namespace: monitoring spec: ports: - port: 3000 protocol: TCP targetPort: http-grafana selector: app: grafana sessionAffinity: None type: ClusterIP --- apiVersion: v1 kind: ConfigMap metadata: name: grafana-config namespace: monitoring labels: grafana_datasource: '1' data: grafana.ini: | dashboard.json: | datasource.yml: |- apiVersion: 1 datasources: - name: mke-prometheus type: prometheus access: proxy orgId: 1 url: https://ucp-metrics.kube-system.svc.cluster.local:443 jsonData: tlsAuth: true tlsAuthWithCACert: false serverName: $CLUSTER_ID secureJsonData: tlsClientCert: "\$__file{/etc/ssl/cert.pem}" tlsClientKey: "\$__file{/etc/ssl/key.pem}" --- EOF
Use port forwarding to access the Grafana UI. Be aware that this may require that you install socat on your manager nodes.
kubectl port-forward service/grafana 3000:3000 -n monitoring
You can now navigate to the Grafana UI, which has the MKE Prometheus data
source installed at http://localhost:3000/. Log in initially using
admin for both the user name and password, taking care to change your
credentials after successful log in.
See also
Configure native Kubernetes role-based access control¶
MKE uses native Kubernetes RBAC, which is active by default for Kubernetes clusters. The YAML files of many ecosystem applications and integrations use Kubernetes RBAC to access service accounts. Also, organizations looking to run MKE both on-premises and in hosted cloud services want to run Kubernetes applications in both environments without having to manually change RBAC in their YAML file.
Note
Kubernetes and Swarm roles have separate views. Using the MKE web UI, you can view all the roles for a particular cluster:
Click Access Control in the navigation menu at the left.
Click Roles.
Select the Kubernetes tab or the Swarm tab to view the specific roles for each.
Create a Kubernetes role¶
You create Kubernetes roles either through the CLI using Kubernetes kubectl tool or through the MKE web UI.
To create a Kubernetes role using the MKE web UI:
Log in to the the MKE web UI.
In the navigation menu at the left, click Access Control to display the available options.
Click Roles.
At the top of the details pane, click the Kubernetes tab.
Click Create to open the Create Kubernetes Object page.
Click Namespace to select a namespace for the role from one of the available options.
Provide the YAML file for the role. To do this, either enter it in the Object YAML editor, or upload an existing .yml file using the Click to upload a .yml file selection link at the right.
Click Create to complete role creation.
See also
Create a Kubernetes role grant¶
Kubernetes provides two types of role grants:
ClusterRoleBinding(applies to all namespaces)RoleBinding(applies to a specific namespace)
To create a grant for a Kubernetes role in the MKE web UI:
Log in to the the MKE web UI.
In the navigation menu at the left, click Access Control to display the available options.
Click the Grants option.
At the top of the details paine, click the Kubernetes tab. All existing grants to Kubernetes roles are present in the details pane.
Click Create Role Binding to open the Create Role Binding page.
Select the subject type at the top of the 1. Subject section (Users, Organizations, or Service Account).
Create a role binding for the selected subject type:
Users: Select a type from the User drop-down list.
Organizations: Select a type from the Organization drop-down list. Optionally, you can also select a team using the Team(optional) drop-down list, if any have been established.
Service Account: Select a NAMESPACE from the Namespace drop-down list, then a type from the Service Account drop-down list.
Click Next to activate the 2. Resource Set section.
Select a resource set for the subject.
By default, the default namespace is indicated. To use a different namespace, select the Select Namespace button associated with the desired namespace.
For
ClusterRoleBinding, slide the Apply Role Binding to all namespace (Cluster Role Binding) selector to the right.Click Next to activate the 3. Role section.
Select the role type.
Role
Cluster Role
Note
Cluster Role type is the only role type available if you enabled Apply Role Binding to all namespace (Cluster Role Binding) in the 2. Resource Set section.
Select the role from the from the drop-down list.
Click Create to complete grant creation.
See also
See also
MKE audit logging¶
Audit logs are a chronological record of security-relevant activities by individual users, administrators, or software components that have had an effect on an MKE system. They focus on external user/agent actions and security, rather than attempting to understand state or events of the system itself.
Audit logs capture all HTTP actions (GET, PUT, POST, PATCH, DELETE) to all MKE API, Swarm API, and Kubernetes API endpoints (with the exception of the ignored list) that are invoked and and sent to Mirantis Container Runtime via stdout.
The benefits that audit logs provide include:
- Historical troubleshooting
You can use audit logs to determine a sequence of past events that can help explain why an issue occurred.
- Security analysis and auditing
A full record of all user interactions with the container infrastructure can provide your security team with the visibility necessary to root out questionable or unauthorized access attempts.
- Chargeback
Use audit log about the resources to generate chargeback information.
- Alerting
With a watch on an event stream or a notification the event creates, you can build alerting features on top of event tools that generate alerts for ops teams (PagerDuty, OpsGenie, Slack, or custom solutions).
Logging levels¶
MKE provides three levels of audit logging to administrators:
None |
Audit logging is disabled. |
Metadata |
|
Request |
Includes all fields from the Metadata level, as well as the request payload. |
Once you enable MKE audit logging, the audit logs will collect within the
container logs of the ucp-controller container on each MKE manager node.
Note
Be sure to configure a logging driver with log rotation set, as audit logging can generate a large amount of data.
Enable MKE audit logging¶
Note
The enablement of auditing in MKE does not automatically enable auditing in
Kubernetes objects. To do this, you must set the
kube_api_server_auditing parameter in the MKE configuration file to
true.
Once you have set the kube_api_server_auditing parameter to true,
the following default auditing values are configured on the Kubernetes API
server:
--audit-log-maxage:
30--audit-log-maxbackup:
10--audit-log-maxsize:
10
For information on how to enable and configure the Kubernetes API server
audit values, refer to cluster_config table detail in the MKE
configuration file.
You can enable MKE audit logging using the MKE web user interface, the MKE API, and the MKE configuration file.
Enable MKE audit logging using the web UI¶
Log in to the MKE web user interface.
Click admin to open the navigation menu at the left.
Click Admin Settings.
Click Logs & Audit Logs to open the Logs & Audit Logs details pane.
In the Configure Audit Log Level section, select the relevant logging level.
Click Save.
Enable MKE audit logging using the API¶
Download the MKE client bundle from the command line, as described in Download the client bundle.
Retrieve the JSON file for current audit log configuration:
export DOCKER_CERT_PATH=~/ucp-bundle-dir/ curl --cert ${DOCKER_CERT_PATH}/cert.pem --key ${DOCKER_CERT_PATH}/key.pem --cacert ${DOCKER_CERT_PATH}/ca.pem -k -X GET https://ucp-domain/api/ucp/config/logging > auditlog.json
In
auditlog.json, edit theauditlevelfield tometadataorrequest:{ "logLevel": "INFO", "auditLevel": "metadata", "supportDumpIncludeAuditLogs": false }
Send the JSON request for the audit logging configuration with the same API path, but using the
PUTmethod:curl --cert ${DOCKER_CERT_PATH}/cert.pem --key ${DOCKER_CERT_PATH}/key.pem --cacert ${DOCKER_CERT_PATH}/ca.pem -k -H "Content-Type: application/json" -X PUT --data $(cat auditlog.json) https://ucp-domain/api/ucp/config/logging
Enable MKE audit logging using the configuration file¶
You can enable MKE audit logging using the MKE configuration file before or after MKE installation.
The section of the MKE configuration file that controls MKE auditing
logging is [audit_log_configuration]:
[audit_log_configuration]
level = "metadata"
support_dump_include_audit_logs = false
The level setting supports the following variables:
"""metadata""request"
Caution
The support_dump_include_audit_logs flag specifies whether user
identification information from the ucp-controller container logs is
included in the support bundle. To prevent this information from being sent
with the support bundle, verify that support_dump_include_audit_logs
is set to false. When disabled, the support bundle collection tool
filters out any lines from the ucp-controller container logs that
contain the substring auditID.
Access audit logs using the docker CLI¶
The audit logs are exposed through the ucp-controller logs. You can access
these logs locally through the Docker CLI.
Note
You can also access MKE audit logs using an external container logging solution, such as ELK.
To access audit logs using the Docker CLI:
Source a MKE client bundle.
Run docker logs to obtain audit logs.
The following example tails the command to show the last log entry.
$ docker logs ucp-controller --tail 1 {"audit":{"auditID":"f8ce4684-cb55-4c88-652c-d2ebd2e9365e","kind":"docker-swarm","level":"metadata","metadata":{"creationTimestamp":null},"requestReceivedTimestamp":"2019-01-30T17:21:45.316157Z","requestURI":"/metricsservice/query?query=(%20(sum%20by%20(instance)%20(ucp_engine_container_memory_usage_bytes%7Bmanager%3D%22true%22%7D))%20%2F%20(sum%20by%20(instance)%20(ucp_engine_memory_total_bytes%7Bmanager%3D%22true%22%7D))%20)%20*%20100\u0026time=2019-01-30T17%3A21%3A45.286Z","sourceIPs":["172.31.45.250:48516"],"stage":"RequestReceived","stageTimestamp":null,"timestamp":null,"user":{"extra":{"licenseKey":["FHy6u1SSg_U_Fbo24yYUmtbH-ixRlwrpEQpdO_ntmkoz"],"username":["admin"]},"uid":"4ec3c2fc-312b-4e66-bb4f-b64b8f0ee42a","username":"4ec3c2fc-312b-4e66-bb4f-b64b8f0ee42a"},"verb":"GET"},"level":"info","msg":"audit","time":"2019-01-30T17:21:45Z"}
Sample audit log for a Kubernetes cluster:
{"audit"; { "metadata": {...}, "level": "Metadata", "timestamp": "2018-08-07T22:10:35Z", "auditID": "7559d301-fa6b-4ad6-901c-b587fab75277", "stage": "RequestReceived", "requestURI": "/api/v1/namespaces/default/pods", "verb": "list", "user": {"username": "alice",...}, "sourceIPs": ["127.0.0.1"], ..., "requestReceivedTimestamp": "2018-08-07T22:10:35.428850Z"}}
Sample audit log for a Swarm cluster:
{"audit"; { "metadata": {...}, "level": "Metadata", "timestamp": "2018-08-07T22:10:35Z", "auditID": "7559d301-94e7-4ad6-901c-b587fab31512", "stage": "RequestReceived", "requestURI": "/v1.30/configs/create", "verb": "post", "user": {"username": "alice",...}, "sourceIPs": ["127.0.0.1"], ..., "requestReceivedTimestamp": "2018-08-07T22:10:35.428850Z"}}
API endpoints logging constraints¶
With regard to audit logging, for reasons having to do with system security a number of MKE API endpoints are either ignored or have their information redacted.
API endpoints ignored¶
The following API endpoints are ignored since they are not considered security events and can create a large amount of log entries:
/_ping/ca/auth/trustedregistryca/kubeauth/metrics/info/version\*/debug/openid_keys/apidocs/kubernetesdocs/manage
API endpoints information redacted¶
For security purposes, information for the following API endpoints is redacted from the audit logs:
/secrets/create(POST)/secrets/{id}/update(POST)/swarm/join(POST)/swarm/update(POST) -/auth/login(POST)Kubernetes secrets create/update endpoints
See also
See also
Enable MKE telemetry¶
You can set MKE to automatically record and transmit data to Mirantis through an encrypted channel for monitoring and analysis purposes. The data collected provides the Mirantis Customer Success Organization with information that helps us to better understand the operational use of MKE by our customers. It also provides key feedback in the form of product usage statistics, which enable our product teams to enhance Mirantis products and services.
Specifically, with MKE you can send hourly usage reports, as well as information on API and UI usage.
Caution
To send the telemetry, verify that dockerd and the MKE application container
can resolve api.segment.io and create a TCP (HTTPS) connection on port
443.
To enable telemetry in MKE:
Log in to the MKE web UI as an administrator.
At the top of the navigation menu at the left, click the user name drop-down to display the available options.
Click Admin Settings to display the available options.
Click Usage to open the Usage Reporting screen.
Toggle the Enable API and UI tracking slider to the right.
(Optional) Enter a unique label to identify the cluster in the usage reporting.
Click Save.
Enable and integrate SAML authentication¶
Security Assertion Markup Language (SAML) is an open standard for exchanging authentication and authorization data between parties. It is commonly supported by enterprise authentication systems. SAML-based single sign-on (SSO) gives you access to MKE through a SAML 2.0-compliant identity provider.
MKE supports the Okta and ADFS identity providers.
The SAML integration process is as follows.
Configure the Identity Provider (IdP).
Enable SAML and configure MKE as the Service Provider under Admin Settings > Authentication and Authorization.
Create (Edit) Teams to link with the Group memberships. This updates team membership information when a user signs in with SAML.
Note
If LDAP integration is enabled, refer to Use LDAP in conjunction with SAML for information on using SAML in parallel with LDAP.
Configure SAML integration on identity provider¶
Identity providers require certain values to successfully integrate with MKE. As these values vary depending on the identity provider, consult your identity provider documentation for instructions on how to best provide the needed information.
Okta integration values¶
Okta integration requires the following values:
Value |
Description |
|---|---|
URL for single signon (SSO) |
URL for MKE, qualified with |
Service provider audience URI |
URL for MKE, qualified with |
NameID format |
Select Unspecified. |
Application user name |
Email. For example, a custom |
Attribute Statements |
|
Group Attribute Statement |
|
Okta configuration
When two or more group names are expected to return with the assertion,
use the regex filter. For example, use the value apple|orange to
return groups apple and orange.
ADFS integration values¶
To enable ADFS integration:
Add a relying party trust.
Obtain the service provider metadata URI.
The service provider metadata URI value is the URL for MKE, qualified with
/enzi/v0/saml/metadata. For example,https://111.111.111.111/enzi/v0/saml/metadata.Add claim rules.
Convert values from AD to SAML
Display-name:Common NameE-Mail-Addresses:E-Mail AddressSAM-Account-Name:Name ID
Create a full name for MKE (custom rule):
c:[Type == "http://schemas.xmlsoap.org/claims/CommonName"] => issue(Type = "fullname", Issuer = c.Issuer, OriginalIssuer = c.OriginalIssuer, Value = c.Value, ValueType = c.ValueType);
Transform account name to
Name ID:Incoming type:
Name IDIncoming format: Unspecified
Outgoing claim type:
Name IDOutgoing format:
Transient ID
Pass admin value to allow admin access based on AD group. Send group membership as claim:
Users group: your admin group
Outgoing claim type:
is*adminOutgoing claim value:
1
Configure group membership for more complex organizations, with multiple groups able to manage access.
Send LDAP attributes as claims
Attribute store:
Active DirectoryAdd two rows with the following information:
LDAP attribute =
email address;outgoing claim type:email addressLDAP attribute =
Display*Name;outgoing claim type:common name
Mapping:
Token-Groups - Unqualified Names :
member-of
Note
Once you enable SAML, Service Provider metadata is available at
https://<SPHost>/enzi/v0/saml/metadata. The metadata link is also
labeled as entityID.
Only POST binding is supported for the Assertion Consumer Service,
which is located at https://<SP Host>/enzi/v0/saml/acs.
Configure SAML integration on MKE¶
SAML configuration requires that you know the metadata URL for your chosen identity provider, as well as the URL for the MKE host that contains the IP address or domain of your MKE installation.
To configure SAML integration on MKE:
Log in to the MKE web UI.
In the navigation menu at the left, click the user name drop-down to display the available options.
Click Admin Settings to display the available options.
Click Authentication & Authorization.
In the Identity Provider section in the details pane, move the slider next to SAML to enable the SAML settings.
In the SAML idP Server subsection, enter the URL for the identity provider metadata in the IdP Metadata URL field.
Note
If the metadata URL is publicly certified, you can continue with the default settings:
Skip TLS Verification unchecked
Root Certificates Bundle blank
Mirantis recommends TLS verification in production environments. If the metadata URL cannot be certified by the default certificate authority store, you must provide the certificates from the identity provider in the Root Certificates Bundle field.
In the SAML Service Provider subsection, in the MKE Host field, enter the URL that includes the IP address or domain of your MKE installation.
The port number is optional. The current IP address or domain displays by default.
(Optional) Customize the text of the sign-in button by entering the text for the button in the Customize Sign In Button Text field. By default, the button text is Sign in with SAML.
Copy the SERVICE PROVIDER METADATA URL, the ASSERTION CONSUMER SERVICE (ACS) URL, and the SINGLE LOGOUT (SLO) URL to paste into the identity provider workflow.
Click Save.
Note
To configure a service provider, enter the Identity Provider’s metadata URL to obtain its metadata. To access the URL, you may need to provide the CA certificate that can verify the remote server.
To link group membership with users, use the Edit or Create team dialog to associate SAML group assertion with the MKE team to synchronize user team membership when the user log in.
SAML security considerations¶
From the MKE web UI you can download a client bundle with which you can access MKE using the CLI and the API.
A client bundle is a group of certificates that enable command-line access and API access to the software. It lets you authorize a remote Docker engine to access specific user accounts that are managed in MKE, absorbing all associated RBAC controls in the process. Once you obtain the client bundle, you can execute Docker Swarm commands from your remote machine to take effect on the remote cluster.
Previously-authorized client bundle users can still access MKE, regardless of the newly configured SAML access controls.
Mirantis recomments that you take the following steps to ensure that access from the client bundle is in sync with the identity provider, and to thus prevent any previously-authorized users from accessing MKE through their existing client bundle:
Remove the user account from MKE that grants the client bundle access.
If group membership in the identity provider changes, replicate the change in MKE.
Continue using LDAP to sync group membership.
To download the client bundle:
Log in to the MKE web UI.
In the navigation menu at the left, click the user name drop-down to display the available options.
Click your account name to display the available options.
Click My Profile.
Click the New Client Bundle drop-down in the details pane and select Generate Client Bundle.
(Optional) Enter a name for the bundle into the Label field.
Click Confirm to initiate the bundle download.
Enable Helm with MKE¶
To use Helm with MKE, you must define the necessary roles in the
kube-system default service account.
Note
For comprehensive information on the use of Helm, refer to the Helm user documentation.
To enable Helm with MKE, enter the following kubectl commands in sequence:
kubectl create rolebinding default-view --clusterrole=view
--serviceaccount=kube-system:default --namespace=kube-system
kubectl create clusterrolebinding add-on-cluster-admin
--clusterrole=cluster-admin --serviceaccount=kube-system:default
Integrate SCIM¶
System for Cross-domain Identity Management (SCIM) is a standard for automating the exchange of user identity information between identity domains or IT systems. It offers an LDAP alternative for provisioning and managing users and groups in MKE, as well as for syncing users and groups with an upstream identity provider. Using SCIM schema and API, you can utilize Single sign-on services (SSO) across various tools.
Mirantis certifies the use of Okta 3.2.0, however MKE offers the discovery endpoints necessary to provide any system or application with the product SCIM configuration.
Configure SCIM for MKE¶
The Mirantis SCIM implementation uses SCIM version 2.0.
MKE SCIM intregration typically involves the following steps:
Enable SCIM.
Configure SCIM for authentication and access.
Specify user attributes.
Enable SCIM¶
Log in to the MKE web UI.
Click Admin Settings > Authentication & Authorization.
In the Identity Provider Integration section in the details pane, move the slider next to SCIM to enable the SCIM settings.
Configure SCIM authentication and access¶
In the SCIM configuration subsection, either enter the API token in the API Token field or click Generate to have MKE generate a UUID.
The base URL for all SCIM API calls is
https://<Host IP>/enzi/v0/scim/v2/. All SCIM methods are accessible
API endpoints of this base URL.
Bearer Auth is the API authentication method. When configured, you access SCIM
API endpoints through the Bearer <token> HTTP Authorization request header.
Note
SCIM API endpoints are not accessible by any other user (or their token), including the MKE administrator and MKE admin Bearer token.
The only SCIM method MKE supports is an HTTP authentication request header that contains a Bearer token.
Specify user attributes¶
The following table maps the user attribute fields in use by Mirantis to SCIM and SAML attributes.
MKE |
SAML |
SCIM |
|---|---|---|
Account name |
|
|
Account full name |
Attribute value in |
User’s |
Team group link name |
Attribute value in |
Group’s |
Team name |
N/A |
When creating a team, use the group’s |
Supported SCIM API endpoints¶
MKE supports SCIM API endpoints across three operational areas: User, Group, and Service Provider Configuration.
User operations¶
The SCIM API endpoints that serve in user operations provide the means to:
Retrieve user information
Create a new user
Update user information
For user GET and POST operations:
Filtering is only supported using the
userNameattribute andeqoperator. For example,filter=userName Eq "john".Attribute name and attribute operator are case insensitive. For example, the following two expressions have the same logical value:
filter=userName Eq "john"filter=Username eq "john"
Pagination is fully supported.
Sorting is not supported.
Returns a list of SCIM users (by default, 200 users per page).
Use the startIndex and count query parameters to paginate long lists of
users. For example, to retrieve the first 20 Users, set startIndex to 1
and count to 20, provide the following JSON request:
GET Host IP/enzi/v0/scim/v2/Users?startIndex=1&count=20
Host: example.com
Accept: application/scim+json
Authorization: Bearer h480djs93hd8
The response to the previous query returns paging metadata that is similar to the following example:
{
"totalResults":100,
"itemsPerPage":20,
"startIndex":1,
"schemas":["urn:ietf:params:scim:api:messages:2.0:ListResponse"],
"Resources":[{
...
}]
}
Retrieves a single user resource.
The value of the {id} should be the user’s ID. You can also use the
userName attribute to filter the results.
GET {Host IP}/enzi/v0/scim/v2/Users?{user ID}
Host: example.com
Accept: application/scim+json
Authorization: Bearer h480djs93hd8
Creates a user.
The operation must include the userName attribute and at least one
email address.
POST {Host IP}/enzi/v0/scim/v2/Users
Host: example.com
Accept: application/scim+json
Authorization: Bearer h480djs93hd8
Updates a user’s active status.
Reactivate inactive users by specifying "active": true. To deactivate
active users, specify "active": false. The value of the {id} should be
the user’s ID.
PATCH {Host IP}/enzi/v0/scim/v2/Users?{user ID}
Host: example.com
Accept: application/scim+json
Authorization: Bearer h480djs93hd8
Updates existing user information.
All attribute values are overwritten, including attributes for which empty
values or no values have been provided. If a previously set attribute value is
left blank during a PUT operation, the value is updated with a blank value
in accordance with the attribute data type and storage provider. The value of
the {id} should be the user’s ID.
Group operations¶
The SCIM API endpoints that serve in group operations provide the means to:
Create a new user group
Retrieve group information
Update user group membership (add/replace/remove users)
For group GET and POST operations:
Pagination is fully supported.
Sorting is not supported.
Retrieves information for a single group.
GET /scim/v1/Groups?{Group ID}
Host: example.com
Accept: application/scim+json
Authorization: Bearer h480djs93hd8
Returns a paginated list of groups (by default, ten groups per page).
Use the startIndex and count query parameters to paginate long lists of
groups.
GET /scim/v1/Groups?startIndex=4&count=500 HTTP/1.1
Host: example.com
Accept: application/scim+json
Authorization: Bearer h480djs93hd8
Creates a new group.
Add users to the group during group creation by supplying user ID
values in the members array.
Updates an existing group resource, allowing the addition or removal of
individual (or groups of) users from the group with a single operation. Add
is the default operation.
To remove members from a group, set the operation attribute of a member object
to delete.
Updates an existing group resource, overwriting all values for a group even if an attribute is empty or is not provided.
PUT replaces all members of a group with members that are provided by way
of the members attribute. If a previously set attribute is left blank
during a PUT operation, the new value is set to blank in accordance with
the data type of the attribute and the storage provider.
Service Provider configuration operations¶
The SCIM API endpoints that serve in Service provider configuration operations provide the means to:
Retrieve service provider resource type metadata
Retrieve schema for service provider and SCIM resources
Retrieve schema for service provider configuration
SCIM defines three endpoints to facilitate discovery of the SCIM service provider features and schema that you can retrieve using HTTP GET:
Discovers the resource types available on a SCIM service provider (for example, Users and Groups).
Each resource type defines the endpoints, the core schema URI that defines the resource, and any supported schema extensions.
Retrieves information about all supported resource schemas supported by a SCIM service provider.
Returns a JSON structure that describes the SCIM specification features
that are available on a service provider using a schemas attribute of
urn:ietf:params:scim:schemas:core:2.0:ServiceProviderConfig.
Integrate with an LDAP directory¶
MKE integrates with LDAP directory services, thus allowing you to manage users and groups from your organization directory and to automatically propagate the information to MKE and MSR.
Once you enable LDAP, MKE uses a remote directory server to create users automatically, and all logins are forwarded thereafter to the directory server.
When you switch from built-in authentication to LDAP authentication, all manually created users whose usernames fail to match any LDAP search results remain available.
When you enable LDAP authentication, you configure MKE to create user accounts only when users log in for the first time.
Note
If SAML integration is enabled, refer to Use LDAP in conjunction with SAML for information on using SAML in parallel with LDAP.
MKE integration with LDAP¶
To control the integration of MKE with LDAP, you create user searches. For
these user searches, you use the MKE web UI to specify multiple search
configurations and specify multiple LDAP servers
with which to integrate. Searches start with the Base DN, the Distinguished
Name of the node in the LDAP directory tree in which the search looks
for users.
MKE to LDAP synchronization workflow
The following occurs when MKE synchronizes with LDAP:
MKE creates a set of search results by iterating over each of the user search configurations, in an order that you specify.
MKE choses an LDAP server from the list of domain servers by considering the
Base DNfrom the user search configuration and selecting the domain server with the longest domain suffix match.Note
If no domain server has a domain suffix that matches the
Base DNfrom the search configuration, MKE uses the default domain server.MKE creates a list of users from the search and creates MKE accounts for each one.
Note
If you select the Just-In-Time User Provisioning option, user accounts are created only when users first log in.
Example workflow:
Consider an example with three LDAP domain servers and three user search configurations.
The example LDAP domain servers:
LDAP domain server name |
URL |
|---|---|
|
ldaps://ldap.example.com |
|
ldaps://ldap.subsidiary1.com |
|
ldaps://ldap.subsidiary2.com |
The example user search configurations:
User search configurations |
Description |
|---|---|
|
For this search configuration, |
|
For this search configuration, two of the domain servers have a
domain that is a suffix of this |
|
For this search configuration, no server with a domain specified is a
suffix of this |
Whenever user search results contain username collisions between the
domains, MKE uses only the first search result, and thus the ordering of the
user search configurations can be important. For example, if both the first and
third user search configurations result in a record with the username
jane.doe, the first has higher precedence and the second is ignored. As
such, it is important to implement a username attribute that is unique for
your users across all domains. As a best practice, choose something that is
specific to the subsidiary, such as the email address for each user.
Configure the LDAP integration¶
Note
MKE saves a minimum amount of user data required to operate, including any user name and full name attributes that you specify in the configuration, as well as the Distinguished Name (DN) of each synced user. MKE does not store any other data from the directory server.
Use the MKE web UI to configure MKE to create and authenticate users using an LDAP directory.
Access the LDAP controls¶
To configure LDAP integration you must first gain access to the controls for the service protocol.
Log in to the MKE web UI.
In the left-side navigation menu, click the user name drop-down to display the available options.
Navigate to Admin Settings > Authentication & Authorization.
In the Identity Provider section in the details pane, move the slider next to LDAP to enable the LDAP settings.
Set up an LDAP server¶
To configure an LDAP server, perform the following steps:
To set up a new LDAP server, configure the settings in the LDAP Server subsection:
Control
Description
LDAP Server URL
The URL for the LDAP server.
Reader DN
The DN of the LDAP account that is used to search entries in the LDAP server. As a best practice, this should be an LDAP read-only user.
Reader Password
The password of the account used to search entries in the LDAP server.
Skip TLS verification
Sets whether to verify the LDAP server certificate when TLS is in use. The connection is still encrypted, however it is vulnerable to man-in-the-middle attacks.
Use Start TLS
Defines whether to authenticate or encrypt the connection after connection is made to the LDAP server over TCP. To ignore the setting, set the LDAP Server URL field to
ldaps://.No Simple Pagination (RFC 2696)
Indicates that your LDAP server does not support pagination.
Just-In-Time User Provisioning
Sets whether to create user accounts only when users log in for the first time. Mirantis recommends using the default
truevalue.Note
Available as of MKE 3.6.4 The
disableReferralChasingsetting, which is currently only available by way of the MKE API, allows you to disable the default behavior that occurs when a referral URL is received as a result of an LDAP search request. Refer to LDAP Configuration through API for more information.Click Save to add your LDAP server.
Add additional LDAP domains¶
To integrate MKE with additional LDAP domains:
In the LDAP Additional Domains subsection, click Add LDAP Domain +. A set of input tools for configuring the additional domain displays.
Configure the settings for the new LDAP domain:
Control
Description
LDAP Domain
Text field in which to enter the root domain component of this server. A longest-suffix match of the
Base DNfor LDAP searches is used to select which LDAP server to use for search requests. If no matching domain is found, the default LDAP server configuration is put to use.LDAP Server URL
Text field in which to enter the URL for the LDAP server.
Reader DN
Text field in which to enter the DN of the LDAP account that is used to search entries in the LDAP server. As a best practice, this should be an LDAP read-only user.
Reader Password
The password of the account used to search entries in the LDAP server.
Skip TLS verification
Sets whether to verify the LDAP server certificate when TLS is in use. The connection is still encrypted, however it is vulnerable to man-in-the-middle attacks.
Use Start TLS
Sets whether to authenticate or encrypt the connection after connection is made to the LDAP server over TCP. To ignore the setting, set the LDAP Server URL field to
ldaps://.No Simple Pagination (RFC 2696)
Select if your LDAP server does not support pagination.
Note
Available as of MKE 3.6.4 The
disableReferralChasingsetting, which is currently only available by way of the MKE API, allows you to disable the default behavior that occurs when a referral URL is received as a result of an LDAP search request. Refer to LDAP Configuration through API for more information.Click Confirm to add the new LDAP domain.
Repeat the procedure to add any additional LDAP domains.
Add LDAP user search configurations¶
To add LDAP user search configurations to your LDAP integration:
In the LDAP User Search Configurations subsection, click Add LDAP User Search Configuration +.A set of input tools for configuring the LDAP user search configurations displays.
Field
Description
Base DN
Text field in which to enter the DN of the node in the directory tree, where the search should begin seeking out users.
Username Attribute
Text field in which to enter the LDAP attribute that serves as username on MKE. Only user entries with a valid username will be created.
A valid username must not be longer than 100 characters and must not contain any unprintable characters, whitespace characters, or any of the following characters:
/\[]:;|=,+*?<>'".Full Name Attribute
Text field in which to enter the LDAP attribute that serves as the user’s full name, for display purposes. If the field is left empty, MKE does not create new users with a full name value.
Filter
Text field in which to enter an LDAP search filter to use to find users. If the field is left empty, all directory entries in the search scope with valid username attributes are created as users.
Search subtree instead of just one level
Whether to perform the LDAP search on a single level of the LDAP tree, or search through the full LDAP tree starting at the
Base DN.Match Group Members
Sets whether to filter users further, by selecting those who are also members of a specific group on the directory server. The feature is helpful when the LDAP server does not support
memberOfsearch filters.Iterate through group members
Sets whether, when the Match Group Members option is enabled to sync users, the sync is done by iterating over the target group’s membership and making a separate LDAP query for each member, rather than through the use of a broad user search filter. This option can increase efficiency in situations where the number of members of the target group is significantly smaller than the number of users that would match the above search filter, or if your directory server does not support simple pagination of search results.
Group DN
Text field in which to enter the DN of the LDAP group from which to select users, when the Match Group Members option is enabled.
Group Member Attribute
Text field in which to enter the name of the LDAP group entry attribute that corresponds to the DN of each of the group members.
Click Confirm to add the new LDAP user search configurations.
Repeat the procedure to add any additional user search configurations. More than one such configuration can be useful in cases where users may be found in multiple distinct subtrees of your organization directory. Any user entry that matches at least one of the search configurations will be synced as a user.
Test LDAP login¶
Prior to saving your configuration changes, you can use the dedicated LDAP Test login tool to test the integration using the login credentials of an LDAP user.
Input the credentials for the test user into the provided Username and Passworfd fields:
Field
Description
Username
An LDAP user name for testing authentication to MKE. The value corresponds to the Username Attribute that is specified in the Add LDAP user search configurations section.
Password
The password used to authenticate (BIND) to the directory server.
Click Test. A search is made against the directory using the provided search
Base DN, scope, and filter. Once the user entry is found in the directory, a BIND request is made using the input user DN and the given password value.
Set LDAP synchronization¶
Following LDAP integration, MKE synchronizes users at the top of the hour, based on an intervial that is defined in hours.
To set LDAP synchronization, configure the following settings in the LDAP Sync Configuration section:
Field |
Description |
|---|---|
Sync interval |
The interval, in hours, to synchronize users between MKE and the LDAP server. When the synchronization job runs, new users found in the LDAP server are created in MKE with the default permission level. MKE users that do not exist in the LDAP server become inactive. |
Enable sync of admin users |
This option specifies that system admins should be synced directly with members of a group in your organization’s LDAP directory. The admins will be synced to match the membership of the group. The configured recovery admin user will also remain a system admin. |
Manually synchronize LDAP¶
In addition to configuring MKE LDAP synchronization, you can also perform a hot synchronization by clicking the Sync Now button in the LDAP Sync Jobs subsection. Here you can also view the logs for each sync jobs by clicking View Logs link associated with a particular job.
Revoke user access¶
Whenever a user is removed from LDAP, the effect on their MKE account is determined by the Just-In-Time User Provisioning setting:
false: Users deleted from LDAP become inactive in MKE following the next LDAP synchronization runs.true: A user deleted from LDAP cannot authenticate. Their MKE accounts remain active, however, and thus they can use their client bundles to run commands. To prevent this, deactivate the user’s MKE user account.
Synchronize teams with LDAP¶
MKE enables the syncing of teams within Organizations with LDAP, using either a search query or by matching a group that is established in your LDAP directory.
Log in to the MKE web UI as an administrator.
Navigate to Access Control > Orgs & Teams to display the Organizations that exist within your MKE instance.
Locate the name of the Organization that contains the MKE team that you want to sync to LDAP and click it to display all of the MKE teams for that Organization.
Hover your cursor over the MKE team that you want to sync with LDAP to reveal its vertical ellipsis, at the far right.
Click the vertical ellipsis and select Edit to call the Details screen for the team.
Toggle ENABLE SYNC TEAM MEMBERS to Yes to reveal the LDAP sync controls.
Toggle LDAP MATCH METHOD to set the LDAP match method you want to use to make the sync, Match Search Results (default) or Match Group Members.
For Match Search Results:
Enter a Base DN into the Search Base DN field, as it is established in LDAP.
Enter a search filter based on one or more attributes into the Search filter field.
Optional. Check Search subtree instead of just one level to enable search down through any sub-groups that exist within the group you entered into the Search Base DN field.
For Match Group Members:
Enter the group Distinguised Name (DN) into the Group DN field.
Enter a member attribute into the Group Member field.
Toggle IMMEDIATELY SYNC TEAM MEMBERS as appropriate.
Toggle ALLOW NON-LDAP MEMBERS as appropriate.
Click Save.
LDAP Configuration through API¶
LDAP-specific GET and PUT API endpoints are available in the
configuration resource. Swarm mode must be enabled to use the following
endpoints:
GET /api/ucp/config/auth/ldap- Returns information on your current system LDAP configuration.PUT /api/ucp/config/auth/ldap- Updates your LDAP configuration.
Configure an OpenID Connect identity provider¶
OpenID Connect (OIDC) allows you to authenticate MKE users with a trusted external identity provider.
Note
Kubernetes users who want client bundles to use OIDC must Download and configure the client bundle and replace the authorization section therein with the parameters presented in the Kubernetes OIDC Authenticator documentation.
For identity providers that require a client redirect URI, use
https://<MKE_HOST>/login. For identity providers that do not permit the use
of an IP address for the host, use https://<mke-cluster-domain>/login.
The requested scopes for all identity providers are "openid email". Claims
are read solely from the ID token that your identity provider returns. MKE does
not use the UserInfo URL to obtain user information. The default username
claim is sub. To use a different username claim, you must specify that
value with the usernameClaim setting in the MKE configuration file.
The following example details the MKE configuration file settings for using an external identity provider.
For the
*signInCriteriaarray,termis set to hosted domain ("hd") andvalueis set to the domain from which the user is permitted to sign in.For the
*adminRoleCriteriaarray,matchTypeis set to"contains", in case any administrators are assigned to multiple roles that includeadmin.
[auth.external_identity_provider]
wellKnownConfigUrl = "https://example.com/.well-known/openid-configuration"
clientId = "4dcdace6-4eb4-461d-892f-01aed344ac80"
clientSecret = "ed89aeddcdb4461ace640"
usernameClaim = "email"
caBundle = "----BEGIN CERTIFICATE----\nMIIF...UfTd\n----END CERTIFICATE----\n"
[[auth.external_identity_provider.signInCriteria]]
term = "hd"
value = "myorg.com"
matchType = "must"
[[auth.external_identity_provider.adminRoleCriteria]]
term = "roles"
value = "admin"
matchType = "contains"
Note
Using an external identity provider to sign in to the MKE web UI creates a new user session, and thus users who sign in this way will not be signed out when their ID token expires. Instead, the session lifetime is set using the auth.sessions parameters in the MKE configuration file.
Refer to the MKE configuration file auth.external_identity_provider (optional) for the complete reference documentation.
Use LDAP in conjunction with SAML¶
In MKE, you can configure LDAP to work together with SAML, though you may need to overcome certain issues to do so.
To enable LDAP and SAML to be used in tandem:
Log in to the MKE web UI.
In the left-side navigation panel, navigate to user name > Admin Settings > Authentication & Authorization.
Scroll down to the Identity Provider Integration section and verify that SAML is toggled to Enabled.
Select the Also allow LDAP users checkbox.
To sync teams with both LDAP and SAML users:
Log in to the MKE web UI.
Verify that LDAP and SAML teams are both enabled for syncing.
In the left-side navigation panel, navigate to Access Control > Orgs & Teams
Select the required organization and then select the required team.
Click the gear icon in the upper right corner.
On the Details tab, select ENABLE SYNC TEAM MEMBERS.
Select ALLOW NON-LDAP MEMBERS.
To determine a user’s authentication protocol:
Log in to the MKE web UI as an administrator.
In the left-side navigation panel, navigate to Access Control > Users and select the target user.
If an LDAP DN attribute is present next to Full Name and Admin, the user is managed by LDAP. If, however, the LDAP DN attribute is not present, the user is not managed by LDAP.
Overlapping user names¶
Unexpected behavior can result from having the same user name in both SAML and LDAP.
If just-in-time (JIT) provisioning is enabled in LDAP, MKE only allows log in attempts from the identity provider that first attempts to log in. MKE then blocks all log in attempts from the second identify provider.
If JIT provisioning is disabled in LDAP, the LDAP synchronization, which occurs at regular intervals, always overrides the ability of the SAML user account to log in.
To allow overlapping user names:
There may at times be a user who has the same name in both LDAP and SAML who you want to be able to sign in using either protocol.
Define a custom SAML attribute with a name of
dnand a value that is equivalent to the user account distinguished name (DN) with the LDAP provider. Refer to Define a custom SAML attribute in the Okta documentation for more information.Note
MKE considers such users to be LDAP users. As such, should their LDAP DN change, the custom SAML attribute must be updated to match.
Log in to the MKE web UI.
From the left-side navigation panel, navigate to <user name> > Admin Settings > Authentication & Authorization and scroll down to the LDAP section.
Under SAML integration, select Allow LDAP users to sign in using SAML.
Manage services node deployment¶
You can configure MKE to allow users to deploy and run services in worker nodes only, to ensure that all cluster management functionality remains performant and to enhance cluster security.
Important
If for whatever reason a user deploys a malicious service that can affect the node on which it is running, that service will not be able to strike any other nodes in the cluster or have any impact on cluster management functionality.
Restrict services deployment to Swarm worker nodes¶
To keep manager nodes performant, it is necessary at times to restrict service deployment to Swarm worker nodes.
To restrict services deployment to Swarm worker nodes:
Log in to the MKE web UI with administrator credentials.
Click the user name at the top of the navigation menu.
Navigate to Admin Settings > Orchestration.
Under Container Scheduling, toggle all of the sliders to the left to restrict the deployment only to worker nodes.
Note
Creating a grant with the Scheduler role against the / collection
takes precedence over any other grants with Node Schedule on
subcollections.
Restrict services deployment to Kubernetes worker nodes¶
By default, MKE clusters use Kubernetes taints and tolerations to prevent user workloads from deploying to MKE manager or MSR nodes.
Note
Workloads deployed by an administrator in the kube-system namespace do
not follow scheduling constraints. If an administrator deploys a
workload in the kube-system namespace, a toleration is applied to bypass
the taint, and the workload is scheduled on all node types.
To view the taints, run the following command:
$ kubectl get nodes <mkemanager> -o json | jq -r '.spec.taints | .[]'
Example of system response:
{
"effect": "NoSchedule",
"key": "com.docker.ucp.manager"
}
Allow services deployment on Kubernetes MKE manager or MSR nodes¶
You can circumvent the protections put in place by Kubernetes taints and tolerations. For details, refer to Restrict services deployment to Kubernetes worker nodes.
Schedule services deployment on manager and MSR nodes¶
Log in to the MKE web UI with administrator credentials.
Click the user name at the top of the navigation menu.
Navigate to Admin Settings > Orchestration.
Select from the following options:
Under Container Scheduling, toggle to the right the slider for Allow administrators to deploy containers on MKE managers or nodes running MSR.
Under Container Scheduling, toggle to the right the slider for Allow all authenticated users, including service accounts, to schedule on all nodes, including MKE managers and MSR nodes..
Following any scheduling action, MKE applies a toleration to new workloads, to allow the Pods to be scheduled on all node types. For existing workloads, however, it is necessary to manually add the toleration to the Pod specification.
Add a toleration to the Pod specification for existing workloads¶
Add the following toleration to the Pod specification, either through the MKE web UI or using the kubectl edit <object> <workload> command:
tolerations: - key: "com.docker.ucp.manager" operator: "Exists"
Run the following command to confirm the successful application of the toleration:
kubectl get <object> <workload> -o json | jq -r '.spec.template.spec.tolerations | .[]'
Example of system response:
{
"key": "com.docker.ucp.manager",
"operator": "Exists"
}
Caution
A NoSchedule taint is present on MKE manager and MSR nodes, and if you
disable scheduling on managers and/or workers a toleration for that taint
will not be applied to the deployments. As such, you should not schedule on
these nodes, except when the Kubernetes workload is deployed in the
kube-system namespace.
Run only the images you trust¶
With MKE you can force applications to use only Docker images that are signed by MKE users you trust. Every time a user attempts to deploy an application to the cluster, MKE verifies that the application is using a trusted Docker image. If a trusted Docker image is not in use, MKE halts the deployment.
By signing and verifying the Docker images, you ensure that the images in use in your cluster are trusted and have not been altered, either in the image registry or on their way from the image registry to your MKE cluster.
Example workflow
A developer makes changes to a service and pushes their changes to a version control system.
A CI system creates a build, runs tests, and pushes an image to the Mirantis Secure Registry (MSR) with the new changes.
The quality engineering team pulls the image, runs more tests, and signs and pushes the image if the image is verified.
IT operations deploys the service, but only if the image in use is signed by the QA team. Otherwise, MKE will not deploy.
To configure MKE to only allow running services that use Docker trusted images:
Log in to the MKE web UI.
In the left-side navigation menu, click the user name drop-down to display the available options.
Click Admin Settings > Docker Content Trust to reveal the Content Trust Settings page.
Enable Run only signed images.
Important
At this point, MKE allows the deployment of any signed image, regardless of signee.
(Optional) Make it necessary for the image to be signed by a particular team or group of teams:
Click Add Team+ to reveal the two-part tool.
From the drop-down at the left, select an organization.
From the drop-down at the right, select a team belonging to the organization you selected.
Repeat the procedure to configure additional teams.
Note
If you specify multiple teams, the image must be signed by a member of each team, or someone who is a member of all of the teams.
Click Save.
MKE immediately begins enforcing the image trust policy. Existing services continue to run and you can restart them as necessary. From this point, however, MKE only allows the deployment of new services that use a trusted image.
Set user session properties¶
MKE enables the setting of various user sessions properties, such as session timeout and the permitted number of concurrent sessions.
To configure MKE login session properties:
Log in to the MKE web UI.
In the left-side navigation menu, click the user name drop-down to display the available options.
Click Admin Settings > Authentication & Authorization to reveal the MKE login session controls.
The following table offers information on the MKE login session controls:
Field |
Description |
|---|---|
Lifetime Minutes |
The set duration of a login session in minutes, starting from the moment MKE generates the session. MKE invalidates the active session once this period expires and the user must re-authenticate to establish a new session.
|
Renewal Threshold Minutes |
The time increment in minutes by which MKE extends an active session prior to session expiration. MKE extends the session by the amount specified in Lifetime Minutes. The threshold value cannot be greater than that set in Lifetime Minutes. To specify that sessions not be extended, set the threshold value to 0. Be aware, though, that this may cause MKE web UI users to be unexpectedly logged out.
|
Per User Limit |
The maximum number of sessions a user can have running
simultaneously. If the creation of a new session results in the
exceeding of this limit, MKE will delete the session least recently put
to use. Specifically, every time you use a session token, the server
marks it with the current time ( To disable the Per User Limit setting, set the value to 0.
|
Configure an MKE cluster¶
Important
The MKE configuration file documentation is up-to-date for the latest MKE release. As such, if you are running an earlier version of MKE, you may encounter detail for configuration options and parameters that are not applicable to the version of MKE you are currently running.
Refer to the MKE Release Notes for specific version-by-version information on MKE configuration file additions and changes.
The configuring of an MKE cluster takes place through the application of a TOML file. You use this file, the MKE configuration file, to import and export MKE configurations, to both create new MKE instances and to modify existing ones.
Refer to example-config in the MKE CLI reference documentation to learn how to download an example MKE configuration file.
Use an MKE configuration file¶
Put the MKE configuration file to work for the following use cases:
Set the configuration file to run at the install time of new MKE clusters
Use the API to import the file back into the same cluster
Use the API to import the file into multiple clusters
To make use of an MKE configuration file, you edit the file using either the MKE web UI or the command line interface (CLI). Using the CLI, you can either export the existing configuration file for editing, or use the example-config command to view and edit an example TOML MKE configuration file.
docker container run --rm
-v /var/run/docker.sock:/var/run/docker.sock \
mirantis/ucp:3.6.19 \ example-config
Modify an existing MKE configuration¶
Working as an MKE admin, use the config-toml API from within the directory
of your client certificate bundle to export the current MKE settings to a TOML
file.
As detailed herein, the command set exports the current configuration for the
MKE hostname MKE_HOST to a file named mke-config.toml:
Define the following environment variables:
export MKE_USERNAME=<mke-username> export MKE_PASSWORD=<mke-password> export MKE_HOST=<mke-fqdm-or-ip-address>
Obtain and define an
AUTHTOKENenvironment variable:AUTHTOKEN=$(curl --silent --insecure --data '{"username":"'$MKE_USERNAME'","password":"'$MKE_PASSWORD'"}' https://$MKE_HOST/auth/login | jq --raw-output .auth_token)
Download the current MKE configuration file.
curl --silent --insecure -X GET "https://$MKE_HOST/api/ucp/config-toml" -H "accept: application/toml" -H "Authorization: Bearer $AUTHTOKEN" > mke-config.toml
Edit the MKE configuration file, as needed. For comprehensive detail, refer to Configuration options.
Upload the newly edited MKE configuration file:
Note
You may need to reacquire the
AUTHTOKEN, if significant time has passed since you first acquired it.curl --silent --insecure -X PUT -H "accept: application/toml" -H "Authorization: Bearer $AUTHTOKEN" --upload-file 'mke-config.toml' https://$MKE_HOST/api/ucp/config-toml
Apply an existing configuration at install time¶
To customize a new MKE instance using a configuration file, you must create the file prior to installation. Then, once the new configuration file is ready, you can configure MKE to import it during the installation process using Docker Swarm.
To import a configuration file at installation:
Create a Docker Swarm Config object named
com.docker.mke.configand the TOML value of your MKE configuration file contents.When installing MKE on the cluster, specify the
--existing-configflag to force the installer to use the new Docker Swarm Config object for its initial configuration.Following the installation, delete the
com.docker.mke.configobject.
Configuration options¶
auth table¶
Parameter |
Required |
Description |
|---|---|---|
|
no |
The name of the authorization backend to use, Default: |
|
no |
The role assigned to new users for their private resource sets. Valid values: Default: |
auth.sessions¶
Parameter |
Required |
Description |
|---|---|---|
|
no |
The initial session lifetime, in minutes. Default: |
|
no |
The length of time, in minutes, before the expiration of a session
where, if used, a session will be extended by the current configured
lifetime from then. A value of Default: |
|
no |
The maximum number of sessions that a user can have simultaneously active. If creating a new session will put a user over this limit, the least recently used session is deleted. A value of Default: |
|
no |
If set, the user token is stored in |
auth.external_identity_provider (optional)¶
Configures MKE with an external OpenID Connect (OIDC) identity provider.
Parameter |
Required |
Description |
|---|---|---|
|
yes |
Sets the OpenID discovery endpoint, ending in
|
|
yes |
Sets the client ID, which you obtain from your identity provider. |
|
no (recommended) |
Sets the client secret, which you obtain from your identity provider. |
|
no |
Sets the unique JWT ID token claim that contains the user names from your identity provider. Default: |
|
no |
Sets the PEM certificate bundle that MKE uses to authenticate the discovery, issuer, and JWKs endpoints. |
|
no |
Sets the HTTP proxy for your identity provider. |
|
no |
Sets the HTTPS proxy for your identity provider. |
|
no |
Sets the ID token issuer. If left blank, the value is obtained automatically from the discovery endpoint. |
|
no |
Sets the MKE service ID with the JWK URI for the identity provider. If left blank, the service ID is generated automatically. Warning Do not remove or replace an existing value. |
auth.external_identity_provider.signInCriteria array (optional)¶
An array of claims that ID tokens require for use with MKE.
Parameter |
Required |
Description |
|---|---|---|
|
yes |
Sets the name of the claim. |
|
yes |
Sets the value for the claim in the form of a string. |
|
yes |
Sets how MKE evaluates the JWT claim. Valid values:
|
auth.external_identity_provider.adminRoleCriteria array (optional)¶
An array of claims that admin user ID tokens require for use with MKE. Creating a new account using a token that satisfies the criteria determined by this array automatically produces an administrator account.
Parameter |
Required |
Description |
|---|---|---|
|
yes |
Sets the name of the claim. |
|
yes |
Sets the value for the claim in the form of a string. |
|
yes |
Sets how the JWT claim is evaluated. Valid values:
|
auth.account_lock (optional)¶
Parameter |
Required |
Description |
|---|---|---|
|
no |
Sets whether the MKE account lockout feature is enabled. |
|
no |
Sets the number of failed log in attempts that can occur before an account is locked. |
|
no |
Sets the desired lockout duration in seconds. A value of |
hardening_configuration (optional)¶
The hardening_enabled option must be set to true to enable all
other hardening_configuration options.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Parent option that when set to Default: |
|
no |
The option can only be enabled when Limits kernel capabilities to the minimum required by each container. Components run using Docker default capabilities by default. When you
enable Default: |
|
no |
The option can only be enabled when Sets the maximum number of PIDs MKE can allow for their respective orchestrators. The Default: |
|
no |
The option can only be enabled when When set to Default: |
|
no |
The option can only be enabled when When set to
Default: |
registries array (optional)¶
An array of tables that specifies the MSR instances that are managed by the current MKE instance.
Parameter |
Required |
Description |
|---|---|---|
|
yes |
Sets the address for connecting to the MSR instance tied to the MKE cluster. |
|
yes |
Sets the MSR instance’s OpenID Connect Client ID, as registered with the Docker authentication provider. |
|
no |
Specifies the root CA bundle for the MSR instance if you are using a
custom certificate authority (CA). The value is a string with the
contents of a |
audit_log_configuration table (optional)¶
Configures audit logging options for MKE components.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Specifies the audit logging level. Valid values: empty (to disable audit logs), Default: empty |
|
no |
Sets support dumps to include audit logs in the logs of
the Valid values: Default: |
scheduling_configuration table (optional)¶
Specifies scheduling options and the default orchestrator for new nodes.
Note
If you run a kubectl command, such as kubectl describe
nodes, to view scheduling rules on Kubernetes nodes, the results that
present do not reflect the MKE admin settings conifguration. MKE uses taints
to control container scheduling on nodes and is thus unrelated to the
kubectl Unschedulable boolean flag.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Determines whether administrators can schedule containers on manager nodes. Valid values: Default: You can also set the parameter using the MKE web UI:
|
|
no |
Sets the type of orchestrator to use for new nodes that join the cluster. Valid values: Default: |
tracking_configuration table (optional)¶
Specifies the analytics data that MKE collects.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Set to disable analytics of usage information. Valid values: Default: |
|
no |
Set to disable analytics of API call information. Valid values: Default: |
|
no |
Set a label to be included with analytics. |
|
no |
Set to enable OpsCare. Valid values: Default: |
trust_configuration table (optional)¶
Specifies whether MSR images require signing.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Set to require the signing of images by content trust. Valid values: Default: You can also set the parameter using the MKE web UI:
|
|
no |
A string array that specifies which users or teams must sign images. |
|
no |
A string array that specifies repos that are to bypass content trust
check, for example, |
log_configuration table (optional)¶
Configures the logging options for MKE components.
Parameter |
Required |
Description |
|---|---|---|
|
no |
The protocol to use for remote logging. Valid values: Default: |
|
no |
Specifies a remote syslog server to receive sent MKE controller logs. If
omitted, controller logs are sent through the default Docker daemon
logging driver from the |
|
no |
The logging level for MKE components. Valid values (syslog priority levels): |
license_configuration table (optional)¶
Enables automatic renewal of the MKE license.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Set to enable attempted automatic license renewal when the license nears expiration. If disabled, you must manually upload renewed license after expiration. Valid values: Default: |
custom headers (optional)¶
Included when you need to set custom API headers. You can repeat this
section multiple times to specify multiple separate headers. If you
include custom headers, you must specify both name and value.
[[custom_api_server_headers]]
Item |
Description |
|---|---|
name |
Set to specify the name of the custom header with |
value |
Set to specify the value of the custom header with |
user_workload_defaults (optional)¶
A map describing default values to set on Swarm services at creation time if those fields are not explicitly set in the service spec.
[user_workload_defaults]
[user_workload_defaults.swarm_defaults]
Parameter |
Required |
Description |
|---|---|---|
|
no |
Delay between restart attempts. The value is input in the <number><value type> formation. Valid value types include:
Default: |
|
no |
Maximum number of restarts before giving up. Default: |
cluster_config table (required)¶
Configures the cluster that the current MKE instance manages.
The dns, dns_opt, and dns_search settings configure the DNS
settings for MKE components. These values, when assigned, override the
settings in a container /etc/resolv.conf file.
Parameter |
Required |
Description |
|---|---|---|
|
yes |
Sets the port that the Default: |
|
yes |
Sets the port the Kubernetes API server monitors. |
|
no |
Protects kernel parameters from being overridden by kubelet. Default: Important When enabled, kubelet can fail to start if the following kernel parameters are not properly set on the nodes before you install MKE or before adding a new node to an existing cluster: vm.panic_on_oom=0
vm.overcommit_memory=1
kernel.panic=10
kernel.panic_on_oops=1
kernel.keys.root_maxkeys=1000000
kernel.keys.root_maxbytes=25000000
For more information, refer to Configure kernel parameters. |
|
no |
Enables auditing to the log file in the kube-apiserver container. Important
For more information, refer to the official Kubernetes documentation Troubleshooting Clusters - Audit backends. Default: |
|
yes |
Sets the port that the Default: |
|
no |
Sets placement strategy for container scheduling. Be aware that this does not affect swarm-mode services. Valid values: |
|
yes |
Array of IP addresses that serve as nameservers. |
|
yes |
Array of options in use by DNS resolvers. |
|
yes |
Array of domain names to search whenever a bare unqualified host name is used inside of a container. |
|
no |
Determines whether specialized debugging endpoints are enabled for profiling MKE performance. Valid values: Default: |
|
no |
Sets the timeout in seconds for the RBAC information cache of MKE non-Kubernetes resource listing APIs. Setting changes take immediate effect and do not require a restart of the MKE controller. Default: Once you enable the cache, the result of non-Kubernetes resource listing APIs only reflects the latest RBAC changes for the user when the cached RBAC info times out. |
|
no |
Sets the key-value store timeout setting, in milliseconds. Default: |
|
Required |
Sets the key-value store snapshot count. Default: |
|
no |
Specifies an optional external load balancer for default links to services with exposed ports in the MKE web interface. |
|
no |
Specifies the URL of a Kubernetes YAML file to use to install a CNI plugin. Only applicable during initial installation. If left empty, the default CNI plugin is put to use. |
|
no |
Sets the metrics retention time. |
|
no |
Sets the interval for how frequently managers gather metrics from nodes in the cluster. |
|
no |
Sets the interval for the gathering of storage metrics, an operation that can become expensive when large volumes are present. |
|
no |
Enables the |
|
no |
Sets the size of the cache for MKE RethinkDB servers. Default: 1GB Leaving the field empty or specifying |
|
no |
Determines whether the Valid values: Default: |
|
no |
Sets the cloud provider for the Kubernetes cluster. |
|
yes |
Sets the subnet pool from which the IP for the Pod should be allocated from the CNI IPAM plugin. Default: |
|
no |
Sets the IPIP MTU size for the Calico IPIP tunnel interface. |
|
yes |
Sets the IP count for Azure allocator to allocate IPs per Azure virtual machine. |
|
yes |
Sets the subnet pool from which the IP for Services should be allocated. Default: |
|
yes |
Sets the port range for Kubernetes services within which the type
Default: |
|
no |
Sets the configuration options for the Kubernetes API server. Be aware that this parameter function is only for development and testing. Arbitrary Kubernetes configuration parameters are not tested and supported under the MKE Software Support Agreement. |
|
no |
Sets the configuration options for the Kubernetes controller manager. Be aware that this parameter function is only for development and testing. Arbitrary Kubernetes configuration parameters are not tested and supported under the MKE Software Support Agreement. |
|
no |
Sets the configuration options for Be aware that this parameter function is only for development and testing. Arbitrary Kubernetes configuration parameters are not tested and supported under the MKE Software Support Agreement. |
|
no |
Sets the configuration options for the Kubernetes scheduler. Be aware that this arameter function is only for development and testing. Arbitrary Kubernetes configuration parameters are not tested and supported under the MKE Software Support Agreement. |
|
no |
Set to store data about collections for volumes in the MKE local KV store instead of on the volume labels. The parameter is used to enforce access control on volumes. |
|
no |
Reserves resources for MKE and Kubernetes components that are running on manager nodes. |
|
no |
Reserves resources for MKE and Kubernetes components that are running on worker nodes. |
|
yes |
Sets the number of Pods that can run on a node. Maximum: Default: |
|
no |
Sets the maximum number of Pods per core.
Recommended: Default: |
|
no |
Enables IPSec network encryption in Kubernetes. Valid values: Default: |
|
no |
Enables image scan result aggregation. The feature displays image vulnerabilities in shared resource/containers and shared resources/images pages. Valid values: Default: |
|
no |
Determines whether resource polling is disabled for both Swarm and Kubernetes resources, which is recommended for production instances. Valid values: Default: |
|
no |
Sets the OIDC client ID, using the eNZi service ID that is in the ODIC authorization flow. |
|
no |
Determines whether the UI is hidden for all Swarm-only object types (has no effect on Admin Settings). Valid values: Default: You can also set the parameter using the MKE web UI:
|
|
yes |
Sets Calico as the CNI provider, managed by MKE. Note that Calico is the default CNI provider. |
|
yes |
Enables Calico eBPF mode. |
|
yes |
Sets the use of Kubernetes default values for iptables drop and masquerade bits. |
|
yes |
Sets the operational mode for Valid values: Default: |
|
no |
Sets the value for the |
|
no |
Sets the value for the |
|
no |
Sets the value for the |
|
no |
Sets the cron expression used for the scheduling of image pruning. The parameter accepts either full crontab specifications or descriptors, but not both.
Refer to the cron documentation for more information. |
|
no |
Sets the CPU usage threshold, above which the MKE web UI displays a warning banner. Default: |
|
no |
Sets the MKE CPU usage measurement interval, which enables the function
of the Default: |
|
no |
Sets the etcd storage size limit. Example values: Default value: |
|
no |
Enables the NVIDIA device partitioner. Default: |
|
no |
Enables profiling for the Kubernetes API server. Default: |
|
no |
Enables profiling for the Kubernetes controller manager. Default: |
|
no |
Enables profiling for the Kubernetes scheduler. Default: |
|
no |
Enables kube scheduler to bind to all available network interfaces, rather than just localhost. Default: |
|
no |
Extends support of FlexVolume drivers, which have been deprecated since the release of MKE 3.4.13. Default: |
|
no |
Warning Implement Enables public key authentication cache. Note
Default: |
|
no |
Subject alternative names for manager nodes. |
|
no |
Allows users to set the threshold for the terminated Pod garbage collector in Kube Controller Manager according to their cluster-specific requirement. Default: |
|
no |
Timeout for Kube API server requests. Default: |
|
no |
Enables the Default: |
|
no |
Sets the maximum number of days for which to retain old audit log files in Kubernetes API server. Default: |
|
no |
Sets the maximum number of audit log files for which to retain in the Kubernetes API server. Default: |
|
no |
Sets the maximum size the audit log file can attain, in megabytes, before it is rotated in Kubernetes API server. Default: |
|
no |
Specifies a Kubernetes audit logging policy. Refer to https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/ for more information. |
|
no |
Enables the use of a specified custom audit policy yaml file. Default: |
cluster_config.image_prune_whitelist (optional)¶
Configures the images that you do not want removed by MKE image pruning.
Note
Where possible, use the image ID to specify the image rather than the image name.
cluster_config.ingress_controller (optional)¶
Set the configuration for the NGINX Ingress Controller to manage traffic that originates outside of your cluster (ingress traffic).
Note
Prior versions of MKE use Istio Ingress to manage traffic that originates from outside of the cluster, which employs many of the same parameters as NGINX Ingress Controller.
Parameter |
Required |
Description |
|---|---|---|
|
No |
Disables HTTP ingress for Kubernetes. Valid values: Default: |
|
No |
Sets the number of NGINX Ingress Controller deployment replicas. Default: |
|
No |
Sets the list of external IPs for Ingress service. Default: |
|
No |
Enables an external load balancer. Valid values: Default: |
|
No |
Enables preserving inbound traffic source IP. Valid values: Default: |
|
No |
Sets ports to expose. For each port, provide arrays that contain the following port information (defaults as displayed):
|
|
No |
Sets node affinity.
|
|
No |
Sets node toleration. For each node, provide an array that contains the following information (defaults as displayed):
|
|
No |
Sets advanced options for the NGINX proxy. NGINX Ingress Controller uses Examples:
|
cluster_config.policy_enforcement (optional)¶
Enable and disable Pod Security Policies (PSPs).
Parameter |
Required |
Description |
|---|---|---|
|
No |
Enables the use of Pod Security Policies (PSPs). Valid values: Default: |
cluster_config.policy_enforcement.gatekeeper (optional)¶
Enable and disable OPA Gatekeeper for policy enforcement.
Note
By design, when the OPA Gatekeeper is disabled using the configuration file, the Pods are deleted but the policies are not cleaned up. Thus, when the OPA Gatekeeper is re-enabled, the cluster can immediately adopt the existing policies.
The retention of the policies poses no risk, as they are just data on the API server and have no value outside of a OPA Gatekeeper deployment.
Parameter |
Required |
Description |
|---|---|---|
|
No |
Enables the Gatekeeper function. Valid values: Default: |
|
No |
Excludes from the Gatekeeper admission webhook all of the resources that are contained in a list of namespaces. Specify as a comma-separated list. For example: |
iSCSI (optional)¶
Configures iSCSI options for MKE.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Enables iSCSI-based Persistent Volumes in Kubernetes. Valid values: Default: |
|
no |
Specifies the path of the Default: |
|
no |
Specifies the path of the Default: |
pre_logon_message¶
Configures a pre-logon message.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Sets a pre-logon message to alert users prior to log in. |
backup_schedule_config (optional)¶
Configures backup scheduling and notifications for MKE.
Parameter |
Required |
Description |
|---|---|---|
|
yes |
Sets the number of days that elapse before a user is notified that they
have not performed a recent backup. Set to Default: |
|
yes |
Enables backup scheduling. Valid values: Default: |
|
yes |
Sets the storage path for scheduled backups. Use
|
|
yes |
Sets whether a passphrase is necessary to encrypt the TAR file. A value
of Default: |
|
yes |
Encrypts the TAR file with a passphrase for all scheduled backups. Must
remain empty if Do not share the configuration file if a passphrase is used, as the passphrase displays in plain text. |
|
yes |
Sets the cron expression in use for scheduling backups. The parameter accepts either full crontab specifications or descriptors, but not both.
For more information, refer to the cron documentation. |
|
yes |
Determines whether a log file is generated in addition to the backup. Refer to backup for more information. |
|
yes |
Sets the number of backups to store. Once this number is reached, older
backups are deleted. Set to |
windows_gmsa¶
Configures use of Windows GMSA credentia specifications.
Parameter |
Required |
Description |
|---|---|---|
|
no |
Allows creation of GMSA credential specifications for the Kubernetes cluster, as well as automatic population of full credential specifications for any Pod on which the GMSA credential specification is referenced in the security context of that Pod. The schema for gmsa credential spec MKE uses is publicly documented at https://github.com/kubernetes-sigs/windows-gmsa/blob/master/charts/gmsa/templates/credentialspec.yaml. For information on how to enable GMSA and how to obtain different components of the GMSA specification for one or more GMSA accounts in your domain, refer to the official Windows documentation. |
Scale an MKE cluster¶
By adding or removing nodes from the MKE cluster, you can horizontally scale MKE to fit your needs as your applications grow in size and use.
Scale using the MKE web UI¶
For detail on how to use the MKE web UI to scale your cluster, refer to Join Linux nodes or Join Windows worker nodes, depending on which operating system you use. In particular, these topics offer information on adding nodes to a cluster and configuring node availability.
Scale using the CLI¶
You can also use the command line to perform all scaling operations.
Scale operation |
Command |
|---|---|
Obtain the join token |
Run the following command on a manager node to obtain the docker swarm join-token <node-type>
|
Configure a custom listen address |
Specify the address and port where the new node listens for inbound cluster management traffic: docker swarm join \
--token SWMTKN-1-2o5ra9t7022neymg4u15f3jjfh0qh3yof817nunoioxa9i7lsp-dkmt01ebwp2m0wce1u31h6lmj \
--listen-addr 234.234.234.234 \
192.168.99.100:2377
|
Verify node addition |
Once your node is added, run the following command on a manager node to verify its presence: docker node ls
|
Set node availability state |
Use the --availability option to set node availability,
indicating docker node update --availability <availability-state> <node-hostname>
|
Remove the node |
docker node rm <node-hostname>
|
Configure KMS plugin for MKE¶
Mirantis Kubernetes Engine (MKE) offers support for a Key Management Service (KMS) plugin that allows access to third-party secrets management solutions, such as Vault. MKE uses this plugin to facilitate access from Kubernetes clusters.
MKE will not health check, clean up, or otherwise manage the KMS plugin. Thus, you must deploy KMS before a machine becomes a MKE manager, or else it may be considered unhealthy.
Configuration¶
Use MKE to configure the KMS plugin configuration. MKE maintains ownership of the Kubernetes EncryptionConfig file, where the KMS plugin is configured for Kubernetes. MKE does not check the file contents following deployment.
MKE adds new configuration options to the cluster configuration table. Configuration of these options takes place through the API and not the MKE web UI.
The following table presents the configuration options for the KMS plugin, all of which are optional.
Parameter |
Type |
Description |
|---|---|---|
|
bool |
Sets MKE to configure a KMS plugin. |
|
string |
Name of the KMS plugin resource (for example, |
|
string |
Path of the KMS plugin socket. The path must refer to a UNIX socket on
the host (for example, |
kms_cachesize |
int |
Number of data encryption keys (DEKs) to cache in the clear. |
Use a local node network in a swarm¶
Mirantis Kubernetes Engine (MKE) can use local network drivers to orchestrate your cluster. You can create a config network with a driver such as MAC VLAN, and use this network in the same way as any other named network in MKE. In addition, if it is set up as attachable you can attach containers.
Warning
Encrypting communication between containers on different nodes only works with overlay networks.
Create node-specific networks with MKE¶
To create a node-specific network for use with MKE, always do so through MKE, using either the MKE web UI or the CLI with an admin bundle. If you create such a network without MKE, it will not have the correct access label and it will not be available in MKE.
Create a MAC VLAN network¶
Log in to the MKE web UI as an administrator.
In the left-side navigation menu, click Swarm > Networks.
Click Create to call the Create Network screen.
Select macvlan from the Drivers` dropdown.
Enter macvlan into the Name field.
Select the type of network to create, Network or Local Config.
If you select Local Config, the SCOPE is automatically set to Local. You subsequently select the nodes for which to create the Local Config from those listed. MKE will prefix the network with the node name for each selected node to ensure consistent application of access labels, and you then select a Collection for the Local Configs to reside in. All Local Configs with the same name must be in the same collection, or MKE returns an error. If you do not not select a Collection, the network is placed in your default collection, which is
/in a new MKE installation.If you select Network, the SCOPE is automatically set to Swarm. Choose an existing Local Config from which to create the network. The network and its labels and collection placement are inherited from the related Local Configs.
Optional. Configure IPAM.
Click Create.
Use your own TLS certificates¶
To ensure all communications between clients and MKE are encrypted, all MKE services are exposed using HTTPS. By default, this is done using self-signed TLS certificates that are not trusted by client tools such as web browsers. Thus, when you try to access MKE, your browser warns that it does not trust MKE or that MKE has an invalid certificate.
You can configure MKE to use your own TLS certificates. As a result, your browser and other client tools will trust your MKE installation.
Mirantis recommends that you make this change outside of peak business hours. Your applications will continue to run normally, but existing MKE client certificates will become invalid, and thus users will have to download new certificates to access MKE from the CLI.
To configure MKE to use your own TLS certificates and keys:
Log in to the MKE web UI as an administrator.
In the left-side navigation panel, navigate to <user name> > Admin Settings > Certificates.
Upload your certificates and keys based on the following table.
Note
All keys and certificates must be uploaded in PEM format.
Type
Description
Private key
The unencrypted private key for MKE. This key must correspond to the public key used in the server certificate. This key does not use a password.
Click Upload Key to upload a PEM file.
Server certificate
The MKE public key certificate, which establishes a chain of trust up to the root CA certificate. It is followed by the certificates of any intermediate certificate authorities.
Click Upload Certificate to upload a PEM file.
CA certificate
The public key certificate of the root certificate authority that issued the MKE server certificate. If you do not have a CA certificate, use the top-most intermediate certificate instead.
Click Upload CA Certificate to upload a PEM file.
Client CA
This field may contain one or more Root CA certificates that the MKE controller uses to verify that client certificates are issued by a trusted entity.
Click Upload CA Certificate to upload a PEM file.
Click Download MKE Server CA Certificate to download the certificate as a PEM file.
Note
MKE is automatically configured to trust its internal CAs, which issue client certificates as part of generated client bundles. However, you may supply MKE with additional custom root CA certificates using this field to enable MKE to trust the client certificates issued by your corporate or trusted third-party certificate authorities. Note that your custom root certificates will be appended to MKE internal root CA certificates.
Click Save.
After replacing the TLS certificates, your users will not be able to authenticate with their old client certificate bundles. Ask your users to access the MKE web UI and download new client certificate bundles.
Finally, Mirantis Secure Registry (MSR) deployments must be reconfigured to trust the new MKE TLS certificates. To do this, MSR 3.1.x users can refer to Add a custom TLS certificate, MSR 3.0.x userr to Add a custom TLS certificate, and MSR 2.9.x users to Add a custom TLS certificate.
Manage and deploy private images¶
Mirantis offers its own image registry, Mirantis Secure Registry (MSR), which you can use to store and manage the images that you deploy to your cluster. This topic describes how to use MKE to push the official WordPress image to MSR and later deploy that image to your cluster.
To create an MSR image repository:
Log in to the MKE web UI.
From the left-side navigation panel, navigate to <user name> > Admin Settings > Mirantis Secure Registry.
In the Installed MSRs section, capture the MSR URL for your cluster.
In a new browser tab, navigate to the MSR URL captured in the previous step.
From the left-side navigation panel, click Repositories.
Click New repository.
In the namespace field under New Repository, select the required namespace. The default namespace is your user name.
In the name field under New Repository, enter the name
wordpress.To create the repository, click Save.
To push an image to MSR:
In this example, you will pull the official WordPress image from Docker Hub, tag it, and push it to MSR. Once pushed to MSR, only authorized users will be able to make changes to the image. Pushing to MSR requires CLI access to a licensed MSR installation.
Pull the public WordPress image from Docker Hub:
docker pull wordpress
Tag the image, using the IP address or DNS name of your MSR instance. For example:
docker tag wordpress:latest <msr-url>:<port>/<namespace>/wordpress:latest
Log in to an MKE manager node.
Push the tagged image to MSR:
docker image push <msr-url>:<port>/admin/wordpress:latest
Verify that the image is stored in your MSR repository:
Log in to the MSR web UI.
In the left-side navigation panel, click Repositories.
Click admin/wordpress to open the repo.
Click the Tags tab to view the stored images.
Verify that the
latesttag is present.
To deploy the private image to MKE:
Log in to the MKE web UI.
In the left-side navigation panel, click Kubernetes.
Click Create to open the Create Kubernetes Object page.
In the Namespace dropdown, select default.
In the Object YAML editor, paste the following Deployment object YAML:
apiVersion: apps/v1 kind: Deployment metadata: name: wordpress-deployment spec: selector: matchLabels: app: wordpress replicas: 2 template: metadata: labels: app: wordpress spec: containers: - name: wordpress image: 52.10.217.20:444/admin/wordpress:latest ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: wordpress-service labels: app: wordpress spec: type: NodePort ports: - port: 80 nodePort: 32768 selector: app: wordpress
The Deployment object YAML specifies your MSR image in the Pod template spec:
image: <msr-url>:<port>/admin/wordpress:latest. Also, the YAML file defines aNodePortservice that exposes the WordPress application so that it is accessible from outside the cluster.Click Create. Creating the new Kubernetes objects will open the Controllers page.
After a few seconds, verify that wordpress-deployment has a green status icon and is thus successfully deployed.
Set the node orchestrator¶
When you add a node to your cluster, by default its workloads are managed by Swarm. Changing the default orchestrator does not affect existing nodes in the cluster. You can also change the orchestrator type for individual nodes in the cluster.
Select the node orchestrator¶
The workloads on your cluster can be scheduled by Kubernetes, Swarm, or a combination of the two. If you choose to run a mixed cluster, be aware that different orchestrators are not aware of each other, and thus there is no coordination between them.
Mirantis recommends that you decide which orchestrator you will use when initially setting up your cluster. Once you start deploying workloads, avoid changing the orchestrator setting. If you do change the node orchestrator, your workloads will be evicted and you will need to deploy them again using the new orchestrator.
Caution
When you promote a worker node to be a manager, its orchestrator type
automatically changes to Mixed. If you later demote that node to be
a worker, its orchestrator type remains as Mixed.
Note
The default behavior for Mirantis Secure Registry (MSR) nodes is to run in
the Mixed orchestration mode. If you change the MSR orchestrator type to
Swarm or Kubernetes only, reconciliation will revert the node back to the
Mixed mode.
Changing a node orchestrator¶
When you change the node orchestrator, existing workloads are evicted and they are not automatically migrated to the new orchestrator. You must manually migrate them to the new orchestrator. For example, if you deploy WordPress on Swarm, and you change the node orchestrator to Kubernetes, MKE does not migrate the workload, and WordPress continues running on Swarm. You must manually migrate your WordPress deployment to Kubernetes.
The following table summarizes the results of changing a node orchestrator.
Workload |
Orchestrator-related change |
|---|---|
Containers |
Containers continue running on the node. |
Docker service |
The node is drained and tasks are rescheduled to another node. |
Pods and other imperative resources |
Imperative resources continue running on the node. |
Deployments and other declarative resources |
New declarative resources will not be scheduled on the node and existing ones will be rescheduled at a time that can vary based on resource details. |
If a node is running containers and you change the node to Kubernetes,
the containers will continue running and Kubernetes will not be aware of
them. This is functionally the same as running the node in the Mixed mode.
Warning
The Mixed mode is not intended for production use and it may impact
the existing workloads on the node.
This is because the two orchestrator types have different views of the node resources and they are not aware of the other orchestrator resources. One orchestrator can schedule a workload without knowing that the node resources are already committed to another workload that was scheduled by the other orchestrator. When this happens, the node can run out of memory or other resources.
Mirantis strongly recommends against using the Mixed mode in production
environments.
Change the node orchestrator¶
This topic describes how to set the default orchestrator and change the orchestrator for individual nodes.
Set the default orchestrator¶
To set the default orchestrator using the MKE web UI:
Log in to the MKE web UI as an administrator.
In the left-side navigation panel, navigate to <user name> > Admin Settings > Orchestration.
Under Scheduler, select the required default orchestrator.
Click Save.
New workloads will now be scheduled by the specified orchestrator type. Existing nodes in the cluster are not affected.
Once a node is joined to the cluster, you can change the orchestrator that schedules its workloads.
To set the default orchestrator using the MKE configuration file:
Obtain the current MKE configuration file for your cluster.
Set
default_node_orchestratorto"swarm"or"kubernetes".Upload the new MKE configuration file. Be aware that this will require a wait time of approximately five minutes.
Change the node orchestrator¶
To change the node orchestrator using the MKE web UI:
Log in to the MKE web UI as an administrator.
From the left-side navigation panel, navigate to Shared Resources > Nodes.
Click the node that you want to assign to a different orchestrator.
In the upper right, click the Edit Node icon.
In the Details pane, in the Role section under ORCHESTRATOR TYPE, select either Swarm, Kubernetes, or Mixed.
Warning
Mirantis strongly recommends against using the
Mixedmode in production environments.Click Save to assign the node to the selected orchestrator.
To change the node orchestrator using the CLI:
Set the orchestrator on a node by assigning the orchestrator labels,
com.docker.ucp.orchestrator.swarm or
com.docker.ucp.orchestrator.kubernetes to true.
Change the node orchestrator. Select from the following options:
Schedule Swarm workloads on a node:
docker node update --label-add com.docker.ucp.orchestrator.swarm=true <node-id>
Schedule Kubernetes workloads on a node:
docker node update --label-add com.docker.ucp.orchestrator.kubernetes=true <node-id>
Schedule both Kubernetes and Swarm workloads on a node:
docker node update --label-add com.docker.ucp.orchestrator.swarm=true <node-id> docker node update --label-add com.docker.ucp.orchestrator.kubernetes=true <node-id>
Warning
Mirantis strongly recommends against using the
Mixedmode in production environments.Change the orchestrator type for a node from Swarm to Kubernetes:
docker node update --label-add com.docker.ucp.orchestrator.kubernetes=true <node-id> docker node update --label-rm com.docker.ucp.orchestrator.swarm <node-id>
Change the orchestrator type for a node from Kubernetes to Swarm:
docker node update --label-add com.docker.ucp.orchestrator.swarm=true <node-id> docker node update --label-rm com.docker.ucp.orchestrator.kubernetes <node-id>
Note
You must first add the target orchestrator label and then remove the old orchestrator label. Doing this in the reverse order can fail to change the orchestrator.
Verify the value of the orchestrator label by inspecting the node:
docker node inspect <node-id> | grep -i orchestrator
Example output:
"com.docker.ucp.orchestrator.kubernetes": "true"
Important
The com.docker.ucp.orchestrator label is not displayed in the MKE web UI
Labels list, which presents in the Overview pane for
each node.
View Kubernetes objects in a namespace¶
MKE administrators can filter the view of Kubernetes objects by the namespace that the objects are assigned to, specifying a single namespace or all available namespaces. This topic describes how to deploy services to two newly created namespaces and then view those services, filtered by namespace.
To create two namespaces:
Log in to the MKE web UI as an administrator.
From the left-side navigation panel, click Kubernetes.
Click Create to open the Create Kubernetes Object page.
Leave the Namespace drop-down blank.
In the Object YAML editor, paste the following YAML code:
apiVersion: v1 kind: Namespace metadata: name: blue --- apiVersion: v1 kind: Namespace metadata: name: green
Click Create to create the
blueandgreennamespaces.
To deploy services:
Create a
NodePortservice in thebluenamespace:From the left-side navigation panel, navigate to Kubernetes > Create.
In the Namespace drop-down, select blue.
In the Object YAML editor, paste the following YAML code:
apiVersion: v1 kind: Service metadata: name: app-service-blue labels: app: app-blue spec: type: NodePort ports: - port: 80 nodePort: 32768 selector: app: app-blue
Click Create to deploy the service in the
bluenamespace.
Create a
NodePortservice in thegreennamespace:From the left-side navigation panel, navigate to Kubernetes > Create.
In the Namespace drop-down, select green.
In the Object YAML editor, paste the following YAML code:
apiVersion: v1 kind: Service metadata: name: app-service-green labels: app: app-green spec: type: NodePort ports: - port: 80 nodePort: 32769 selector: app: app-green
Click Create to deploy the service in the
greennamespace.
To view the newly created services:
In the left-side navigation panel, click Namespaces.
In the upper-right corner, click the Set context for all namespaces toggle. The indicator in the left-side navigation panel under Namespaces changes to All Namespaces.
Click Services to view your services.
Filter the view by namespace:
In the left-side navigation panel, click Namespaces.
Hover over the blue namespace and click Set Context. The indicator in the left-side navigation panel under Namespaces changes to blue.
Click Services to view the app-service-blue service. Note that the app-service-green service does not display.
Perform the forgoing steps on the green namespace to view only the services deployed in the green namespace.
Join Nodes¶
Set up high availability¶
MKE is designed to facilitate high availability (HA). You can join multiple manager nodes to the cluster, so that if one manager node fails, another one can automatically take its place without impacting the cluster.
Including multiple manager nodes in your cluster allows you to handle manager node failures and load-balance user requests across all manager nodes.
The following table exhibits the relationship between the number of manager nodes used and the number of faults that your cluster can tolerate:
Manager nodes |
Failures tolerated |
|---|---|
1 |
0 |
3 |
1 |
5 |
2 |
For deployment into product environments, follow these best practices:
For HA with minimal network overhead, Mirantis recommends using three manager nodes and a maximum of five. Adding more manager nodes than this can lead to performance degradation, as configuration changes must be replicated across all manager nodes.
You should bring failed manager nodes back online as soon as possible, as each failed manager node decreases the number of failures that your cluster can tolerate.
You should distribute your manager nodes across different availability zones. This way your cluster can continue working even if an entire availability zone goes down.
Join Linux nodes¶
MKE allows you to add or remove nodes from your cluster as your needs change over time.
Because MKE leverages the clustering functionality provided by Mirantis Container Runtime (MCR), you use the docker swarm join command to add more nodes to your cluster. When you join a new node, MCR services start running on the node automatically.
You can add both Linux manager and worker nodes to your cluster.
Join a node to the cluster¶
Important
Prior to adding a node that was previously a part of the same MKE cluster or a different one, you must run the following command to remove any stale MKE volumes:
docker volume rm `docker volume list --filter name=ucp* -q`
Next, run the following command to verify the removal of the stale volumes:
docker volume list --filter name=ucp*
Log in to the MKE web UI.
In the left-side navigation panel, navigate to Shared Resources > Nodes.
Click Add Node.
Select Linux for the node type.
Select either Manager or Worker, as required.
Optional. Select Use a custom listen address to specify the address and port where the new node listens for inbound cluster management traffic.
Optional. Select Use a custom advertise address to specify the IP address that is advertised to all members of the cluster for API access.
Copy the displayed command, which looks similar to the following:
docker swarm join --token <token> <mke-node-ip>
Use SSH to log in to the host that you want to join to the cluster.
Run the docker swarm join command captured previously.
The node will display in the Shared Resources > Nodes page.
Pause or drain a node¶
Note
You can pause or drain a node only with swarm workloads.
You can configure the availability of a node so that it is in one of the following three states:
- Active
The node can receive and execute tasks.
- Paused
The node continues running existing tasks, but does not receive new tasks.
- Drained
Existing tasks are stopped, while replica tasks are launched in active nodes. The node does not receive new tasks.
To pause or drain a node:
Log in to the MKE web UI.
In the left-side navigation panel, navigate to Shared Resources > Nodes and select the required node.
In the Details pane, click Configure and select Details to open the Edit Node page.
In the upper right, select the Edit Node icon.
In the Availability section, click Active, Pause, or Drain.
Click Save.
Promote or demote a node¶
You can promote worker nodes to managers to make MKE fault tolerant. You can also demote a manager node into a worker node.
Log in to the MKE web UI.
In the left-side navigation panel, navigate to Shared Resources > Nodes and select the required node.
In the upper right, select the Edit Node icon.
In the Role section, click Manager or Worker.
Click Save and wait until the operation completes.
Navigate to Shared Resources > Nodes and verify the new node role.
Note
If you are load balancing user requests to MKE across multiple manager nodes, you must remove these nodes from the load-balancing pool when demoting them to workers.
Remove a node from the cluster¶
To remove an inaccessible worker node or one that is down:
Log in to the MKE web UI.
In the left-side navigation panel, navigate to Shared Resources > Nodes and select the required node.
In the upper right, select the vertical ellipsis and click Remove.
When prompted, click Confirm.
To remove an inactive worker node:
Log in to the MKE web UI.
In the left-side navigation panel, navigate to Shared Resources > Nodes and select the required node.
Drain the node, to ensure that the workload is scheduled to another node.
Click the vertical ellipsis in the upper right and select Force Remove.
When prompted, click Confirm.
To remove a manager node:
Verify that all nodes in the cluster are healthy.
Warning
Do not remove the manager node if all nodes are not healthy.
Remove the newly-demoted worker from the cluster, as described in the preceding steps.
Join Windows worker nodes¶
MKE allows you to add or remove nodes from your cluster as your needs change over time.
Because MKE leverages the clustering functionality provided by Mirantis Container Runtime (MCR), you use the docker swarm join command to add more nodes to your cluster. When you join a new node, MCR services start running on the node automatically.
MKE supports running worker nodes on Windows Server. You must run all manager nodes on Linux.
Windows nodes limitations¶
The following features are not yet supported using Windows Server:
Category |
Feature |
|---|---|
Networking |
Encrypted networks are not supported. If you have upgraded from a
previous version of MKE, you will need to recreate an unencrypted
version of the |
Secrets |
|
Mounts |
On Windows, Docker cannot listen on a Unix socket. Use TCP or a named pipe instead. |
Configure the Docker daemon for Windows nodes¶
Note
If the cluster is deployed in a site that is offline, sideload MKE images onto the Windows Server nodes. For more information, refer to Install MKE offline.
On a manager node, list the images that are required on Windows nodes:
docker container run --rm -v /var/run/docker.sock:/var/run/docker.sock mirantis/ucp:3.6.19 images --list --enable-windows
Example output:
mirantis/ucp-agent-win:3.6.19 mirantis/ucp-dsinfo-win:3.6.19
Pull the required images. For example:
docker image pull mirantis/ucp-agent-win:3.6.19 docker image pull mirantis/ucp-dsinfo-win:3.6.19
Join Windows nodes to the cluster¶
Log in to the MKE web UI as an administrator.
In the left-side navigation panel, navigate to Shared Resources > Nodes.
Click Add Node.
Select Windows for the node type.
Optional. Select Use a custom listen address to specify the address and port where the new node listens for inbound cluster management traffic.
Optional. Select Use a custom advertise address to specify the IP address that is advertised to all members of the cluster for API access.
Copy the displayed command, which looks similar to the following:
docker swarm join --token <token> <mke-worker-ip>
Alternatively, you can use the command line to obtain the join token. Using your MKE client bundle, run:
docker swarm join-token worker
Run the
docker swarm joincommand captured in the previous step on each instance of Windows Server that will be a worker node.
Use a load balancer¶
After joining multiple manager nodes for high availability (HA), you can configure your own load balancer to balance user requests across all manager nodes.
Use of a load balancer allows users to access MKE using a centralized domain name. The load balancer can detect when a manager node fails and stop forwarding requests to that node, so that users are unaffected by the failure.
Configure load balancing on MKE¶
Because MKE uses TLS, do the following when configuring your load balancer:
Load-balance TCP traffic on ports
443and6443.Do not terminate HTTPS connections.
On each manager node, use the
/_pingendpoint to verify whether the node is healthy and whether or not it should remain in the load balancing pool.
Use the following examples to configure your load balancer for MKE:
user nginx; worker_processes 1; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 1024; } stream { upstream ucp_443 { server <UCP_MANAGER_1_IP>:443 max_fails=2 fail_timeout=30s; server <UCP_MANAGER_2_IP>:443 max_fails=2 fail_timeout=30s; server <UCP_MANAGER_N_IP>:443 max_fails=2 fail_timeout=30s; } server { listen 443; proxy_pass ucp_443; } upstream ucp_6443 { server <UCP_MANAGER_1_IP>:6443 max_fails=2 fail_timeout=30s; server <UCP_MANAGER_2_IP>:6443 max_fails=2 fail_timeout=30s; server <UCP_MANAGER_N_IP>:6443 max_fails=2 fail_timeout=30s; } server { listen 6443; proxy_pass ucp_6443; } }
global log /dev/log local0 log /dev/log local1 notice defaults mode tcp option dontlognull timeout connect 5s timeout client 50s timeout server 50s timeout tunnel 1h timeout client-fin 50s ### frontends # Optional HAProxy Stats Page accessible at http://<host-ip>:8181/haproxy?stats frontend ucp_stats mode http bind 0.0.0.0:8181 default_backend ucp_stats frontend ucp_443 mode tcp bind 0.0.0.0:443 default_backend ucp_upstream_servers_443 frontend ucp_6443 mode tcp bind 0.0.0.0:6443 default_backend ucp_upstream_servers_6443 ### backends backend ucp_stats mode http option httplog stats enable stats admin if TRUE stats refresh 5m backend ucp_upstream_servers_443 mode tcp option httpchk GET /_ping HTTP/1.1\r\nHost:\ <UCP_FQDN> server node01 <UCP_MANAGER_1_IP>:443 weight 100 check check-ssl verify none server node02 <UCP_MANAGER_2_IP>:443 weight 100 check check-ssl verify none server node03 <UCP_MANAGER_N_IP>:443 weight 100 check check-ssl verify none backend ucp_upstream_servers_6443 mode tcp option httpchk GET /_ping HTTP/1.1\r\nHost:\ <UCP_FQDN> server node01 <UCP_MANAGER_1_IP>:6443 weight 100 check check-ssl verify none server node02 <UCP_MANAGER_2_IP>:6443 weight 100 check check-ssl verify none server node03 <UCP_MANAGER_N_IP>:6443 weight 100 check check-ssl verify none
{ "Subnets": [ "subnet-XXXXXXXX", "subnet-YYYYYYYY", "subnet-ZZZZZZZZ" ], "CanonicalHostedZoneNameID": "XXXXXXXXXXX", "CanonicalHostedZoneName": "XXXXXXXXX.us-west-XXX.elb.amazonaws.com", "ListenerDescriptions": [ { "Listener": { "InstancePort": 443, "LoadBalancerPort": 443, "Protocol": "TCP", "InstanceProtocol": "TCP" }, "PolicyNames": [] }, { "Listener": { "InstancePort": 6443, "LoadBalancerPort": 6443, "Protocol": "TCP", "InstanceProtocol": "TCP" }, "PolicyNames": [] } ], "HealthCheck": { "HealthyThreshold": 2, "Interval": 10, "Target": "HTTPS:443/_ping", "Timeout": 2, "UnhealthyThreshold": 4 }, "VPCId": "vpc-XXXXXX", "BackendServerDescriptions": [], "Instances": [ { "InstanceId": "i-XXXXXXXXX" }, { "InstanceId": "i-XXXXXXXXX" }, { "InstanceId": "i-XXXXXXXXX" } ], "DNSName": "XXXXXXXXXXXX.us-west-2.elb.amazonaws.com", "SecurityGroups": [ "sg-XXXXXXXXX" ], "Policies": { "LBCookieStickinessPolicies": [], "AppCookieStickinessPolicies": [], "OtherPolicies": [] }, "LoadBalancerName": "ELB-UCP", "CreatedTime": "2017-02-13T21:40:15.400Z", "AvailabilityZones": [ "us-west-2c", "us-west-2a", "us-west-2b" ], "Scheme": "internet-facing", "SourceSecurityGroup": { "OwnerAlias": "XXXXXXXXXXXX", "GroupName": "XXXXXXXXXXXX" } }
Create either the
nginx.conforhaproxy.cfgfile, as required.For instruction on deploying with AWS LB, refer to Getting Started with Network Load Balancers in the AWS documentation.
Deploy the load balancer:
docker run --detach \ --name ucp-lb \ --restart=unless-stopped \ --publish 443:443 \ --publish 6443:6443 \ --volume ${PWD}/nginx.conf:/etc/nginx/nginx.conf:ro \ nginx:stable-alpine
docker run --detach \ --name ucp-lb \ --publish 443:443 \ --publish 6443:6443 \ --publish 8181:8181 \ --restart=unless-stopped \ --volume ${PWD}/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro \ haproxy:1.7-alpine haproxy -d -f /usr/local/etc/haproxy/haproxy.cfg
Load balancing MKE and MSR together¶
By default, both MKE and Mirantis Secure Registry (MSR) use port 443. If you plan to deploy MKE and MSR together, your load balancer must distinguish traffic between the two by IP address or port number.
If you want MKE and MSR both to use port 443, then you must either use separate load balancers for each or use two virtual IPs. Otherwise, you must configure your load balancer to expose MKE or MSR on a port other than 443.
Use two-factor authentication¶
Two-factor authentication (2FA) adds an extra layer of security when logging in to the MKE web UI. Once enabled, 2FA requires the user to submit an additional authentication code generated on a separate mobile device along with their user name and password at login.
Configure 2FA¶
MKE 2FA requires the use of a time-based one-time password (TOTP) application installed on a mobile device to generate a time-based authentication code for each login to the MKE web UI. Examples of such applications include 1Password, Authy, and LastPass Authenticator.
To configure 2FA:
Install a TOTP application to your mobile device.
In the MKE web UI, navigate to My Profile > Security.
Toggle the Two-factor authentication control to enabled.
Open the TOTP application and scan the offered QR code. The device will display a six-digit code.
Enter the six-digit code in the offered field and click Register. The TOTP application will save your MKE account.
Important
A set of recovery codes displays in the MKE web UI when two-factor authentication is enabled. Save these codes in a safe location, as they can be used to access the MKE web UI if for any reason the configured mobile device becomes unavailable. Refer to Recover 2FA for details.
Access MKE using 2FA¶
Once 2FA is enabled, you will need to provide an authentication code each time you log in to the MKE web UI. Typically, the TOTP application installed on your mobile device generates the code and refreshes it every 30 seconds.
Access the MKE web UI with 2FA enabled:
In the MKE web UI, click Sign in. The Sign in page will display.
Enter a valid user name and password.
Access the MKE code in the TOTP application on your mobile device.
Enter the current code in the 2FA Code field in the MKE web UI.
Note
Multiple authentication failures may indicate a lack of synchronization between the mobile device clock and the mobile provider.
Disable 2FA¶
Mirantis strongly recommends using 2FA to secure MKE accounts. If you need to temporarily disable 2FA, re-enable it as soon as possible.
To disable 2FA:
In the MKE web UI, navigate to My Profile > Security.
Toggle the Two-factor authentication control to disabled.
Recover 2FA¶
If the mobile device with authentication codes is unavailable, you can re-access MKE using any of the recovery codes that display in the MKE web UI when 2FA is first enabled.
To recover 2FA:
Enter one of the recovery codes when prompted for the two-factor authentication code upon login to the MKE web UI.
Navigate to My Profile > Security.
Disable 2FA and then re-enable it.
Open the TOTP application and scan the offered QR code. The device will display a six-digit code.
Enter the six-digit code in the offered field and click Register. The TOTP application will save your MKE account.
If there are no recovery codes to draw from, ask your system administrator to disable 2FA in order to regain access to the MKE web UI. Once done, repeat the Configure 2FA procedure to reinstate 2FA protection.
MKE administrators are not able to re-enable 2FA for users.
Account lockout¶
You can configure MKE so that a user account is temporarily blocked from logging in following a series of unsuccessful login attempts. The account lockout feature only prevents log in attempts that are made using basic authorization or LDAP. Log in attempts using either SAML or OIDC do not trigger the account lockout feature. Admin accounts are never locked.
Account lockouts expire after a set amount of time, after which the affected user can log in as normal. Subsequent log in attempts on a locked account do not extend the lockout period. Log in attempts against a locked account always cause a standard incorrect credentials error, providing no indication to the user that the account is locked. Only MKE admins can see account lockout status.
Configure account lockout functionality¶
Obtain the current MKE configuration file for your cluster.
Set the following parameters in the
auth.account_locksection of the MKE configuration file:Set the value of
enabledtotrue.Set the value of
failureTriggersto the number of failed log in attempts that can be made before an account is locked.Set the value of
durationSecondsto the desired lockout duration. A value of0indicates that the account will remain locked until it is unlocked by an administrator.
Note
You can verify the lockout status of your organization accounts by issuing a
GET request to the /accounts endpoint.
Unlock an account¶
The account remains locked until the specified amount of time has elapsed. Otherwise, you must either have an administrator unlock the account or globally disable the account lockout feature.
To unlock a locked account:
Log in to the MKE web UI as an administrator.
In the left-side navigation panel, navigate to Access Control > Users and select the user who is locked out of their account.
Click the gear icon in the upper right corner.
Navigate to the Security tab.
Note
An expired account lock only resets once a new log in attempt is made. Until such time, the account will present as locked to administrators.
Click the Unlock account button.
To globally disable the account lockout feature:
Obtain the current MKE configuration file for your cluster.
In the
auth.account_locksection of the MKE configuration file, set the value ofenabledtofalse.
Configure and use OpsCare¶
Any time there is an issue with your cluster, OpsCare routes notifications from your MKE deployment to Mirantis support engineers. These company personnel will then either directly resolve the problem or arrange to troubleshoot the matter with you.
For more information, refer to Mirantis OpsCare Plus, OpsCare & LabCare.
Configure OpsCare¶
To configure OpsCare you must first obtain a Salesforce username, password, and environment ID from your Mirantis Customer Success Manager. You then store these credentials as Swarm secrets using the following naming convention:
User name:
sfdc_opscare_api_usernamePassword:
sfdc_opscare_api_passwordEnvironment ID:
sfdc_environment_id
Note
Every cluster that uses OpsCare must have its own unique
sfdc_environment_id.OpsCare requires that MKE has access to
mirantis.my.salesforce.comon port 443.Any custom certificates in use must contain all of the manager node private IP addresses.
The provided Salesforce credentials are not associated with the Mirantis support portal login, but are for Opscare alerting only.
To configure OpsCare using the CLI:
Create secrets for your Salesforce login credentials:
printf "<username-obtained-from-csm>" | docker secret create sfdc_opscare_api_username - printf "<password-obtained-from-csm>" | docker secret create sfdc_opscare_api_password - printf "<environment-id-obtained-from-csm>" | docker secret create sfdc_environment_id -
Enable OpsCare:
MKE_USERNAME=<mke-username> MKE_PASSWORD=<mke-password> MKE_HOST=<mke-host> AUTHTOKEN=$(curl --silent --insecure --data "{\"username\":\"$MKE_USERNAME\",\"password\":\"$MKE_PASSWORD\"}" https://$MKE_HOST/auth/login | jq --raw-output .auth_token) curl --silent --insecure -X GET "https://$MKE_HOST/api/ucp/config-toml" -H "accept: application/toml" -H "Authorization: Bearer $AUTHTOKEN" > ucp-config.toml sed -i 's/ops_care = false/ops_care = true/' ucp-config.toml curl --silent --insecure -X PUT -H "accept: application/toml" -H "Authorization: Bearer $AUTHTOKEN" --upload-file './ucp-config.toml' https://$MKE_HOST/api/ucp/config-toml
To configure OpsCare using the MKE web UI:
Log in to the MKE web UI.
Using the left-side navigation panel, navigate to <username> > Admin Settings > Usage.
In the Salesforce Username field, enter your Salesforce user name.
Next, enter your Salesforce password and Salesforce environment ID.
Click Create Secrets.
Under OpsCare Settings, toggle the Enable OpsCare slider to the right.
Click Save.
Manage Salesforce alerts¶
OpsCare uses a predefined group of MKE alerts to notify your Customer Success Manager of problems with your deployment. This alerts group is identical to those seen in any MKE cluster that is provisioned by Mirantis Container Cloud. A single watchdog alert serves to verify the proper function of the OpsCare alert pipeline as a whole.
To verify that the OpsCare alerts are functioning properly:
Log in to Salesforce.
Navigate to Cases and verify that the watchdog alert is present. It presents as Watchdog alert. It is always firing.
Disable OpsCare¶
You must disable OpsCare before you can delete the three secrets in use.
To disable OpsCare:
Log in to the MKE web UI.
Using the left-side navigation panel, navigate to <username> > Admin Settings > Usage.
Toggle the Enable Ops Care slider to the left.
Alternatively, you can disable OpsCare by changing the ops_care entry in
the MKE configuration file to false.
Configure cluster and service networking in an existing cluster¶
On systems that use the managed CNI, you can switch existing clusters to either kube-proxy with ipvs proxier or eBPF mode.
MKE does not support switching kube-proxy in an existing cluster from ipvs proxier to iptables proxier, nor does it support disabling eBPF mode after it has been enabled. Using a CNI that supports both cluster and service networking requires that you disable kube-proxy.
Refer to Cluster and service networking options in the MKE Installation Guide for information on how to configure cluster and service networking at install time.
Caution
The configuration changes described here cannot be reversed. As such, Mirantis recommends that you make a cluster backup, drain your workloads, and take your cluster offline prior to performing any of these changes.
Caution
Swarm workloads that require the use of encrypted overlay networks must use iptables proxier. Be aware that the other networking options detailed here automatically disable Docker Swarm encrypted overlay networks.
To switch an existing cluster to kube-proxy with ipvs proxier while using the managed CNI:
Obtain the current MKE configuration file for your cluster.
Set
kube_proxy_modeto"ipvs".Upload the new MKE configuration file. Be aware that this will require a wait time of approximately five minutes.
Verify that the following values are set in your MKE configuration file:
unmanaged_cni = false calico_ebpf_enabled = false kube_default_drop_masq_bits = false kube_proxy_mode = "ipvs" kube_proxy_no_cleanup_on_start = false
Verify that the
ucp-kube-proxycontainer logs on all nodes contain the following:KUBE_PROXY_MODE (ipvs) CLEANUP_ON_START_DISABLED false Performing cleanup kube-proxy cleanup succeeded Actually starting kube-proxy....
Obtain the current MKE configuration file for your cluster.
Set
kube_proxy_no_cleanup_on_starttotrue.Upload the new MKE configuration file. Be aware that this will require a wait time of approximately five minutes.
Reboot all nodes.
Verify that the following values are set in your MKE configuration file and that your cluster is in a healthy state with all nodes ready:
unmanaged_cni = false calico_ebpf_enabled = false kube_default_drop_masq_bits = false kube_proxy_mode = "ipvs" kube_proxy_no_cleanup_on_start = true
Verify that the
ucp-kube-proxycontainer logs on all nodes contain the following:KUBE_PROXY_MODE (ipvs) CLEANUP_ON_START_DISABLED true Actually starting kube-proxy.... ..... I1111 02:41:05.559641 1 server_others.go:274] Using ipvs Proxier. W1111 02:41:05.559951 1 proxier.go:445] IPVS scheduler not specified, use rr by default
Optional. Configure the following ipvs-related parameters in the MKE configuration file (otherwise, MKE will use the Kubernetes default parameter settings):
ipvs_exclude_cidrs = ""ipvs_min_sync_period = ""ipvs_scheduler = ""ipvs_strict_arp = falseipvs_sync_period = ""ipvs_tcp_timeout = ""ipvs_tcpfin_timeout = ""ipvs_udp_timeout = ""
For more information on using these parameters, refer to kube-proxy in the Kubernetes documentation.
To switch an existing cluster to eBPF mode while using the managed CNI:
Verify that the prerequisites for eBPF use have been met, including kernel compatibility, for all Linux manager and worker nodes. Refer to the Calico documentation Enable the eBPF dataplane for more information.
Obtain the current MKE configuration file for your cluster.
Set
kube_default_drop_masq_bitstotrue.Upload the new MKE configuration file. Be aware that this will require a wait time of approximately five minutes.
Verify that the
ucp-kube-proxycontainer started on all nodes, that the kube-proxy cleanup took place, and thatucp-kube-proxylaunched kube-proxy.for cont in $(docker ps -a|rev | cut -d' ' -f 1 | rev|grep ucp-kube-proxy); \ do nodeName=$(echo $cont|cut -d '/' -f1); \ docker logs $cont 2>/dev/null|grep -q 'kube-proxy cleanup succeeded'; \ if [ $? -ne 0 ]; \ then echo $nodeName; \ fi; \ done|sort
Expected output in the
ucp-kube-proxylogs:KUBE_PROXY_MODE (iptables) CLEANUP_ON_START_DISABLED false Performing cleanup kube-proxy cleanup succeeded Actually starting kube-proxy....
Note
If the count returned by the command does not quickly converge at
0, check theucp-kube-proxylogs on the nodes where either of the following took place:The
ucp-kube-proxycontainer did not launch.The kube-proxy cleanup did not happen.
Reboot all nodes.
Obtain the current MKE configuration file for your cluster.
Verify that the following values are set in your MKE configuration file:
unmanaged_cni = false calico_ebpf_enabled = false kube_default_drop_masq_bits = true kube_proxy_mode = "iptables" kube_proxy_no_cleanup_on_start = false
Verify that the
ucp-kube-proxycontainer logs on all nodes contain the following:KUBE_PROXY_MODE (iptables) CLEANUP_ON_START_DISABLED false Performing cleanup .... kube-proxy cleanup succeeded Actually starting kube-proxy.... .... I1111 03:29:25.048458 1 server_others.go:212] Using iptables Proxier.
Set
kube_proxy_modeto"disabled".Set
calico_ebpf_enabledtotrue.Upload the new MKE configuration file. Be aware that this will require a wait time of approximately five minutes.
Verify that the
ucp-kube-proxycontainer started on all nodes, that the kube-proxy cleanup took place, and thatucp-kube-proxydid not launch kube-proxy.for cont in $(docker ps -a|rev | cut -d' ' -f 1 | rev|grep ucp-kube-proxy); \ do nodeName=$(echo $cont|cut -d '/' -f1); \ docker logs $cont 2>/dev/null|grep -q 'Sleeping forever'; \ if [ $? -ne 0 ]; \ then echo $nodeName; \ fi; \ done|sort
Expected output in the
ucp-kube-proxylogs:KUBE_PROXY_MODE (disabled) CLEANUP_ON_START_DISABLED false Performing cleanup kube-proxy cleanup succeeded Sleeping forever....
Note
If the count returned by the command does not quickly converge at
0, check theucp-kube-proxylogs on the nodes where either of the following took place:The
ucp-kube-proxycontainer did not launch.The
ucp-kube-proxycontainer launched kube-proxy.
Obtain the current MKE configuration file for your cluster.
Verify that the following values are set in your MKE configuration file:
unmanaged_cni = false calico_ebpf_enabled = true kube_default_drop_masq_bits = true kube_proxy_mode = "disabled" kube_proxy_no_cleanup_on_start = false
Set
kube_proxy_no_cleanup_on_starttotrue.Upload the new MKE configuration file. Be aware that this will require a wait time of approximately five minutes.
Verify that the following values are set in your MKE configuration file and that your cluster is in a healthy state with all nodes ready:
unmanaged_cni = false calico_ebpf_enabled = true kube_default_drop_masq_bits = true kube_proxy_mode = "disabled" kube_proxy_no_cleanup_on_start = true
Verify that eBPF mode is operational by confirming the presence of the following lines in the
ucp-kube-proxycontainer logs:KUBE_PROXY_MODE (disabled) CLEANUP_ON_START_DISABLED true "Sleeping forever...."
Verify that you can SSH into all nodes.
Schedule image pruning¶
MKE administrators can schedule the cleanup of unused images, whitelisting which images to keep. To determine which images will be removed, they can perform a dry run prior to setting the image-pruning schedule.
Schedule image pruning using the CLI¶
To perform a dry run without whitelisting any images:
Perform a dry run to determine which images will be pruned:
AUTHTOKEN=$(curl --silent --insecure --data '{"username":"<username>","password":"<password>"}' https://MKE_HOST/auth/login | jq --raw-output .auth_token)
curl --insecure -H "Authorization: Bearer $AUTHTOKEN" https://MKE_HOST/api/ucp/images/prune/dry
Example response:
[
{
"Containers":-1,
"Created":1647029986,
"Id":"sha256:2fb6fc2d97e10c79983aa10e013824cc7fc8bae50630e32159821197dda95fe3",
"Labels":null,
"ParentId":"",
"RepoDigests":[
"busybox@sha256:caa382c432891547782ce7140fb3b7304613d3b0438834dce1cad68896ab110a"
],
"RepoTags":[
"busybox:latest"
],
"SharedSize":-1,
"Size":1239748,
"VirtualSize":1239748
}
]
To perform a dry run with whitelisted images:
Obtain the current MKE configuration file for your cluster.
Whitelist the images that should not be removed.
Note
Where possible, use the image ID to specify the image rather than the image name.
For example:
[[cluster_config.image_prune_whitelist]] key = "label" value = "<label-value>" [[cluster_config.image_prune_whitelist]] key = "before" value = "<image-id>"
Refer to cluster_config.image_prune_whitelist (optional) for more information.
Perform a dry run to determine which images will be pruned:
AUTHTOKEN=$(curl --silent --insecure --data '{"username":"<username>","password":"<password>"}' https://MKE_HOST/auth/login | jq --raw-output .auth_token) curl --insecure -H "Authorization: Bearer $AUTHTOKEN" https://MKE_HOST/api/ucp/images/prune/dry
Example response:
[ { "Containers":-1, "Created":1647029986, "Id":"sha256:2fb6fc2d97e10c79983aa10e013824cc7fc8bae50630e32159821197dda95fe3", "Labels":null, "ParentId":"", "RepoDigests":[ "busybox@sha256:caa382c432891547782ce7140fb3b7304613d3b0438834dce1cad68896ab110a" ], "RepoTags":[ "busybox:latest" ], "SharedSize":-1, "Size":1239748, "VirtualSize":1239748 } ]
To schedule image pruning:
Obtain the current MKE configuration file for your cluster.
Optional. Whitelist the images that should not be removed, if you have not already done so.
Note
Where possible, use the image ID to specify the image rather than the image name.
For example:
[[cluster_config.image_prune_whitelist]] key = "label" value = "<label-value>" [[cluster_config.image_prune_whitelist]] key = "before" value = "<image-id>"
Refer to cluster_config.image_prune_whitelist (optional) for more information.
Set the value of
image_prune_scheduleto the desired cron schedule. Refer to cluster_config table (required) for more information.The following example schedules image pruning for every day at midnight:
[cluster_config] image_prune_schedule = "0 0 0 * * *"
Schedule image pruning using the MKE web UI¶
Log in to the MKE web UI as an administrator.
From the left-side navigation panel, navigate to <user name> > Admin Settings > Tuning and scroll to Image pruning config.
Enter the desired pruning schedule.
Optional. Select the desired whitelist rules.
Optional. Test your image pruning configuration by clicking Start a dry run under Test configuration.
Manage etcd¶
etcd is a consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It handles leader elections during network partitions and can tolerate machine failure, even in the leader node.
For MKE, etcd serves as the Kubernetes backing store for all cluster data, with an etcd replica deployed on each MKE manager node. This is a primary reason why Mirantis recommends that you deploy an odd number of MKE manager nodes, as etcd uses the Raft consensus algorithm and thus requires that a quorum of nodes agree on any updates to the cluster state.
Configure etcd storage quota¶
You can control the etcd distributed key-value storage quota using the
etcd_storage_quota parameter in the MKE configuration file. By default, the value of the parameter is 2GB.
Note
MKE may cease to function if etcd exceeds the set quota. As such, Mirantis recommends either cleaning etcd or increasing the key-value storage quota size when the size of the database quota approaches 85%, the point at which MKE will start presenting warning banners.
For information on how to adjust the parameter, refer to Configure an MKE cluster.
Apply etcd defragmentation¶
The etcd distributed key-value store retains a history of its keyspace. That history is set for compaction following a specified number of revisions, however it only releases the used space back to the host filesystem following defragmentation. For more information, refer to the etcd documentation.
With MKE you can defragment the etcd cluster while avoiding cluster outages. To do this, you apply defragmentation to etcd members one at a time. MKE will defragment the current etcd leader last, to prevent the triggering of multiple leader elections.
Important
In a High Availability (HA) cluster, the defragmentation process subtly affects cluster dynamics, because when a node undergoes defragmentation it temporarily leaves the pool of active nodes. This subsequent reduction in the active node count results in a proportional increase of the load on the remaining nodes, which can lead to performance degradation if the remaining nodes do not have the capacity to handle the additional load. In addition, at the end of the process, when the leader node is undergoing defragmentation, there is a brief period during which cluster write operations do not take place. This pause occurs when the system initiates and completes the leader election process, and though it is automated and brief it does result in a momentary write block on the cluster.
Taking the described factors into account, Mirantis recommends taking a cautious scheduling approach in defragmenting HA clusters. Ideally, the defragmentation should occur during planned maintenance windows rather than relying on a recurring cron job, as during such periods you can closely monitor potential impacts on performance and availability and mitigate as necessary.
To defragment the etcd cluster:
Trigger the etcd cluster defragmentation by issuing a POST to the
https://MKE_HOST/api/ucp/etcd/defragendpoint.You can specify two parameters:
timeoutSecondsSets how long MKE waits for each member to finish defragmentation. Default: 60 seconds. MKE will cancel the defragmentation if the timeout occurs before the member defragmentation completes.
pauseSecondsSets how long MKE waits between each member defragmentation. Default: 60 seconds.
Mirantis recommends that you adjust these parameters based on the size of the etcd database and the amount of time that has elapsed since the last defragmentation.
Example command:
AUTHTOKEN=$(curl --silent --insecure --data '{"username":"<username>","password":"<password>"}' https://MKE_HOST/auth/login | jq --raw-output .auth_token) curl --insecure -H "Authorization: Bearer $AUTHTOKEN" https://MKE_HOST/api/ucp/etcd/defrag --data '{"timeoutSeconds": 60, "pauseSeconds": 60}'
Example response:
"Cluster Defragmentation Initiated"Review the state of individual etcd cluster members and the state of the cluster defragmentation by running the following command:
AUTHTOKEN=$(curl --silent --insecure --data '{"username":"<username>","password":"<password>"}' https://MKE_HOST/auth/login | jq --raw-output .auth_token) curl --insecure -H "Authorization: Bearer $AUTHTOKEN" https://MKE_HOST/api/ucp/etcd/info
Example output:
{ "DefragInProgress": true, "DefragResult": "Cluster Defrag Initiated", "MemberInfo": [ { "MemberID": 5051939019959384922, "Endpoint": "https://172.31.21.33:12379", "EtcdVersion": "3.4.16", "DbSize": "2 MB", "IsLeader": true, "Alarms": null }, { "MemberID": 10749614093923491478, "Endpoint": "https://172.31.30.179:12379", "EtcdVersion": "3.4.16", "DbSize": "2 MB", "IsLeader": false, "Alarms": null }, { "MemberID": 7837950661722744517, "Endpoint": "https://172.31.30.44:12379", "EtcdVersion": "3.4.16", "DbSize": "2 MB", "IsLeader": false, "Alarms": null } ] }
You can monitor this endpoint until the defragmentation is complete. The information is also available in the
ucp-controllerlogs.
To manually remove the etcd defragmentation lock file:
To maintain etcd cluster availability, MKE uses a lock file that prevents multiple defragmentations from being simultaneously implemented. MKE removes the lock file at the conclusion of defragmentation, however you can manually remove it as necessary.
Manually remove the lock file by running the following command:
docker exec ucp-controller rm /var/lock/etcd-defrag
etcd alarms response¶
Available since MKE 3.6.10
etcd issues alarms to indicate problems that need to be quickly addressed to ensure uninterrupted function.
NOSPACE alarm¶
A NOSPACE alarm is issued in the event that etcd runs low on storage space, to
protect the cluster from further writes. Once this low storage space state is
reached, etcd will respond to all write requests with the
mvcc: database space exceeded error message until the issue is rectified.
When MKE detects the NOSPACE alarm condition, it displays a critical banner to inform administrators. In addition, MKE restarts etcd with an increased value for the etcd datastore quota, thus allowing administrators to resolve the NOSPACE alarm without interference.
To resolve the NOSPACE alarm:
Identify what data occupies most of the storage space. Be aware that in MKE the recommended
etcdctlcommands must be run in theucp-kvcontainer, the instruction for which is available in Troubleshoot the etcd key-value store with the CLI.If a bug-ridden appliction is the cause of the unexpected use of storage space, stop that application.
Manually delete the unused data from etcd, if possible.
If necessary, increase the
etcd_storage_quotasetting in thecluster_configtable of the MKE configuration file.
Note
Contact Mirantis Support if you require assistance in resolving the etcd NOSPACE alarm.
CORRUPT alarm¶
The CORRUPT alarm is issued when a cluster corruption is detected by etcd. MKE cluster administrators are informed of the condition by way of a critical banner. To resolve such an issue, contact Mirantis Support and refer to the official etcd documentation regarding data corruption recovery.
Operate a hybrid Windows cluster¶
Hybrid Windows clusters concurrently run two versions of Windows Server, with one version deployed on one set of nodes and the second version deployed on a different set of nodes. The Windows versions that MKE supports are:
Windows Server 2019, build number 10.0.17763
Windows Server 2022, build number 10.0.20348
For more information on Windows releases and build numbers, refer to Windows container version compatibility.
To learn how to upgrade to Windows Server 2022, refer to Upgrade nodes to Windows Server 2022.
Limitations¶
A Windows Server 2019 node cannot run a container that uses a Windows Server 2022 image.
For a Windows Server 2022 node to run a container that uses a Windows Server 2019 image, you must run the container with Hyper-V isolation. Refer to the Microsoft documentation Hyper-V isolation for containers for more information.
Mirantis recommends that you use the same version of Windows Server for both your container images and for the node on which the containers run. For reference purposes, in both Kubernetes and Swarm clusters, MKE assigns a label to Windows nodes that includes the Windows Server version.
Run hybrid workloads in Kubernetes¶
To run Windows workloads in a hybrid Windows Kubernetes cluster, you must target your workloads to nodes that are running the correct Windows version. Failure to correctly target your workloads may result in an error when Kubernetes schedules the Pod on an incompatible node:
Error response from daemon: hcsshim::CreateComputeSystem win2019-deployment-no-nodeselect: The container operating system does not match the host operating system.
Note the Windows version associated with each of the nodes in your cluster:
kubectl get node
Example output:
NAME STATUS ROLES AGE VERSION manager-node Ready master 51m v1.23.4-mirantis-1 win2019-node Ready <none> 44m v1.23.4 win2022-node Ready <none> 38m v1.23.4
Create a deployment with the appropriate node selectors. Use
10.0.17763for Windows Server 2019 workloads and10.0.20348for Windows Server 2022 workloads.For example purposes, paste the following content into a file called
win2019-deployment.yaml:apiVersion: apps/v1 kind: Deployment metadata: labels: app: win2019-deployment name: win2019-deployment spec: replicas: 5 selector: matchLabels: app: win2019-deployment template: metadata: labels: app: win2019-deployment name: win2019-deployment spec: containers: - name: win2019-deployment image: mcr.microsoft.com/windows/nanoserver:1809 command: ["cmd", "/c", "ping -t localhost"] ports: - containerPort: 80 nodeSelector: kubernetes.io/os: windows node.kubernetes.io/windows-build: 10.0.17763
Apply the deployment:
kubectl apply -f win2019-deployment.yaml
Verify that the Pods are scheduled on the required node:
kubectl get pods -o wide
Example output:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES win2019-deployment-57d75f6f9f-ldsqf 1/1 Running 0 6m39s 192.168.50.76 win2019-node <none> <none> win2019-deployment-57d75f6f9f-n5b25 1/1 Running 0 6m39s 192.168.50.79 win2019-node <none> <none> win2019-deployment-57d75f6f9f-r5mz6 1/1 Running 0 6m39s 192.168.50.78 win2019-node <none> <none> win2019-deployment-57d75f6f9f-xggmt 1/1 Running 0 6m39s 192.168.50.77 win2019-node <none> <none> win2019-deployment-57d75f6f9f-zltk2 1/1 Running 0 7m7s 192.168.50.73 win2019-node <none> <none>
Run hybrid workloads in Swarm¶
To run Windows workloads in a hybrid Windows Swarm cluster, you must target your workloads to nodes that are running the correct Windows version. Failure to correctly target your workloads may result in an operating system mismatch error.
Verify that nodes running the appropriate Windows version are present in the cluster. Use an
OsVersionlabel of10.0.17763for Windows Server 2019 and10.0.20348for Windows Server 2022. For example:docker node ls -f "node.label=OsVersion=10.0.20348"
Example output:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION yft1t1mnytt524y03zmdevzuk win2022-node-1 Ready Active 20.10.12
Create a service that runs the required version of Windows Server, in this case Windows Server 2022. The service requires the inclusion of various constraints, to ensure that it is scheduled on the correct node. For example:
docker service create --name windows2022-example-service \ --constraint "node.platform.OS == windows" \ --constraint "node.labels.OsVersion == 10.0.20348" \ mcr.microsoft.com/windows/nanoserver:ltsc2022 cmd "/c ping -t localhost"
Verify that the service is scheduled on the required node:
docker service ps windows2022-example-service
Example output:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS uqrosib62602 windows2022-example-service.1 mcr.microsoft.com/windows/nanoserver:ltsc2022 win2022-node-1 Running Running 9 minutes ago