Contents Menu Expand Light mode Dark mode Auto light/dark mode
Documentation Portal
Logo
Mirantis Container Cloud
  • Product Overview
  • Reference Architecture
    • Overview
    • Cloud provider
    • Release Controller
    • Web UI
    • Bare metal
      • Bare metal components
      • Bare metal networking
        • IP Address Management
        • Management cluster networking
        • Managed cluster networking
        • Host networking
      • Extended hardware configuration
      • Automatic upgrade of a host operating system
    • Built-in load balancing
    • VMware vSphere network objects and IPAM recommendations
    • Equinix Metal with private networking
      • Infrastructure prerequisites
      • Design details
    • Kubernetes lifecycle management
      • LCM custom resources
      • LCM Controller
      • LCM Agent
      • Helm Controller
    • Identity and access management
      • External identity provider integration
      • Authentication and authorization
        • Implementation flow
        • Kubernetes CLI authentication flow
    • Storage
      • Overview
      • Limitations
    • Monitoring
      • Deployment architecture
      • Authentication flow
      • Supported features
      • Monitored components
      • Outbound cluster metrics
      • StackLight proxy
    • Hardware and system requirements
      • Requirements for a bootstrap node
      • Requirements for a baremetal-based cluster
        • Reference hardware configuration
        • System requirements for the seed node
        • Network fabric
        • Management cluster storage
      • Requirements for an OpenStack-based cluster
      • Requirements for an AWS-based cluster
      • Requirements for an Azure-based cluster
      • Requirements for an Equinix Metal based cluster
      • Requirements for a VMware vSphere-based cluster
      • StackLight requirements for an MKE attached cluster
    • Proxy and cache support
    • Mirantis Kubernetes Engine API limitations
  • Deployment Guide
    • Deploy a baremetal-based management cluster
      • Workflow overview
      • Bootstrap a management cluster
        • Prepare the seed node
        • Verify the seed node
        • Prepare metadata and deploy the management cluster
      • Configure NIC bonding
      • Customize the default bare metal host profile
      • Separate PXE and management networks
      • Configure multiple DHCP ranges using Subnet resources
    • Deploy an OpenStack-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an AWS-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an Azure-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an Equinix Metal based management cluster with public networking
      • Workflow overview
      • Prerequisites
      • Equinix Metal project setup
      • Verify the capacity of the Equinix Metal facility
      • Bootstrap a management cluster
    • Deploy an Equinix Metal based management cluster with private networking
      • Workflow overview
      • Prerequisites
      • Verify the capacity of the Equinix Metal facility
      • Bootstrap a management cluster
    • Deploy a VMware vSphere-based management cluster
      • Workflow overview
      • Deployment resources requirements
      • Prerequisites
      • Prepare the VMware deployment user setup and permissions
      • Bootstrap a management cluster
      • Prepare the OVF template
      • Configure squid-proxy
      • RHEL 8 mirrors configuration
    • Deploy an additional regional cluster (optional)
      • Configure the bootstrap node
      • Deploy an AWS-based regional cluster
      • Deploy an Azure-based regional cluster
      • Deploy an OpenStack-based regional cluster
      • Deploy an Equinix Metal based regional cluster with public networking
      • Deploy an Equinix Metal based regional cluster with private networking
      • Deploy a baremetal-based regional cluster
      • Deploy a VMware vSphere-based regional cluster
    • Requirements for a MITM proxy
    • Create initial users after a management cluster bootstrap
    • Troubleshooting
      • Collect the bootstrap logs
      • Troubleshoot the bootstrap node configuration
        • DNS settings
        • Default network address
      • Troubleshoot OpenStack-based deployments
        • TLS handshake timeout
      • Troubleshoot vSphere-based deployments
        • Virtual machine issues with obtaining an IP
    • Configure external identity provider for IAM
      • Configure LDAP for IAM
      • Configure Google OAuth IdP for IAM
  • Operations Guide
    • Mirantis Container Cloud CLI
    • Create and operate managed clusters
      • Create a project for managed clusters
      • Generate a kubeconfig for a managed cluster using API
      • Create and operate a baremetal-based managed cluster
        • Add a bare metal host
          • Add a bare metal host using web UI
          • Add a bare metal host using CLI
        • Create a custom bare metal host profile
          • Default configuration of the host system storage
          • Create a custom host profile
          • Enable huge pages
          • Configure RAID support
            • Create an LVM software RAID level 1 (raid1)
            • Create an mdadm software RAID level 1 (raid1)
            • Create an mdadm software RAID level 10 (raid10)
        • Add a managed baremetal cluster
          • Create a cluster using web UI
          • Workflow of network interface naming
          • Create subnets
            • Service labels and their life cycle
            • Create subnets for a managed cluster using web UI
            • Create subnets for a managed cluster using CLI
          • Automate multiple subnet creation using SubnetPool
          • Create L2 templates
            • L2 template example with bonds and bridges
            • L2 template example for automatic multiple subnet creation
        • Add a machine
          • Create a machine using web UI
          • Create a machine using CLI
            • Deploy a machine to a specific bare metal host
            • Assign L2 templates to machines
            • Override network interfaces naming and order
        • Add a Ceph cluster
        • Manage an existing bare metal cluster
          • Expand IP addresses capacity in an existing cluster
        • Manage machines of a bare metal cluster
          • Modify network configuration on an existing machine since 2.19.0
          • Modify network configuration on an existing machine before 2.19.0
        • Delete a managed cluster
        • Example of a complete L2 templates configuration for cluster creation
      • Create and operate an OpenStack-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Boot a machine from a block storage volume
        • Delete a managed cluster
      • Create and operate an AWS-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Azure-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Equinix Metal based managed cluster with public networking
        • Create a managed cluster
        • Increase BGP max_prefix
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Equinix Metal based managed cluster with private networking
        • Create a managed cluster
        • Add a machine
        • Establish connection to the cluster private network
        • Delete a managed cluster
      • Create and operate a VMware vSphere-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Add or update a CA certificate for a MITM proxy
      • Operate machine pools
        • Assign or unassign a machine to or from a machine pool
        • Change replicas count of a machine pool
        • Delete a machine pool
      • Change the upgrade order of a machine or machine pool
      • Change a cluster configuration
      • Update a managed cluster
        • Verify the Container Cloud status before managed cluster update
        • Update a managed cluster
    • Add a Container Cloud cluster to Lens
    • Attach an existing Mirantis Kubernetes Engine cluster
    • Connect to the Mirantis Kubernetes Engine web UI
    • Connect to a Mirantis Container Cloud cluster
    • Operate management and regional clusters
      • Automatic upgrade workflow
      • Schedule Mirantis Container Cloud upgrades
      • Renew the Container Cloud and MKE licenses
      • Configure NTP server for a regional cluster
      • Automatically propagate Salesforce configuration to all clusters
      • Update the Keycloak IP address on bare metal clusters
      • Back up MariaDB on an AWS or OpenStack-based management cluster
        • Configure periodic backups of MariaDB for AWS and OpenStack providers
        • Verify operability of the MariaDB backup jobs
        • Restore MariaDB databases
      • Back up and restore a management cluster
      • Remove a management cluster
      • Remove a regional cluster
    • Increase CPU and memory limits for cluster components
    • Configure TLS certificates for cluster applications
    • Define a custom CA certificate for a private Docker registry
    • Enable cluster and machine maintenance mode
      • Enable maintenance mode on a cluster and machine using web UI
      • Enable maintenance mode on a cluster and machine using CLI
    • Delete a cluster machine
    • Manage IAM
      • Manage user roles through Container Cloud API
        • Available IAM roles and use cases
        • Mapping of Keycloak roles to IAM*RoleBinding objects
      • Manage user roles through the Container Cloud web UI
      • Manage user roles through Keycloak
        • Container Cloud roles and scopes
        • Use cases
        • Access the Keycloak Admin Console
      • Change passwords for IAM users
    • Manage StackLight
      • Access StackLight web UIs
      • View Grafana dashboards
      • View OpenSearch Dashboards
      • Available StackLight alerts
        • Alert dependencies
        • Alertmanager
        • Calico
        • Ceph
        • Docker Swarm
        • Elasticsearch Exporter
        • Etcd
        • External endpoint
        • Fluentd
        • General alerts
        • General node alerts
        • Grafana
        • Helm Controller
        • Ironic
        • Kubernetes applications
        • Kubernetes resources
        • Kubernetes storage
        • Kubernetes system
        • Mirantis Container Cloud
        • Mirantis Kubernetes Engine
        • NGINX
        • Node network
        • Node time
        • OpenSearch
        • PostgreSQL
        • Prometheus
        • Prometheus MS Teams
        • Prometheus Relay
        • Pushgateway
        • Release Controller
        • ServiceNow
        • Salesforce notifier
        • SMART disks
        • SSL certificates
        • Telegraf
        • Telemeter
      • Troubleshoot alerts
        • Troubleshoot Helm Controller alerts
        • Troubleshoot Kubernetes applications alerts
        • Troubleshoot Kubernetes resources alerts
        • Troubleshoot Kubernetes storage alerts
        • Troubleshoot Metric Collector alerts
        • Troubleshoot Mirantis Kubernetes Engine alerts
        • Troubleshoot Release Controller alerts
        • Troubleshoot Kubernetes system alerts
      • Silence alerts
      • Configure StackLight
        • StackLight configuration procedure
        • StackLight configuration parameters
        • Verify StackLight after configuration
      • Access OpenSearch clusters using Cerebro
      • Enable generic metric scraping
      • Enable remote logging to syslog
      • Create logs-based metrics
    • Manage Ceph
      • Ceph advanced configuration
      • Ceph default configuration options
      • Automated Ceph LCM
        • High-level workflow of Ceph OSD or node removal
          • Creating a Ceph OSD removal request
          • KaaSCephOperationRequest OSD removal specification
          • KaaSCephOperationRequest OSD removal status
        • Add, remove, or reconfigure Ceph OSDs
        • Add, remove, or reconfigure Ceph nodes
        • Replace a failed Ceph OSD
        • Replace a failed Ceph node
      • Increase Ceph cluster storage size
      • Move a Ceph Monitor daemon to another node
      • Migrate a Ceph Monitor before machine replacement
      • Enable Ceph RGW Object Storage
      • Enable Multisite for Ceph RGW Object Storage
      • Verify Ceph
        • Verify the Ceph core services
        • Verify rook-discover
        • Verify Ceph cluster state through CLI
          • Verify Ceph cluster state
          • KaaSCephCluster.status description
        • View Ceph cluster summary through Container Cloud web UI
        • Verify Ceph Controller and Rook
      • Enable Ceph tolerations and resources management
        • Enable Ceph tolerations and resources management
        • Verify Ceph tolerations and resources management
      • Enable Ceph multinetwork
      • Enable TLS for Ceph public endpoints
      • Enable Ceph RBD mirroring
      • Enable Ceph Shared File System (CephFS)
      • Calculate target ratio for Ceph pools
      • Specify placement of Ceph cluster daemons
      • Update Ceph on a management or regional cluster
    • Troubleshooting
      • Collect cluster logs
      • Cluster deletion or detachment freezes
      • Authentication failure with the 401 Unauthorized error
      • Helm releases get stuck in FAILED or UNKNOWN state during cluster deployment
      • Stuck kubelet on the Cluster release 5.x.x series
      • Equinix or MOSK clusters update fails with stuck kubelet
      • Troubleshoot AWS-based clusters
        • AWS-based managed cluster requiring PVs fails to be deployed
      • Troubleshoot baremetal-based clusters
        • Bare metal hosts in ‘provisioned registration error’ state after update
        • Troubleshoot an operating system upgrade with host restart
        • Troubleshoot iPXE boot issues
      • Troubleshoot Equinix Metal based clusters
        • Machine provisioning issues during cluster deployment with private network
      • Troubleshoot vSphere-based clusters
        • Node leaves the cluster after IP address change
      • Troubleshoot Ceph
        • Ceph disaster recovery
        • Ceph Monitors recovery
        • Remove Ceph OSD manually
        • KaaSCephOperationRequest failure with a timeout during rebalance
        • Ceph Monitors store.db size rapidly growing
      • Troubleshoot StackLight
        • Patroni replication lag
        • Alertmanager does not send resolve notifications for custom alerts
  • API Reference
    • Public key resources
    • License resource
    • IAM resources
    • MachinePool resource
    • MCCUpgrade resource
    • ContainerRegistry resource
    • TLSConfig resource
    • AWS resources
      • Cluster
      • Machine
      • AWSCredential
      • AWSResources
    • Bare metal resources
      • Cluster
      • Machine
      • BareMetalHostProfile
      • BareMetalHost
      • IpamHost
      • Subnet
      • SubnetPool
      • IPaddr
      • L2Template
    • vSphere resources
      • VsphereCredential
      • Cluster
      • Machine
      • VsphereResources
    • OpenStack resources
      • Cluster
      • Machine
      • OpenStackCredential
      • OpenStackResources

Release notes

  • Release Compatibility Matrix
    • Compatibility matrix of components versions
    • Support matrix of MKE versions for cluster attachment
    • Container Cloud web UI browser compatibility
  • Release Notes
    • Mirantis Container Cloud releases
      • 2.20.1 (current)
      • Unsupported releases
        • 2.20.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.19.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.18.1
        • 2.18.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.17.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.16.1
        • 2.16.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.15.1
        • 2.15.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • Releases delivered in 2020-2021
          • 2.14.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.13.1
          • 2.13.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.12.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.11.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
            • Upgrade managed clusters with StackLight deployed in HA mode
          • 2.10.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.9.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
            • Switch L2 templates to the new format
          • 2.8.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.7.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.6.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.5.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.4.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.3.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.2.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.1.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
            • Apply updates to the AWS-based management clusters
          • 2.0.0
            • Known issues
            • Components versions
            • Artifacts
    • Cluster releases (managed)
      • 11.4.0
        • Enhancements
        • Components versions
        • Artifacts
      • 8.10.0
        • Enhancements
        • Components versions
        • Artifacts
      • 7.10.0
        • Enhancements
        • Components versions
        • Artifacts
      • Deprecated Cluster releases
        • 11.3.0
          • Enhancements
          • Components versions
          • Artifacts
        • 8.8.0
          • Enhancements
          • Components versions
          • Artifacts
        • 7.9.0
          • Enhancements
          • Components versions
          • Artifacts
      • Unsupported Cluster releases
        • 11.x series
          • 11.2.0
            • Enhancements
            • Components versions
            • Artifacts
          • 11.1.0
            • Enhancements
            • Components versions
            • Artifacts
          • 11.0.0
            • Enhancements
            • Components versions
            • Artifacts
        • 8.x series
          • 8.6.0
            • Enhancements
            • Components versions
            • Artifacts
          • 8.5.0
            • Enhancements
            • Components versions
            • Artifacts
        • 7.x series
          • 7.8.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.7.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.6.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.5.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.4.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.3.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.2.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.1.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.0.0
            • Enhancements
            • Components versions
            • Artifacts
        • 6.x series
          • 6.20.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.19.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.18.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.8.1
        • 5.x series
          • 5.22.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.21.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.20.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.19.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.18.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.17.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.15.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.13.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.11.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.9.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.8.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.7.0
            • Components versions
            • Artifacts
    • Deprecation notes

QuickStart guides

  • QuickStart guides overview
  • QuickStart: Container Cloud on AWS
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the AWS configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Microsoft Azure
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the Azure configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Equinix Metal with public networking
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Set up the Equinix Metal project
    • Verify the capacity of the Equinix Metal facility
    • Prepare the Equinix Metal configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Equinix Metal with private networking
    • Before you begin
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Verify the capacity of the Equinix Metal facility
    • Prepare the Equinix Metal configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on OpenStack
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the OpenStack configuration
    • Configure the cluster and machines metadata
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on VMware vSphere
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the deployment user setup and permissions
    • Configure the cluster and vSphere credentials
    • Prepare the OVF template
    • Finalize the bootstrap
    • What’s next

Troubleshoot StackLight¶

This section provides solutions to the issues that may occur during StackLight usage.

  • Patroni replication lag
  • Alertmanager does not send resolve notifications for custom alerts
Next
Patroni replication lag
Previous
Ceph Monitors store.db size rapidly growing
  • Multi-page view
  • Single-page view

Mirantis Inc. 900 E Hamilton Avenue, Suite 650, Campbell, CA 95008 +1-650-963-9828

© 2005 - 2022 Mirantis, Inc. All rights reserved. "Mirantis" and "FUEL" are registered trademarks of Mirantis, Inc. All other trademarks are the property of their respective owners.