Contents Menu Expand Light mode Dark mode Auto light/dark mode
Documentation Portal
Logo
Mirantis Container Cloud
  • Product Overview
  • Reference Architecture
    • Overview
    • Cloud provider
    • Release Controller
    • Web UI
    • Bare metal
      • Bare metal components
      • Bare metal networking
        • IP Address Management
        • Management cluster networking
        • Managed cluster networking
        • Host networking
      • Storage
        • Overview
        • Limitations
      • Extended hardware configuration
      • Automatic upgrade of a host operating system
    • Built-in load balancing
    • VMware vSphere network objects and IPAM recommendations
    • Kubernetes lifecycle management
      • LCM custom resources
      • LCM Controller
      • LCM Agent
      • Helm Controller
    • Identity and access management
      • External identity provider integration
      • Authentication and authorization
        • Implementation flow
        • Kubernetes CLI authentication flow
    • Monitoring
      • Deployment architecture
      • Authentication flow
      • Supported features
      • Monitored components
      • Outbound cluster metrics
      • StackLight proxy
      • Reference Application for workload monitoring
    • Hardware and system requirements
      • Requirements for a bootstrap node
      • Requirements for a baremetal-based cluster
        • Reference hardware configuration
        • System requirements for the seed node
        • Network fabric
        • Management cluster storage
      • Requirements for an OpenStack-based cluster
      • Requirements for a VMware vSphere-based cluster
      • StackLight requirements for an MKE attached cluster
    • Proxy and cache support
    • Firewall configuration
      • Container Cloud
      • Mirantis Kubernetes Engine
      • StackLight
      • Ceph
    • Mirantis Kubernetes Engine API limitations
  • Deployment Guide
    • Deploy a baremetal-based management cluster
      • Workflow overview
      • Bootstrap a management cluster
        • Prepare the seed node
        • Configure BIOS on a bare metal host
        • Prepare metadata and deploy the management cluster
      • Configure NIC bonding
      • Customize the default bare metal host profile
      • Separate PXE and management networks
      • Configure multiple DHCP ranges using Subnet resources
    • Deploy an OpenStack-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy a VMware vSphere-based management cluster
      • Workflow overview
      • Deployment resources requirements
      • Prerequisites
      • Prepare the VMware deployment user setup and permissions
      • Bootstrap a management cluster
      • Prepare the virtual machine template
      • Configure squid-proxy
      • RHEL 8 mirrors configuration
    • Deploy an additional regional cluster (optional)
      • Configure the bootstrap node
      • Deploy an OpenStack-based regional cluster
      • Deploy a baremetal-based regional cluster
      • Deploy a VMware vSphere-based regional cluster
    • Requirements for a MITM proxy
    • Create initial users after a management cluster bootstrap
    • Troubleshooting
      • Collect the bootstrap logs
      • Troubleshoot the bootstrap node configuration
        • DNS settings
        • Default network address
      • Troubleshoot OpenStack-based deployments
        • TLS handshake timeout
      • Troubleshoot vSphere-based deployments
        • Virtual machine issues with obtaining an IP
    • Configure external identity provider for IAM
      • Configure LDAP for IAM
      • Configure Google OAuth IdP for IAM
  • Operations Guide
    • Mirantis Container Cloud CLI
    • Create and operate managed clusters
      • Create a project for managed clusters
      • Generate a kubeconfig for a managed cluster using API
      • Create and operate a baremetal-based managed cluster
        • Add a bare metal host
          • Add a bare metal host using web UI
          • Add a bare metal host using CLI
        • Create a custom bare metal host profile
          • Default configuration of the host system storage
          • Create a custom host profile
          • Enable huge pages
          • Configure RAID support
            • Create an LVM software RAID level 1 (raid1)
            • Create an mdadm software RAID level 1 (raid1)
            • Create an mdadm software RAID level 10 (raid10)
        • Add a managed baremetal cluster
          • Create a cluster using web UI
          • Workflow of network interface naming
          • Create subnets
            • Service labels and their life cycle
            • Create subnets for a managed cluster using web UI
            • Create subnets for a managed cluster using CLI
          • Automate multiple subnet creation using SubnetPool
          • Create L2 templates
            • L2 template example with bonds and bridges
            • L2 template example for automatic multiple subnet creation
        • Add a machine
          • Create a machine using web UI
          • Create a machine using CLI
            • Deploy a machine to a specific bare metal host
            • Assign L2 templates to machines
            • Override network interfaces naming and order
        • Add a Ceph cluster
        • Example of a complete L2 templates configuration for cluster creation
        • Manage an existing bare metal cluster
          • Expand IP addresses capacity in an existing cluster
        • Manage machines of a bare metal cluster
          • Modify network configuration on an existing machine
        • Manage Ceph
          • Ceph advanced configuration
          • Ceph default configuration options
          • Automated Ceph LCM
            • High-level workflow of Ceph OSD or node removal
              • Creating a Ceph OSD removal request
              • KaaSCephOperationRequest OSD removal specification
              • KaaSCephOperationRequest OSD removal status
            • Add, remove, or reconfigure Ceph OSDs
            • Add, remove, or reconfigure Ceph nodes
            • Replace a failed Ceph OSD
            • Replace a failed Ceph node
          • Increase Ceph cluster storage size
          • Move a Ceph Monitor daemon to another node
          • Migrate a Ceph Monitor before machine replacement
          • Enable Ceph RGW Object Storage
          • Enable Multisite for Ceph RGW Object Storage
          • Manage Ceph RBD or CephFS clients and RGW users
            • Manage Ceph RBD or CephFS clients
            • Manage Ceph Object Storage users
          • Set an Amazon S3 bucket policy
            • Create Ceph Object Storage users
            • Set a bucket policy for a Ceph Object Storage user
          • Verify Ceph
            • Verify the Ceph core services
            • Verify rook-discover
            • Verify Ceph cluster state through CLI
              • Verify Ceph cluster state
              • KaaSCephCluster.status description
            • View Ceph cluster summary through Container Cloud web UI
            • Verify Ceph Controller and Rook
          • Enable Ceph tolerations and resources management
            • Enable Ceph tolerations and resources management
            • Verify Ceph tolerations and resources management
          • Enable Ceph multinetwork
          • Enable TLS for Ceph public endpoints
          • Enable Ceph RBD mirroring
          • Enable Ceph Shared File System (CephFS)
          • Share Ceph across the region
            • Connect to and manage a shared Ceph cluster
            • Share a Ceph cluster with an attached MKE cluster
          • Calculate target ratio for Ceph pools
          • Specify placement of Ceph cluster daemons
        • Delete a managed cluster
      • Create and operate an OpenStack-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Boot a machine from a block storage volume
        • Delete a managed cluster
      • Create and operate a VMware vSphere-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Add or update a CA certificate for a MITM proxy using API
      • Operate machine pools
        • Assign or unassign a machine to or from a machine pool
        • Change replicas count of a machine pool
        • Delete a machine pool
      • Change the upgrade order of a machine or machine pool
      • Change a cluster configuration
      • Update a managed cluster
        • Verify the Container Cloud status before managed cluster update
        • Update a managed cluster
      • Update a patch Cluster release of a managed cluster
    • Add a Container Cloud cluster to Lens
    • Attach an existing Mirantis Kubernetes Engine cluster
    • Connect to the Mirantis Kubernetes Engine web UI
    • Connect to a Mirantis Container Cloud cluster
    • Inspect the history of a cluster and machine deployment or update
    • Operate management and regional clusters
      • Automatic upgrade workflow
      • Schedule Mirantis Container Cloud upgrades
      • Renew the Container Cloud and MKE licenses
      • Configure NTP server for a regional cluster
      • Automatically propagate Salesforce configuration to all clusters
      • Update the Keycloak IP address on bare metal clusters
      • Back up MariaDB on an OpenStack-based management cluster
        • Configure periodic backups of MariaDB for the OpenStack provider
        • Verify operability of the MariaDB backup jobs
        • Restore MariaDB databases
      • Back up and restore a management cluster
      • Remove a management cluster
      • Remove a regional cluster
    • Increase CPU and memory limits for cluster components
    • Configure TLS certificates for cluster applications
    • Define a custom CA certificate for a private Docker registry
    • Enable cluster and machine maintenance mode
      • Enable maintenance mode on a cluster and machine using web UI
      • Enable maintenance mode on a cluster and machine using CLI
    • Perform a graceful reboot of a cluster
    • Delete a cluster machine
      • Precautions for a cluster machine deletion
      • Delete a cluster machine using web UI
      • Delete a cluster machine using CLI
    • Manage IAM
      • Manage user roles through Container Cloud API
        • Available IAM roles and use cases
        • Mapping of Keycloak roles to IAM*RoleBinding objects
      • Manage user roles through the Container Cloud web UI
      • Manage user roles through Keycloak
        • Container Cloud roles and scopes
        • Use cases
        • Access the Keycloak Admin Console
      • Change passwords for IAM users
      • Obtain MariaDB credentials for IAM
    • Manage StackLight
      • Access StackLight web UIs
      • View Grafana dashboards
      • OpenSearch Dashboards
        • View OpenSearch Dashboards
        • Search in OpenSearch Dashboards
      • Available StackLight alerts
        • Alert dependencies
        • Alertmanager
        • Bond interface
        • Calico
        • Ceph
        • Docker Swarm
        • Elasticsearch Exporter
        • Etcd
        • External endpoint
        • Fluentd
        • General alerts
        • General node alerts
        • Grafana
        • Helm Controller
        • Ironic
        • Kubernetes applications
        • Kubernetes resources
        • Kubernetes storage
        • Kubernetes system
        • Mirantis Container Cloud
        • Mirantis Kubernetes Engine
        • NGINX
        • Node network
        • Node time
        • OpenSearch
        • PostgreSQL
        • Prometheus
        • Prometheus MS Teams
        • Prometheus Relay
        • Reference Application
        • Release Controller
        • ServiceNow
        • Salesforce notifier
        • SMART disks
        • SSL certificates
        • Telegraf
        • Telemeter
      • Troubleshoot alerts
        • Troubleshoot Helm Controller alerts
        • Troubleshoot Kubernetes applications alerts
        • Troubleshoot Kubernetes resources alerts
        • Troubleshoot Kubernetes storage alerts
        • Troubleshoot Mirantis Container Cloud Exporter alerts
        • Troubleshoot Mirantis Kubernetes Engine alerts
        • Troubleshoot Release Controller alerts
        • Troubleshoot Kubernetes system alerts
        • Troubleshoot Telemeter client alerts
      • Silence alerts
      • Configure StackLight
        • StackLight configuration procedure
        • StackLight configuration parameters
        • Verify StackLight after configuration
      • Access OpenSearch clusters using Cerebro
      • Enable generic metric scraping
      • Enable log forwarding to external destinations
      • Enable remote logging to syslog
      • Create logs-based metrics
      • Deschedule StackLight Pods from a worker machine
      • Calculate the storage retention time
    • Troubleshooting
      • Collect cluster logs
      • Cluster deletion or detachment freezes
      • Authentication failure with the 401 Unauthorized error
      • Stuck kubelet on the Cluster release 5.x.x series
      • MOSK clusters update fails with stuck kubelet
      • Troubleshoot baremetal-based clusters
        • Bare metal hosts in ‘provisioned registration error’ state after update
        • Troubleshoot an operating system upgrade with host restart
        • Troubleshoot iPXE boot issues
      • Troubleshoot vSphere-based clusters
        • Node leaves the cluster after IP address change
      • Troubleshoot Ceph
        • Ceph disaster recovery
        • Ceph Monitors recovery
        • Remove Ceph OSD manually
        • KaaSCephOperationRequest failure with a timeout during rebalance
        • Ceph Monitors store.db size rapidly growing
      • Troubleshoot StackLight
        • Patroni replication lag
        • Alertmanager does not send resolve notifications for custom alerts
        • OpenSearchPVCMismatch alert raises due to the OpenSearch PVC size mismatch
        • StackLight pods get stuck with the ‘NodeAffinity failed’ error
  • API Reference
    • Public key resources
    • License resource
    • IAM resources
    • MachinePool resource
    • MCCUpgrade resource
    • GracefulRebootRequest resource
    • ContainerRegistry resource
    • TLSConfig resource
    • Bare metal resources
      • Cluster
      • Machine
      • BareMetalHostProfile
      • BareMetalHostCredential
      • BareMetalHost
      • IpamHost
      • Subnet
      • SubnetPool
      • MetalLBConfig
      • IPaddr
      • L2Template
    • vSphere resources
      • VsphereCredential
      • Cluster
      • Machine
      • VsphereResources
    • OpenStack resources
      • Cluster
      • Machine
      • OpenStackCredential
      • OpenStackResources

Release notes

  • Release Compatibility Matrix
    • Compatibility matrix of component versions
    • Support matrix of MKE versions for cluster attachment
    • Container Cloud web UI browser compatibility
  • Release Notes
    • Container Cloud releases
      • 2.23.5 (current)
        • Artifacts
        • Security notes
        • Known issues
      • Unsupported releases
        • 2.23.4
          • Artifacts
          • Security notes
          • Addressed issues
        • 2.23.3
          • Artifacts
          • Security notes
        • 2.23.2
          • Artifacts
          • Security notes
        • 2.23.1 (current)
        • 2.23.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
          • Security notes
        • 2.22.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
          • Security notes
        • Releases delivered in 2022
          • 2.21.1
          • 2.21.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
            • Post-upgrade actions
          • 2.20.1
          • 2.20.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.19.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.18.1
          • 2.18.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.17.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.16.1
          • 2.16.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.15.1
          • 2.15.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
        • Releases delivered in 2020-2021
          • 2.14.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.13.1
          • 2.13.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.12.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.11.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
            • Upgrade managed clusters with StackLight deployed in HA mode
          • 2.10.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.9.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
            • Switch L2 templates to the new format
          • 2.8.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.7.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.6.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.5.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.4.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.3.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.2.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.1.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
            • Apply updates to the AWS-based management clusters
          • 2.0.0
            • Known issues
            • Components versions
            • Artifacts
    • Cluster releases (managed)
      • 12.7.x series
        • 12.7.4
        • 12.7.3
        • 12.7.2
        • 12.7.1
        • 12.7.0
          • Enhancements
          • Components versions
          • Artifacts
      • 11.7.x series
        • 11.7.4
        • 11.7.3
        • 11.7.2
        • 11.7.1
        • 11.7.0
          • Enhancements
          • Components versions
          • Artifacts
      • Deprecated Cluster releases
        • 12.5.0
          • Enhancements
          • Components versions
          • Artifacts
        • 11.6.0
          • Enhancements
          • Components versions
          • Artifacts
      • Unsupported Cluster releases
        • 11.x series
          • 11.5.0
            • Enhancements
            • Components versions
            • Artifacts
          • 11.4.0
            • Enhancements
            • Components versions
            • Artifacts
          • 11.3.0
            • Enhancements
            • Components versions
            • Artifacts
          • 11.2.0
            • Enhancements
            • Components versions
            • Artifacts
          • 11.1.0
            • Enhancements
            • Components versions
            • Artifacts
          • 11.0.0
            • Enhancements
            • Components versions
            • Artifacts
        • 8.x series
          • 8.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 8.8.0
            • Enhancements
            • Components versions
            • Artifacts
          • 8.6.0
            • Enhancements
            • Components versions
            • Artifacts
          • 8.5.0
            • Enhancements
            • Components versions
            • Artifacts
        • 7.x series
          • 7.11.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.9.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.8.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.7.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.6.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.5.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.4.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.3.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.2.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.1.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.0.0
            • Enhancements
            • Components versions
            • Artifacts
        • 6.x series
          • 6.20.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.19.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.18.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.8.1
        • 5.x series
          • 5.22.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.21.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.20.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.19.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.18.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.17.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.15.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.13.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.11.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.9.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.8.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.7.0
            • Components versions
            • Artifacts
    • Patch releases
    • Deprecation notes

QuickStart guides

  • QuickStart guides overview
  • QuickStart: Container Cloud on OpenStack
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the OpenStack configuration
    • Configure the cluster and machines metadata
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on VMware vSphere
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the deployment user setup and permissions
    • Configure the cluster and vSphere credentials
    • Prepare the virtual machine template
    • Finalize the bootstrap
    • What’s next

Troubleshoot Ceph¶

This section provides solutions to the issues that may occur during Ceph usage.

  • Ceph disaster recovery
  • Ceph Monitors recovery
  • Remove Ceph OSD manually
  • KaaSCephOperationRequest failure with a timeout during rebalance
  • Ceph Monitors store.db size rapidly growing
Next
Ceph disaster recovery
Previous
Node leaves the cluster after IP address change
  • Multi-page view
  • Single-page view

Mirantis Inc. 900 E Hamilton Avenue, Suite 650, Campbell, CA 95008 +1-650-963-9828

© 2005 - 2023 Mirantis, Inc. All rights reserved. "Mirantis" and "FUEL" are registered trademarks of Mirantis, Inc. All other trademarks are the property of their respective owners.