Contents Menu Expand Light mode Dark mode Auto light/dark mode
Documentation Portal
Logo
Miranis Container Cloud documentation
  • Product Overview
  • Reference Architecture
    • Overview
    • Cloud provider
    • Release controller
    • Web UI
    • Bare metal
      • Bare metal components
      • Bare metal networking
        • IP Address Management
        • Management cluster networking
        • Managed cluster networking
        • Host networking
      • Extended hardware configuration
    • Built-in load balancing
    • VMware vSphere network objects and IPAM recommendations
    • Kubernetes lifecycle management
      • LCM custom resources
      • LCM controller
      • LCM agent
      • Helm controller
    • Identity and access management
      • IAM API and CLI
      • External identity provider integration
      • Authentication and authorization
        • Implementation flow
        • Kubernetes CLI authentication flow
    • Storage
      • Overview
      • Limitations
    • Monitoring
      • Deployment architecture
      • Authentication flow
      • Supported features
      • Monitored components
      • Outbound cluster metrics
      • StackLight proxy
    • Hardware and system requirements
      • Requirements for a bootstrap node
      • Requirements for a baremetal-based cluster
        • Reference hardware configuration
        • System requirements for the seed node
        • Network fabric
        • Management cluster storage
      • Requirements for an OpenStack-based cluster
      • Requirements for an AWS-based cluster
      • Requirements for an Azure-based cluster
      • Requirements for an Equinix Metal based cluster
      • Requirements for a VMware vSphere-based cluster
    • Proxy and cache support
    • Mirantis Kubernetes Engine API limitations
  • Deployment Guide
    • Deploy a baremetal-based management cluster
      • Workflow overview
      • Bootstrap a management cluster
        • Prepare the seed node
        • Verify the seed node
        • Prepare metadata and deploy the management cluster
      • Configure NIC bonding
      • Customize the default bare metal host profile
    • Deploy an OpenStack-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an AWS-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an Azure-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an Equinix Metal based management cluster
      • Workflow overview
      • Prerequisites
      • Equinix Metal project setup
      • Verify the capacity of the Equinix Metal facility
      • Bootstrap a management cluster
    • Deploy a VMware vSphere-based management cluster
      • Workflow overview
      • Deployment resources requirements
      • Prerequisites
      • Prepare the VMware deployment user setup and permissions
      • Bootstrap a management cluster
      • Prepare the OVF template
      • Configure squid-proxy
    • Deploy an additional regional cluster (optional)
      • Configure the bootstrap node
      • Deploy an AWS-based regional cluster
      • Deploy an Azure-based regional cluster
      • Deploy an Equinix Metal based regional cluster
      • Deploy an OpenStack-based regional cluster
      • Deploy a VMware vSphere-based regional cluster
    • Create initial users after a management cluster bootstrap
    • Troubleshooting
      • Collect the bootstrap logs
      • DNS settings
      • Default network address
      • TLS handshake timeout
      • Troubleshoot vSphere-based deployments
        • Virtual machine issues with obtaining an IP
    • Configure external identity provider for IAM
      • Configure LDAP for IAM
      • Configure Google OAuth IdP for IAM
  • Operations Guide
    • Mirantis Container Cloud CLI
    • Create and operate managed clusters
      • Create and operate a baremetal-based managed cluster
        • Add a bare metal host
          • Add a bare metal host using web UI
          • Add a bare metal host using CLI
        • Create a custom bare metal host profile
          • Default configuration of the host system storage
          • Create a custom host profile
          • Enable huge pages in a host profile
        • Configure RAID support
          • Create an LVM software RAID level 1 (raid1)
          • Create an mdadm software RAID level 1 (raid1)
        • Create a managed cluster
        • Advanced networking configuration
          • Workflow of network interface naming
          • Create subnets
          • Automate multiple subnet creation using SubnetPool
          • Create L2 templates
            • L2 template example with bonds and bridges
            • L2 template example for automatic multiple subnet creation
          • Assign L2 templates to machines
          • Example of a complete L2 templates configuration for cluster creation
        • Add a machine
          • Create a machine using web UI
          • Create a machine using CLI
            • Deploy a machine to a specific bare metal host
            • Override network interfaces naming and order
        • Delete a managed cluster
      • Create and operate an OpenStack-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate an AWS-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Azure-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Equinix Metal based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate a VMware vSphere-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Add a Ceph cluster
      • Change a cluster configuration
      • Update a managed cluster
      • Delete a machine
    • Add a Container Cloud cluster to Lens
    • Attach an existing Mirantis Kubernetes Engine cluster
    • Connect to the Mirantis Kubernetes Engine web UI
    • Connect to a Mirantis Container Cloud cluster
    • Operate management and regional clusters
      • Automatic upgrade workflow
      • Renew the Container Cloud and MKE licenses
      • Configure NTP server for a regional cluster
      • Configure TLS certificates for management cluster applications
      • Update the Keycloak IP address on bare metal clusters
      • Back up MariaDB on an AWS or OpenStack-based management cluster
        • Configure periodic backups of MariaDB for AWS and OpenStack providers
        • Verify operability of the MariaDB backup jobs
        • Restore MariaDB databases
      • Back up and restore a management cluster
      • Remove a management cluster
      • Remove a regional cluster
    • Manage IAM
      • Manage user roles through Keycloak
        • Container Cloud roles and scopes
        • Use cases
        • Access the Keycloak Admin Console
      • Change passwords for IAM users
      • IAM CLI
        • Configure IAM CLI
        • Available IAM CLI commands
    • Manage StackLight
      • Access StackLight web UIs
      • View Grafana dashboards
      • View Kibana dashboards
      • Available StackLight alerts
        • Alert dependencies
        • Alertmanager
        • Calico
        • Ceph
        • Docker Swarm
        • Elasticsearch
        • etcd
        • External endpoint
        • General alerts
        • General node alerts
        • Ironic
        • Kubernetes applications
        • Kubernetes resources
        • Kubernetes storage
        • Kubernetes system
        • NGINX
        • Node network
        • Node time
        • PostgreSQL
        • Prometheus
        • Prometheus MS Teams
        • ServiceNow
        • Salesforce notifier
        • SMART disks
        • SSL certificates
        • Telegraf
        • Telemeter
        • Mirantis Kubernetes Engine
      • Configure StackLight
        • StackLight configuration procedure
        • StackLight configuration parameters
        • Verify StackLight after configuration
      • Access Elasticsearch clusters using Cerebro
      • Enable generic metric scraping
      • Enable remote logging to syslog
      • Create logs-based metrics
      • Silence alerts
    • Manage Ceph
      • Ceph advanced configuration
      • Ceph default configuration options
      • Enable automated Ceph LCM
      • Add, remove, or reconfigure Ceph nodes
      • Replace a failed Ceph OSD
      • Move a Ceph Monitor to another node
      • Update Ceph cluster
      • Enable Ceph RGW Object Storage
      • Enable Multisite for Ceph RGW Object Storage
      • Verify Ceph
        • Verify the Ceph core services
        • Verify rook-discover
        • Verify Ceph cluster state
          • Verify Ceph cluster state
          • KaaSCephCluster.status description
          • Verify Ceph cluster state prior to Container Cloud 2.11.0
        • Verify Ceph controller and Rook
      • Enable Ceph tolerations and resources management
        • Enable Ceph tolerations and resources management
        • Enable Ceph tolerations and resources prior to Container Cloud 2.11.0
        • Verify Ceph tolerations and resources management
      • Enable Ceph multinetwork
      • Enable TLS for Ceph public endpoints
      • Enable Ceph RBD mirroring
    • Troubleshooting
      • Collect cluster logs
      • Nodes in Not Ready state on long-running clusters
      • Authentication failure with the 401 Unauthorized error
      • Troubleshoot AWS-based clusters
        • Manual migration of the AWS-based Docker data and LVP volumes to an NVMe disk
        • AWS-based managed cluster requiring PVs fails to be deployed
      • Troubleshoot Ceph
        • Ceph disaster recovery
        • Cannot automatically remove a Ceph node
        • Ceph Monitors recovery
        • Remove Ceph OSD manually
  • API Reference
    • Public key resources
    • AWS resources
      • Cluster
      • Machine
      • AWSCredential
      • AWSResources
    • Bare metal resources
      • Cluster
      • Machine
      • BareMetalHostProfile
      • BareMetalHost
      • IpamHost
      • Subnet
      • SubnetPool
      • IPaddr
      • L2Template
    • OpenStack resources
      • Cluster
      • Machine
      • OpenStackCredential
      • OpenStackResources

Release notes

  • Release compatibility matrix
  • Release Notes
    • Product highlights
    • Mirantis Container Cloud releases
      • 2.12.0 (current)
        • Enhancements
        • Addressed issues
        • Known issues
        • Components versions
        • Artifacts
      • Unsupported releases
        • 2.11.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
          • Upgrade managed clusters with StackLight deployed in HA mode
        • 2.10.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.9.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
          • Switch L2 templates to the new format
        • 2.8.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.7.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.6.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.5.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.4.0
          • Enhancements
          • Known issues
          • Addressed issues
          • Components versions
          • Artifacts
        • 2.3.0
          • Enhancements
          • Known issues
          • Addressed issues
          • Components versions
          • Artifacts
        • 2.2.0
          • Enhancements
          • Known issues
          • Addressed issues
          • Components versions
          • Artifacts
        • 2.1.0
          • Enhancements
          • Known issues
          • Addressed issues
          • Components versions
          • Artifacts
          • Apply updates to the AWS-based management clusters
        • 2.0.0
          • Known issues
          • Components versions
          • Artifacts
    • Cluster releases (managed)
      • 7.2.0
        • Enhancements
        • Components versions
        • Artifacts
      • 6.19.0
        • Enhancements
        • Components versions
        • Artifacts
      • 5.19.0
        • Enhancements
        • Components versions
        • Artifacts
      • Deprecated Cluster releases
        • 7.1.0
          • Enhancements
          • Components versions
          • Artifacts
        • 6.18.0
          • Enhancements
          • Components versions
          • Artifacts
        • 5.18.0
          • Enhancements
          • Components versions
          • Artifacts
      • Unsupported Cluster releases
        • 7.x series
          • 7.0.0
            • Enhancements
            • Components versions
            • Artifacts
        • 6.x series
          • 6.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.8.1
        • 5.x series
          • 5.17.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.15.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.13.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.11.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.9.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.8.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.7.0
            • Components versions
            • Artifacts
    • Deprecation notes

QuickStart guides

  • QuickStart guides overview
  • QuickStart: Container Cloud on AWS
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the AWS configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Microsoft Azure
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the Azure configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Equinix Metal
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Set up the Equinix Metal project
    • Verify the capacity of the Equinix Metal facility
    • Prepare the Equinix Metal configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on OpenStack
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the OpenStack configuration
    • Configure the cluster and machines metadata
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on VMware vSphere
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the deployment user setup and permissions
    • Configure the cluster and vSphere credentials
    • Prepare the OVF template
    • Finalize the bootstrap
    • What’s next

Troubleshoot Ceph¶

This section provides solutions to the issues that may occur during Ceph usage.

  • Ceph disaster recovery
  • Cannot automatically remove a Ceph node
  • Ceph Monitors recovery
  • Remove Ceph OSD manually
Next
Ceph disaster recovery
Previous
AWS-based managed cluster requiring PVs fails to be deployed

Mirantis Inc. 900 E Hamilton Avenue, Suite 650, Campbell, CA 95008 +1-650-963-9828

© 2005 - 2021 Mirantis, Inc. All rights reserved. "Mirantis" and "FUEL" are registered trademarks of Mirantis, Inc. All other trademarks are the property of their respective owners.