Contents Menu Expand Light mode Dark mode Auto light/dark mode
Documentation Portal
Logo
Mirantis Container Cloud
  • Product Overview
  • Reference Architecture
    • Overview
    • Cloud provider
    • Release Controller
    • Web UI
    • Bare metal
      • Bare metal components
      • Bare metal networking
        • IP Address Management
        • Management cluster networking
        • Managed cluster networking
        • Host networking
      • Extended hardware configuration
      • Automatic upgrade of a host operating system
    • Built-in load balancing
    • VMware vSphere network objects and IPAM recommendations
    • Equinix Metal with private networking
      • Infrastructure prerequisites
      • Design details
    • Kubernetes lifecycle management
      • LCM custom resources
      • LCM Controller
      • LCM Agent
      • Helm Controller
    • Identity and access management
      • External identity provider integration
      • Authentication and authorization
        • Implementation flow
        • Kubernetes CLI authentication flow
    • Storage
      • Overview
      • Limitations
    • Monitoring
      • Deployment architecture
      • Authentication flow
      • Supported features
      • Monitored components
      • Outbound cluster metrics
      • StackLight proxy
    • Hardware and system requirements
      • Requirements for a bootstrap node
      • Requirements for a baremetal-based cluster
        • Reference hardware configuration
        • System requirements for the seed node
        • Network fabric
        • Management cluster storage
      • Requirements for an OpenStack-based cluster
      • Requirements for an AWS-based cluster
      • Requirements for an Azure-based cluster
      • Requirements for an Equinix Metal based cluster
      • Requirements for a VMware vSphere-based cluster
      • StackLight requirements for an MKE attached cluster
    • Proxy and cache support
    • Mirantis Kubernetes Engine API limitations
  • Deployment Guide
    • Deploy a baremetal-based management cluster
      • Workflow overview
      • Bootstrap a management cluster
        • Prepare the seed node
        • Verify the seed node
        • Prepare metadata and deploy the management cluster
      • Configure NIC bonding
      • Customize the default bare metal host profile
      • Separate PXE and management networks
      • Configure multiple DHCP ranges using Subnet resources
    • Deploy an OpenStack-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an AWS-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an Azure-based management cluster
      • Workflow overview
      • Prerequisites
      • Bootstrap a management cluster
    • Deploy an Equinix Metal based management cluster with public networking
      • Workflow overview
      • Prerequisites
      • Equinix Metal project setup
      • Verify the capacity of the Equinix Metal facility
      • Bootstrap a management cluster
    • Deploy an Equinix Metal based management cluster with private networking
      • Workflow overview
      • Prerequisites
      • Verify the capacity of the Equinix Metal facility
      • Bootstrap a management cluster
    • Deploy a VMware vSphere-based management cluster
      • Workflow overview
      • Deployment resources requirements
      • Prerequisites
      • Prepare the VMware deployment user setup and permissions
      • Bootstrap a management cluster
      • Prepare the OVF template
      • Configure squid-proxy
      • RHEL 8 mirrors configuration
    • Deploy an additional regional cluster (optional)
      • Configure the bootstrap node
      • Deploy an AWS-based regional cluster
      • Deploy an Azure-based regional cluster
      • Deploy an OpenStack-based regional cluster
      • Deploy an Equinix Metal based regional cluster with public networking
      • Deploy an Equinix Metal based regional cluster with private networking
      • Deploy a baremetal-based regional cluster
      • Deploy a VMware vSphere-based regional cluster
    • Requirements for a MITM proxy
    • Create initial users after a management cluster bootstrap
    • Troubleshooting
      • Collect the bootstrap logs
      • Troubleshoot the bootstrap node configuration
        • DNS settings
        • Default network address
      • Troubleshoot OpenStack-based deployments
        • TLS handshake timeout
      • Troubleshoot vSphere-based deployments
        • Virtual machine issues with obtaining an IP
    • Configure external identity provider for IAM
      • Configure LDAP for IAM
      • Configure Google OAuth IdP for IAM
  • Operations Guide
    • Mirantis Container Cloud CLI
    • Create and operate managed clusters
      • Create a project for managed clusters
      • Generate a kubeconfig for a managed cluster using API
      • Create and operate a baremetal-based managed cluster
        • Add a bare metal host
          • Add a bare metal host using web UI
          • Add a bare metal host using CLI
        • Create a custom bare metal host profile
          • Default configuration of the host system storage
          • Create a custom host profile
          • Enable huge pages
          • Configure RAID support
            • Create an LVM software RAID level 1 (raid1)
            • Create an mdadm software RAID level 1 (raid1)
            • Create an mdadm software RAID level 10 (raid10)
        • Add a managed baremetal cluster
          • Create a cluster using web UI
          • Workflow of network interface naming
          • Create subnets
            • Service labels and their life cycle
            • Create subnets for a managed cluster using web UI
            • Create subnets for a managed cluster using CLI
          • Automate multiple subnet creation using SubnetPool
          • Create L2 templates
            • L2 template example with bonds and bridges
            • L2 template example for automatic multiple subnet creation
        • Add a machine
          • Create a machine using web UI
          • Create a machine using CLI
            • Deploy a machine to a specific bare metal host
            • Assign L2 templates to machines
            • Override network interfaces naming and order
        • Add a Ceph cluster
        • Manage an existing bare metal cluster
          • Expand IP addresses capacity in an existing cluster
        • Manage machines of a bare metal cluster
          • Modify network configuration on an existing machine
        • Delete a managed cluster
        • Example of a complete L2 templates configuration for cluster creation
      • Create and operate an OpenStack-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Boot a machine from a block storage volume
        • Delete a managed cluster
      • Create and operate an AWS-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Azure-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Equinix Metal based managed cluster with public networking
        • Create a managed cluster
        • Increase BGP max_prefix
        • Add a machine
        • Delete a managed cluster
      • Create and operate an Equinix Metal based managed cluster with private networking
        • Create a managed cluster
        • Add a machine
        • Establish connection to the cluster private network
        • Delete a managed cluster
      • Create and operate a VMware vSphere-based managed cluster
        • Create a managed cluster
        • Add a machine
        • Delete a managed cluster
      • Add a CA certificate for a MITM proxy
      • Operate machine pools
        • Assign or unassign a machine to or from a machine pool
        • Change replicas count of a machine pool
        • Delete a machine pool
      • Change a cluster configuration
      • Update a managed cluster
        • Verify the Container Cloud status before managed cluster update
        • Update a managed cluster
    • Add a Container Cloud cluster to Lens
    • Attach an existing Mirantis Kubernetes Engine cluster
    • Connect to the Mirantis Kubernetes Engine web UI
    • Connect to a Mirantis Container Cloud cluster
    • Operate management and regional clusters
      • Automatic upgrade workflow
      • Schedule Mirantis Container Cloud upgrades
      • Renew the Container Cloud and MKE licenses
      • Configure NTP server for a regional cluster
      • Automatically propagate Salesforce configuration to all clusters
      • Configure TLS certificates for management cluster applications
      • Update the Keycloak IP address on bare metal clusters
      • Back up MariaDB on an AWS or OpenStack-based management cluster
        • Configure periodic backups of MariaDB for AWS and OpenStack providers
        • Verify operability of the MariaDB backup jobs
        • Restore MariaDB databases
      • Back up and restore a management cluster
      • Remove a management cluster
      • Remove a regional cluster
    • Increase CPU and memory limits for cluster components
    • Define a custom CA certificate for a private Docker registry
    • Enable cluster and machine maintenance mode
      • Enable maintenance mode on a cluster and machine using web UI
      • Enable maintenance mode on a cluster and machine using CLI
    • Delete a cluster machine
    • Manage IAM
      • Manage user roles through Container Cloud API
        • Available IAM roles and use cases
        • Mapping of Keycloak roles to IAM*RoleBinding objects
      • Manage user roles through the Container Cloud web UI
      • Manage user roles through Keycloak
        • Container Cloud roles and scopes
        • Use cases
        • Access the Keycloak Admin Console
      • Change passwords for IAM users
    • Manage StackLight
      • Access StackLight web UIs
      • View Grafana dashboards
      • View OpenSearch Dashboards
      • Available StackLight alerts
        • Alert dependencies
        • Alertmanager
        • Calico
        • Ceph
        • Docker Swarm
        • Elasticsearch Exporter
        • Etcd
        • External endpoint
        • Fluentd
        • General alerts
        • General node alerts
        • Grafana
        • Helm Controller
        • Ironic
        • Kubernetes applications
        • Kubernetes resources
        • Kubernetes storage
        • Kubernetes system
        • Metric Collector
        • Release Controller
        • NGINX
        • Node network
        • Node time
        • OpenSearch
        • PostgreSQL
        • Prometheus
        • Prometheus MS Teams
        • Prometheus Relay
        • Pushgateway
        • ServiceNow
        • Salesforce notifier
        • SMART disks
        • SSL certificates
        • Telegraf
        • Telemeter
        • Mirantis Kubernetes Engine
      • Troubleshoot alerts
        • Troubleshoot Kubernetes applications alerts
        • Troubleshoot Kubernetes resources alerts
        • Troubleshoot Kubernetes storage alerts
      • Silence alerts
      • Configure StackLight
        • StackLight configuration procedure
        • StackLight configuration parameters
        • Verify StackLight after configuration
      • Access OpenSearch clusters using Cerebro
      • Enable generic metric scraping
      • Enable remote logging to syslog
      • Create logs-based metrics
    • Manage Ceph
      • Ceph advanced configuration
      • Ceph default configuration options
      • Automated Ceph LCM
        • High-level workflow of Ceph OSD or node removal
          • Creating a Ceph OSD removal request
          • KaaSCephOperationRequest CR specification
          • KaaSCephOperationRequest status
        • Add, remove, or reconfigure Ceph OSDs
        • Add, remove, or reconfigure Ceph nodes
        • Replace a failed Ceph OSD
        • Replace a failed Ceph node
      • Move a Ceph Monitor daemon to another node
      • Migrate a Ceph Monitor before machine replacement
      • Update Ceph cluster
      • Enable Ceph RGW Object Storage
      • Enable Multisite for Ceph RGW Object Storage
      • Verify Ceph
        • Verify the Ceph core services
        • Verify rook-discover
        • Verify Ceph cluster state
          • Verify Ceph cluster state
          • KaaSCephCluster.status description
        • Verify Ceph Controller and Rook
      • Enable Ceph tolerations and resources management
        • Enable Ceph tolerations and resources management
        • Verify Ceph tolerations and resources management
      • Enable Ceph multinetwork
      • Enable TLS for Ceph public endpoints
      • Enable Ceph RBD mirroring
      • Enable Ceph Shared File System (CephFS)
      • Calculate target ratio for Ceph pools
      • Specify placement of Ceph cluster daemons
    • Troubleshooting
      • Collect cluster logs
      • Cluster deletion or detachment freezes
      • Nodes in Not Ready state on long-running clusters
      • Authentication failure with the 401 Unauthorized error
      • Helm releases get stuck in FAILED or UNKNOWN state during cluster deployment
      • Stuck kubelet on the Cluster release 5.x.x series
      • Equinix or MOSK managed clusters update fails with stuck kubelet
      • Troubleshoot AWS-based clusters
        • AWS-based managed cluster requiring PVs fails to be deployed
      • Troubleshoot baremetal-based clusters
        • Bare metal hosts in ‘provisioned registration error’ state after update
        • Troubleshoot an operating system upgrade with host restart
        • Troubleshoot iPXE boot issues
      • Troubleshoot Equinix Metal based clusters
        • Machine provisioning issues during cluster deployment with private network
      • Troubleshoot vSphere-based clusters
        • Node leaves the cluster after IP address change
      • Troubleshoot Ceph
        • Ceph disaster recovery
        • Cannot automatically remove a Ceph node
        • Ceph Monitors recovery
        • Remove Ceph OSD manually
        • KaaSCephOperationRequest failure with a timeout during rebalance
        • Ceph Monitors store.db size rapidly growing
      • Troubleshoot StackLight
        • Patroni replication lag
        • Alertmanager does not send resolve notifications for custom alerts
  • API Reference
    • Public key resources
    • License resource
    • IAM resources
    • MachinePool resource
    • AWS resources
      • Cluster
      • Machine
      • AWSCredential
      • AWSResources
    • Bare metal resources
      • Cluster
      • Machine
      • BareMetalHostProfile
      • BareMetalHost
      • IpamHost
      • Subnet
      • SubnetPool
      • IPaddr
      • L2Template
    • vSphere resources
      • VsphereCredential
      • Cluster
      • Machine
      • VsphereResources
    • OpenStack resources
      • Cluster
      • Machine
      • OpenStackCredential
      • OpenStackResources
    • MCCUpgrade resource
    • ContainerRegistry resource

Release notes

  • Release Compatibility Matrix
    • Compatibility matrix of components versions
    • Support matrix of MKE versions for cluster attachment
    • Container Cloud web UI browser compatibility
  • Release Notes
    • Mirantis Container Cloud releases
      • 2.18.0 (current)
        • Enhancements
        • Addressed issues
        • Known issues
        • Components versions
        • Artifacts
      • Unsupported releases
        • 2.17.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.16.1
        • 2.16.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • 2.15.1
        • 2.15.0
          • Enhancements
          • Addressed issues
          • Known issues
          • Components versions
          • Artifacts
        • Releases delivered in 2020-2021
          • 2.14.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.13.1
          • 2.13.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.12.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.11.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
            • Upgrade managed clusters with StackLight deployed in HA mode
          • 2.10.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.9.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
            • Switch L2 templates to the new format
          • 2.8.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.7.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.6.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.5.0
            • Enhancements
            • Addressed issues
            • Known issues
            • Components versions
            • Artifacts
          • 2.4.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.3.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.2.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
          • 2.1.0
            • Enhancements
            • Known issues
            • Addressed issues
            • Components versions
            • Artifacts
            • Apply updates to the AWS-based management clusters
          • 2.0.0
            • Known issues
            • Components versions
            • Artifacts
    • Cluster releases (managed)
      • 11.2.0
        • Enhancements
        • Components versions
        • Artifacts
      • 8.6.0
        • Enhancements
        • Components versions
        • Artifacts
      • 7.8.0
        • Enhancements
        • Components versions
        • Artifacts
      • Deprecated Cluster releases
        • 11.1.0
          • Enhancements
          • Components versions
          • Artifacts
        • 8.5.0
          • Enhancements
          • Components versions
          • Artifacts
        • 7.7.0
          • Enhancements
          • Components versions
          • Artifacts
      • Unsupported Cluster releases
        • 11.x series
          • 11.0.0
            • Enhancements
            • Components versions
            • Artifacts
        • 7.x series
          • 7.6.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.5.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.4.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.3.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.2.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.1.0
            • Enhancements
            • Components versions
            • Artifacts
          • 7.0.0
            • Enhancements
            • Components versions
            • Artifacts
        • 6.x series
          • 6.20.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.19.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.18.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 6.8.1
        • 5.x series
          • 5.22.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.21.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.20.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.19.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.18.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.17.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.16.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.15.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.14.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.13.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.12.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.11.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.10.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.9.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.8.0
            • Enhancements
            • Components versions
            • Artifacts
          • 5.7.0
            • Components versions
            • Artifacts
    • Deprecation notes

QuickStart guides

  • QuickStart guides overview
  • QuickStart: Container Cloud on AWS
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the AWS configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Microsoft Azure
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the Azure configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Equinix Metal with public networking
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Set up the Equinix Metal project
    • Verify the capacity of the Equinix Metal facility
    • Prepare the Equinix Metal configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on Equinix Metal with private networking
    • Before you begin
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Verify the capacity of the Equinix Metal facility
    • Prepare the Equinix Metal configuration
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on OpenStack
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the OpenStack configuration
    • Configure the cluster and machines metadata
    • Finalize the bootstrap
    • What’s next
  • QuickStart: Container Cloud on VMware vSphere
    • Before you begin
    • Prepare the bootstrap node
    • Download the bootstrap script
    • Obtain the Mirantis license
    • Prepare the deployment user setup and permissions
    • Configure the cluster and vSphere credentials
    • Prepare the OVF template
    • Finalize the bootstrap
    • What’s next

Troubleshoot alerts¶

Available since 2.18.0

This section describes the root causes, investigation, and mitigation steps for the available predefined StackLight alerts.

Note

The list of alerts in this section is not full and will be expanded.

  • Troubleshoot Kubernetes applications alerts
  • Troubleshoot Kubernetes resources alerts
  • Troubleshoot Kubernetes storage alerts
Next
Troubleshoot Kubernetes applications alerts
Previous
Mirantis Kubernetes Engine
  • Multi-page view
  • Single-page view

Mirantis Inc. 900 E Hamilton Avenue, Suite 650, Campbell, CA 95008 +1-650-963-9828

© 2005 - 2022 Mirantis, Inc. All rights reserved. "Mirantis" and "FUEL" are registered trademarks of Mirantis, Inc. All other trademarks are the property of their respective owners.