Ceph known issues¶

This section lists the Ceph known issues with workarounds for the Mirantis OpenStack for Kubernetes release 22.3.

[28783] Ceph conditon stuck in absence of Ceph cluster secrets info¶

Ceph conditon gets stuck in absence of the Ceph cluster secrets information. The observed behaviour can be found on the MOSK clusters that have automatically updated their management cluster to Container Cloud 2.21 but are still running the MOSK 22.3 version.

The list of the symptoms includes:

The Cluster object contains the following condition:

Failed to configure Ceph cluster: ceph cluster status info is not \
updated at least for 5 minutes, ceph cluster secrets info is not available yet

The ceph-kcc-controller logs from the kaas namespace contain the following loglines:

2022-11-30 19:39:17.393595 E | ceph-spec: failed to update cluster condition to \
{Type:Ready Status:True Reason:ClusterCreated Message:Cluster created successfully \
LastHeartbeatTime:2022-11-30 19:39:17.378401993 +0000 UTC m=+2617.717554955 \
LastTransitionTime:2022-05-16 16:14:37 +0000 UTC}. failed to update object \
"rook-ceph/rook-ceph" status: Operation cannot be fulfilled on \
cephclusters.ceph.rook.io "rook-ceph": the object has been modified; please \
apply your changes to the latest version and try again

Workaround:

Edit KaaSCephCluster of the affected managed cluster:
```
kubectl -n <managedClusterProject> edit kaascephcluster
```
Substitute <managedClusterProject> with the corresponding managed cluster namespace.
Define the version parameter in the KaaSCephCluster spec:
```
spec:
  cephClusterSpec:
    version: 15.2.13
```
Note

Starting from MOSK 22.4, the Ceph cluster version updates to 15.2.17. Therefore, remove the version parameter definition from KaaSCephCluster after the managed cluster update.

Save the updated KaaSCephCluster spec.

Find the MiraCeph Custom Resource on a managed cluster and copy all annotations starting with meta.helm.sh:

kubectl --kubeconfig <managedClusterKubeconfig> get crd miracephs.lcm.mirantis.com -o yaml

Substitute <managedClusterKubeconfig> with a corresponding managed cluster kubeconfig.

Example of a system output:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.0
    # save all annotations with "meta.helm.sh" somewhere
    meta.helm.sh/release-name: ceph-controller
    meta.helm.sh/release-namespace: ceph
...

Create the miracephsecretscrd.yaml file and fill it with the following template:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.0
    <insert all "meta.helm.sh" annotations here>
  labels:
    app.kubernetes.io/managed-by: Helm
  name: miracephsecrets.lcm.mirantis.com
spec:
  conversion:
    strategy: None
  group: lcm.mirantis.com
  names:
    kind: MiraCephSecret
    listKind: MiraCephSecretList
    plural: miracephsecrets
    singular: miracephsecret
  scope: Namespaced
  versions:
    - name: v1alpha1
      schema:
        openAPIV3Schema:
          description: MiraCephSecret aggregates secrets created by Ceph
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
            status:
              properties:
                lastSecretCheck:
                  type: string
                lastSecretUpdate:
                  type: string
                messages:
                  items:
                    type: string
                  type: array
                state:
                  type: string
              type: object
          type: object
      served: true
      storage: true

Insert the copied meta.helm.sh annotations to the metadata.annotations section of the template.

Apply miracephsecretscrd.yaml on the managed cluster:
```
kubectl --kubeconfig <managedClusterKubeconfig> apply -f miracephsecretscrd.yaml
```
Substitute <managedClusterKubeconfig> with a corresponding managed cluster kubeconfig.
Obtain the MiraCeph name from the managed cluster:
```
kubectl --kubeconfig <managedClusterKubeconfig> -n ceph-lcm-mirantis get miraceph -o name
```
Substitute <managedClusterKubeconfig> with the corresponding managed cluster kubeconfig.

Example of a system output:
```
miraceph.lcm.mirantis.com/rook-ceph
```
Copy the MiraCeph name after slash, the rook-ceph part from the example above.

Create the mcs.yaml file and fill it with the following template:

apiVersion: lcm.mirantis.com/v1alpha1
kind: MiraCephSecret
metadata:
  name: <miracephName>
  namespace: ceph-lcm-mirantis
status: {}

Substitute <miracephName> with the MiraCeph name from the previous step.

Apply mcs.yaml on the managed cluster:
```
kubectl --kubeconfig <managedClusterKubeconfig> apply -f mcs.yaml
```
Substitute <managedClusterKubeconfig> with a corresponding managed cluster kubeconfig.

After some delay, the cluster condition will be updated to the health state.