Ceph known issues

This section lists the Ceph known issues with workarounds for the Mirantis OpenStack for Kubernetes release 21.3.


[16229] Ceph Controller failure during update

During the MOS cluster update to Cluster release 6.16.0, the Ceph Controller may fail with the following traceback:

panic: runtime error: invalid memory address on nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x15d2c58]

goroutine 352 [running]:
github.com/Mirantis/ceph-controller/pkg/controller/miraceph.VerifyCertificateExpireDate(0x0, 0x0, 0x0, 0x6, 0x28a7bc0)
        ceph-controller/pkg/controller/miraceph/util.go:250 +0x48

Workaround:

  1. Obtain the cert base64-encoded value of the rook-ceph/rgw-ssl-certificate secret:

    kubectl -n rook-ceph get secret rgw-ssl-certificate -o jsonpath='{.data.cert}' | base64 -d
    

    Example of system response:

    -----BEGIN RSA PRIVATE KEY-----
    ...
    -----END RSA PRIVATE KEY-----
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
    
  2. Copy last certificate in the chain and save it to the temp file, for example, tmp-cacert.crt.

  3. Encode the certificate from tmp-cacert.crt with base64 encoding in one line:

    cat tmp-cacert.crt | base64 -w 0
    
  4. Create a new cacert key in the rook-ceph/rgw-ssl-certificate secret and copy the base64-encoded cacert to its value. The following is an example of the resulting secret data:

    data:
      cert: <base64 string>
      cacert: <copied base64 cacert string>
    
  5. Restart the ceph-lcm-mirantis/ceph-controller pod:

    kubectl -n ceph-lcm-mirantis delete pod -l app=ceph-controller