Configure Ceph Object Gateway TLS¶

Once you enable Ceph Object Gateway (radosgw) as described in Enable Ceph RGW Object Storage, you can configure the Transport Layer Security (TLS) protocol for a Ceph Object Gateway public endpoint using the following options:

Using MOSK TLS, if it is enabled and exposes its certificates and domain for Ceph. In this case, Ceph Object Gateway will automatically create an ingress rule with MOSK certificates and domain to access the Ceph Object Gateway public endpoint. Therefore, you only need to reach the Ceph Object Gateway public and internal endpoints and set the CA certificates for a trusted TLS connection.
Using custom ingress specified in the KaaSCephCluster CR. In this case, Ceph Object Gateway public endpoint will use the public domain specified using the ingress parameters.

Caution

External Ceph Object Gateway service is not supported and will be deleted during update. If your system already uses endpoints of an external Ceph Object Gateway service, reconfigure them to the ingress endpoints.

Caution

When using a custom or OpenStack ingress, ensure to configure the DNS name for RGW to target an external IP address of that ingress. If there is no OpenStack or custom ingress available, point the DNS to an external load balancer of RGW.

Note

Since MOSK 23.3, if the cluster has tls-proxy enabled, TLS certificates specified in ingress objects, including those configured in the KaaSCephCluster specification, are disregarded. Instead, common certificates are applied to all ingresses from the OpenStackDeployment object. This implies that tlsCert and other ingress certificates specified in KaaSCephCluster are ignored, and the common certificate from the OpenStackDeployment object is used.

This section also describes how to specify a custom public endpoint for the Object Storage service.

To configure Ceph Object Gateway TLS:

Verify whether MOSK TLS is enabled. The spec.features.ssl.public_endpoints section should be specified in the OpenStackDeployment CR.
To generate an SSL certificate for internal usage, verify that the gateway securePort parameter is specified in the KaasCephCluster CR. For details, see Enable Ceph RGW Object Storage.

Select from the following options:

Since MOSK 25.1

Configure TLS for Ceph Object Gateway using a custom ingressConfig:

Open the KaasCephCluster CR for editing.

Specify the ingressConfig parameters:

Description of the tlsConfig section parameters¶
`certs`	TLS configuration for ingress including certificates. Contains the following parameters: `cacert` The Certificate Authority (CA) certificate, used for the ingress rule TLS support. `tlsCert` The TLS certificate, used for the ingress rule TLS support. `tlsKey` The TLS private key, used for the ingress rule TLS support.
`publicDomain`	Mandatory. The domain name to use for public endpoints. Caution The default ingress controller does not support `publicDomain` values different from the OpenStack ingress public domain. Therefore, if you intend to use the default OpenStack Ingress Controller for your Ceph Object Storage public endpoint, plan to use the same public domain as your OpenStack endpoints.
`hostname`	Custom name to override the Objectstore RGW name for public RGW access. Public RGW endpoint has the `https://<hostname>.<publicDomain>` format.
`tlsSecretRefName`	Optional. Secret name with TLS certs on the managed cluster in the `rook-ceph` namespace prepared by the operator. Allows avoiding exposure of certs directly in `spec`. Must contain the following format: data: ca.cert: <base64encodedCaCertificate> tls.crt: <base64encodedTlsCert> tls.key: <base64encodedTlsKey> Caution When using `tlsSecretRefName`, remove the following fields: `cacert`, `tlsCert`, and `tlsKey`.

Description of optional parameters in the ingressConfig section¶
`controllerClassName`	Name of the custom Ingress Controller. By default, the `openstack-ingress-nginx` class name is specified and Ceph uses the OpenStack Ingress Controller based on NGINX.
`annotations`	Extra annotations for the ingress proxy that are a key-value mapping of strings to add or override ingress rule annotations. For details, see NGINX Ingress Controller: Annotations. By default, the following annotations are set: `nginx.ingress.kubernetes.io/rewrite-target` is set to `/` `nginx.ingress.kubernetes.io/upstream-vhost` is set to `<rgwName>.rook-ceph.svc` The value for `<rgwName>` is located in `spec.cephClusterSpec.objectStorage.rgw.name`. Optional annotations: `nginx.ingress.kubernetes.io/proxy-request-buffering: "off"` that disables buffering for `ingress` to prevent the 413 (Request Entity Too Large) error when uploading large files using `radosgw`. `nginx.ingress.kubernetes.io/proxy-body-size: <size>` that increases the default uploading size limit to prevent the 413 (Request Entity Too Large) error when uploading large files using `radosgw`. Set the value in MB (`m`) or KB (`k`). For example, `100m`. Note By default, an ingress rule is created with an internal Ceph Object Gateway service endpoint as a backend. Also, `rgw dns name` is specified in the Ceph configuration and is set to `<rgwName>.rook-ceph.svc` by default. You can override `rgw dns name` using the `spec.cephClusterSpec.rookConfig` key-value parameter. In this case, also change the corresponding ingress annotation. Configuration example with the `rgw dns name` override spec: cephClusterSpec: objectStorage: rgw: name: rgw-store ingressConfig: tlsConfig: publicDomain: public.domain.name certs: cacert: \| -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- tlsCert: \| -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- tlsKey: \| -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----- controllerClassName: openstack-ingress-nginx annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/upstream-vhost: rgw-store.public.domain.name nginx.ingress.kubernetes.io/proxy-body-size: 100m rookConfig: "rgw dns name": rgw-store.public.domain.name For clouds with the `publicDomain` parameter specified, align the `upstream-vhost` ingress annotation with the name of the Ceph Object Storage and the specified public domain. Ceph Object Storage requires the `upstream-vhost` and `rgw dns name` parameters to be equal. Therefore, override the default `rgw dns name` with the corresponding ingress annotation value.

Before MOSK 25.1

Configure Ceph Object Gateway TLS using a custom ingress:

Warning

The rgw section is deprecated and the ingress parameters are moved under cephClusterSpec.ingress. If you continue using rgw.ingress, it will be automatically translated into cephClusterSpec.ingress during the MOSK cluster release update.

Open the KaasCephCluster CR for editing.
Specify the ingress parameters:
- publicDomain - domain name to use for the external service.
  
  Caution
  
  Since MOSK 23.3, the default ingress controller does not support publicDomain values different from the OpenStack ingress public domain. Therefore, if you intend to use the default OpenStack ingress controller for your Ceph Object Storage public endpoint, plan to use the same public domain as your OpenStack endpoints.
- cacert - Certificate Authority (CA) certificate, used for the ingress rule TLS support.
- tlsCert - TLS certificate, used for the ingress rule TLS support.
- tlsKey - TLS private key, used for the ingress rule TLS support.
- customIngress ^Optional - includes the following custom Ingress Controller parameters:
  - className - the custom Ingress Controller class name. If not specified, the openstack-ingress-nginx class name is used by default.
  - annotations - extra annotations for the ingress proxy. For details, see NGINX Ingress Controller: Annotations.
    
    By default, the following annotations are set:
    - nginx.ingress.kubernetes.io/rewrite-target is set to /
    - nginx.ingress.kubernetes.io/upstream-vhost is set to <rgwName>.rook-ceph.svc.
      
      The value for <rgwName> is spec.cephClusterSpec.objectStorage.rgw.name.
    Optional annotations:
    - nginx.ingress.kubernetes.io/proxy-request-buffering: "off" that disables buffering for ingress to prevent the 413 (Request Entity Too Large) error when uploading large files using radosgw.
    - nginx.ingress.kubernetes.io/proxy-body-size: <size> that increases the default uploading size limit to prevent the 413 (Request Entity Too Large) error when uploading large files using radosgw. Set the value in MB (m) or KB (k). For example, 100m.
    For example:
    customIngress: className: openstack-ingress-nginx annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/upstream-vhost: openstack-store.rook-ceph.svc nginx.ingress.kubernetes.io/proxy-body-size: 100m
    Note
    
    An ingress rule is by default created with an internal Ceph Object Gateway service endpoint as a backend. Also, rgw dns name is specified in the Ceph configuration and is set to <rgwName>.rook-ceph.svc by default. You can override this option using the spec.cephClusterSpec.rookConfig key-value parameter. In this case, also change the corresponding ingress annotation.
    
    For example:
    spec: cephClusterSpec: objectStorage: rgw: name: rgw-store ingress: publicDomain: public.domain.name cacert: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- tlsCert: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- tlsKey: | -----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY----- customIngress: annotations: "nginx.ingress.kubernetes.io/upstream-vhost": rgw-store.public.domain.name rookConfig: "rgw dns name": rgw-store.public.domain.name
    Warning
    - For clouds with the publicDomain parameter specified, align the upstream-vhost ingress annotation with the name of the Ceph Object Storage and the specified public domain.
    - Ceph Object Storage requires the upstream-vhost and rgw dns name parameters to be equal. Therefore, override the default rgw dns name to the corresponding ingress annotation value.

If MOSK TLS is enabled

Obtain the MOSK CA certificate for a trusted connection:

kubectl -n openstack-ceph-shared get secret openstack-rgw-creds -o jsonpath="{.data.ca_cert}" | base64 -d

Access internal and public Ceph Object Gateway endpoints by selecting one of the following options:

Note

If you are using the HTTP scheme instead of HTTPS:

Since MOSK 25.1

In the KaaSCephCluster object on the management cluster, find the ingressConfig section and add custom annotations:

spec:
  cephClusterSpec:
    ingressConfig:
      annotations:
        "nginx.ingress.kubernetes.io/force-ssl-redirect": "false"
        "nginx.ingress.kubernetes.io/ssl-redirect": "false"

You can omit TLS configuration for the default settings provided by OpenStack to be applied.

If both HTTP and HTTPS must be used, apply the following configuration in the KaaSCephCluster object:

spec:
  cephClusterSpec:
    ingressConfig:
      tlsConfig:
        publicDomain: public.domain.name
        cacert: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
        tlsCert: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
        tlsKey: |
          -----BEGIN RSA PRIVATE KEY-----
          ...
          -----END RSA PRIVATE KEY-----
      annotations:
        "nginx.ingress.kubernetes.io/force-ssl-redirect": "false"
        "nginx.ingress.kubernetes.io/ssl-redirect": "false"

Since MOSK 23.3

In the KaaSCephCluster object on the management cluster, find the ingress section and add custom annotations:

spec:
  cephClusterSpec:
    ingress:
      publicDomain: public.domain.name
      cacert: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      tlsCert: |
        -----BEGIN CERTIFICATE-----
        ...
        -----END CERTIFICATE-----
      tlsKey: |
        -----BEGIN RSA PRIVATE KEY-----
        ...
        -----END RSA PRIVATE KEY-----
      customIngress:
        annotations:
          "nginx.ingress.kubernetes.io/force-ssl-redirect": "false"
          "nginx.ingress.kubernetes.io/ssl-redirect": "false"

Before MOSK 23.3

In the KaaSCephCluster object on the management cluster, find the ingress section and add custom annotations:

spec:
  cephClusterSpec:
    rookConfig:
      rgw_remote_addr_param: "HTTP_X_FORWARDED_FOR"

For a public endpoint

Obtain the Ceph Object Gateway public endpoint:
```
kubectl -n rook-ceph get ingress
```

Obtain the public endpoint TLS CA certificate:

kubectl -n rook-ceph get secret $(kubectl -n rook-ceph get ingress -o jsonpath='{.items[0].spec.tls[0].secretName}{"\n"}') -o jsonpath='{.data.ca\.crt}' | base64 -d; echo

For an internal endpoint

Obtain the internal endpoint name for Ceph Object Gateway:
```
kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
```
The internal endpoint for Ceph Object Gateway has the https://<internal-svc-name>.rook-ceph.svc:<rgw-secure-port>/ format, where <rgw-secure-port> is spec.rgw.gateway.securePort specified in the KaaSCephCluster CR.

Obtain the internal endpoint TLS CA certificate:

kubectl -n rook-ceph get secret rgw-ssl-certificate -o jsonpath="{.data.cacert}" | base64 -d

Update the zonegroup hostnames of Ceph Object Gateway:
MOSK 22.5 and later
The hostnames zonegroup of Ceph Object Gateway updates automatically and you can skip this step if one of the following requirements is met:
- The public hostname matches the public domain name set by the spec.cephClusterSpec.ingressConfig.tlsConfig.publicDomain field
- The OpenStack configuration has been applied
Otherwise, complete the steps for MOSK 22.4 and earlier product versions.

Note

Prior to MOSK 25.1, use the spec.cephClusterSpec.ingress.publicDomain field instead.
MOSK 22.4 and earlier
1. Enter the rook-ceph-tools pod:
  kubectl -n rook-ceph exec -it deployment/rook-ceph-tools -- bash
2. Obtain Ceph Object Gateway default zonegroup configuration:
  radosgw-admin zonegroup get --rgw-zonegroup=<objectStorageName> --rgw-zone=<objectStorageName> | tee zonegroup.json
  Substitute <objectStorageName> with the Ceph Object Storage name from spec.cephClusterSpec.objectStorage.rgw.name.
3. Inspect zonegroup.json and verify that the hostnames key is a list that contains two endpoints: an internal endpoint and a custom public endpoint:
  "hostnames": ["rook-ceph-rgw-<objectStorageName>.rook-ceph.svc", <customPublicEndpoint>]
  Substitute <objectStorageName> with the Ceph Object Storage name and <customPublicEndpoint> with the public endpoint with a custom public domain.
4. If one or both endpoints are omitted in the list, add the missing endpoints to the hostnames list in the zonegroup.json file and update Ceph Object Gateway zonegroup configuration:
  radosgw-admin zonegroup set --rgw-zonegroup=<objectStorageName> --rgw-zone=<objectStorageName> --infile zonegroup.json radosgw-admin period update --commit
5. Verify that the hostnames list contains both the internal and custom public endpoint:
  radosgw-admin --rgw-zonegroup=<objectStorageName> --rgw-zone=<objectStorageName> zonegroup get | jq -r ".hostnames"
  Example of system response:
  [ "rook-ceph-rgw-obj-store.rook-ceph.svc", "obj-store.mcc1.cluster1.example.com" ]
6. Exit the rook-ceph-tools pod:
  exit

Once done, Ceph Object Gateway becomes available by the custom public endpoint with an S3 API client, OpenStack Swift CLI, and OpenStack Horizon Containers plugin.