Migrate public endpoints to Envoy Proxy

This section describes the rolling migration of the MOSK cloud public endpoints from NGINX Ingress to Application Gateway (app-gateway) based on Envoy Proxy. The procedure is designed to minimize downtime for the public API.

Limitations

Custom settings applied to existing Ingress resources are not automatically translated to Application Gateway resource settings during migration. You might need to manually adjust the OpenStackDeployment object for your specific cloud.
By default, Envoy Proxy does not retry upstream connection resets. This behavior is not safe for non-idempotent requests.

Impact

During migration, the public endpoint IP address changes for the following components:

OpenStack public API endpoints
Ceph Object Storage public API endpoint
OpenSDN
OpenStack Designate PowerDNS endpoint

Prerequisites

Verify that at least one IP address is available in the MetalLB address pool. To identify the available IP addresses in the MetalLB address pool for a MOSK cluster:

On the management cluster, obtain the IP address pool from the MetalLBConfig object in your MOSK cluster namespace:

kubectl -n <MOSK_CLUSTER_NAMESPACE> get metallbconfig <CONFIG_NAME> -o jsonpath='{.spec.ipAddressPools}'

Example of system response:

[{"name":"default","spec":{"addresses":["172.16.56.151-172.16.56.250"],"autoAssign":true,"avoidBuggyIPs":false}}]

On the MOSK cluster, obtain the number of IP addresses used by LoadBalancer services:
```
kubectl get services -A --output=jsonpath='{.items[?(@.spec.type=="LoadBalancer")].status.loadBalancer.ingress[*].ip}' | tr ' \t' '\n' | uniq | wc -l
```
Example of system response:
```
7
```
In this example, the IP address pool contains 99 addresses and 7 are in use, leaving 92 available.

Perform pre-migration baseline checks

After you update to MOSK 26.2, your environment contains both the Ingress and Application Gateway services. To verify this, run the following command:

kubectl -n openstack get svc -l gateway.networking.k8s.io/gateway-name=app-gateway && \
kubectl -n openstack get svc -l application=ingress

Example of system response:

NAME          TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)                                   AGE
app-gateway   LoadBalancer   10.96.248.86   10.172.1.104   53:34443/TCP,53:34443/UDP,443:33513/TCP   50d
NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                      AGE
ingress               LoadBalancer   10.96.217.203   10.172.1.105   443:34616/TCP,10246:34498/TCP,53:34477/TCP   46d
ingress-error-pages   ClusterIP      None            <none>         80/TCP                                       46d
ingress-udp           LoadBalancer   10.96.190.235   10.172.1.105   53:35142/UDP                                 46d

Verify that public endpoints are available through the new IP address:

kubectl -n openstack get secret osh-dev-hidden -o jsonpath='{.data.ca_cert}' | base64 -d > ca.crt
ingresses=$(kubectl get ingress -A | awk '{print $4}' | grep -v HOSTS)
app_gateway_ip=$(kubectl -n openstack get svc app-gateway -o jsonpath='{.status.loadBalancer.ingress[*].ip}')

for i in $(echo $ingresses); do
    curl -s -o /dev/null --cacert ca.crt --max-time 5 --resolve $i:443:$app_gateway_ip https://$i/
    if [ $? -ne 0 ]; then
      echo "Host $i NOK"
    else
      echo "Host $i OK"
    fi
done

Example of system response:

Host barbican.<public_domain_name> OK
Host cinder.<public_domain_name> OK
Host cloudformation.<public_domain_name> OK
Host designate.<public_domain_name> OK
Host glance.<public_domain_name> OK
Host heat.<public_domain_name> OK
Host horizon.<public_domain_name> OK
Host keystone.<public_domain_name> OK
...
<list omitted>

If the OpenStack Designate PowerDNS endpoint was exposed through the Ingress service, verify that DNS resolves through the new Application Gateway service address:

nslookup test-record.test-zone.test $app_gateway_ip

Example of system response:

Server:         10.172.1.104
Address:        10.172.1.104#53

Name:   test-record.test-zone.test
Address: 1.2.3.4

Adjust the Application Gateway configuration

OpenStack public endpoints

The OpenStackDeployment object might contain custom settings for the NGINX Ingress configuration. Some of these settings can be translated to Application Gateway resource settings. For example, the following NGINX Ingress option is set through the OpenStackDeployment object for the Keystone API backend:

spec:
  services:
    keystone:
      values:
        network:
          api:
            ingress:
              annotations:
                nginx.ingress.kubernetes.io/proxy-read-timeout: "300"

This setting translates to:

spec:
  services:
    keystone:
      values:
        network:
          api:
            http_route:
              rule:
                timeouts:
                  request: 300s

Note

The OpenStackDeployment object service-level API is not stable. For advanced configuration use cases, contact Mirantis support.

Ceph public endpoints

The CephDeployment object might contain custom settings for the NGINX Ingress configuration. Some of these settings can be translated to Application Gateway resource settings. For example, the following NGINX Ingress option is set through the CephDeployment object for Ceph RGW access:

spec:
  ...
  objectStorage:
    objectStores:
      - name: openstack-store
  ...
  ingressConfig:
    annotations:
      nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
  ...

This setting translates to:

spec:
  ...
  objectStorage:
    objectStores:
      - name: openstack-store
    ...
    gatewayHTTPRoutes:
      - name: openstack-store-route
        objectStoreName: openstack-store
        spec:
          rules:
            timeouts:
              request: 300s

Note

For advanced configuration use cases, contact Mirantis support.

Perform pre-migration advanced checks

For more thorough testing, configure the /etc/hosts file on your local workstation to use the new IP address of the public endpoints. Open /etc/hosts and add the following entries:
```
<APP_GATEWAY_IP> barbican.<public_domain_name>
<APP_GATEWAY_IP> cinder.<public_domain_name>
...
<full list omitted>
```
Verify standard cloud operations by using CLI clients.
Access Horizon and verify standard cloud operations, including access to the various instance console types.

Migrate public endpoints

Change the DNS server configuration. The exact steps depend on your DNS server setup. If you use CoreDNS to resolve MOSK cluster public endpoints as described in Configure DNS to access OpenStack, complete the following steps:

Obtain the CoreDNS HelmBundle:

kubectl -n osh-system get helmbundle coredns -o yaml > coredns.yaml

Update the public endpoint IP address to the Application Gateway service IP address:

sed -i 's/<INGRESS_IP>/<APP_GATEWAY_IP>/' coredns.yaml
kubectl apply -f coredns.yaml

Wait for the CoreDNS pods to restart.

Verify the migration

Verify that the Ingress pods stopped receiving requests from external clients:

kubectl -n openstack logs -l application=ingress,component=server -c ingress

Verify that the Application Gateway pods started receiving requests from external clients:
```
kubectl -n openstack logs -l application=app_gateway
```
Remove the /etc/hosts entries that you added during the pre-migration advanced checks.
Verify that DNS resolution on your local workstation points to the DNS server used to resolve MOSK cluster public endpoints.
Verify that endpoints resolve to the Application Gateway service IP address:
```
nslookup keystone.<PUBLIC_DOMAIN_NAME>
```
Repeat the pre-migration advanced checks, excluding the /etc/hosts setup.

Clean up the cluster

Because the Ingress service in the openstack namespace also handles traffic for other components, complete the following cleanup procedure:

On the management cluster, in the Cluster object, locate the ceph-controller Helm release and disable Ingress:

helmReleases:
  - name: ceph-controller
    values:
      pelagia-ceph:
        ...
        lcmConfig:
          useIngress: false
        ...

Remove the ingressConfig section from the CephDeployment object, if present:
```
spec:
  ...
  ingressConfig:
    ...
```
Set useIngress to false in the TFOperator custom resource:
```
spec:
  features:
    useIngress: false
```
Remove the Ingress service and DaemonSets from the cluster by editing the OpenStackDeployment object:
```
spec:
  migration:
    ingress:
      state: absent
```
Wait until the OpenStackDeployment status changes to Applied.

Note

After the Ingress pods and service are removed from the openstack namespace, the OpenStack Ingress resources remain. These resources will be removed in a future release, when Ingress support is fully removed from MOSK.

Roll back the migration

Add the Ingress service and DaemonSets back to the cluster through the OpenStackDeployment object:
```
spec:
  migration:
    ingress:
      state: present
```
Wait until the OpenStackDeployment status changes to Applied.
Set useIngress to true in the TFOperator custom resource:
```
spec:
  features:
    useIngress: true
```

On the management cluster, in the Cluster object, locate the ceph-controller Helm release and remove useIngress: false:

helmReleases:
  - name: ceph-controller
    values:
      pelagia-ceph:
        ...
        lcmConfig:
          useIngress: false
        ...

Verify that the Ingress service is present in the openstack namespace:

kubectl -n openstack get svc -l application=ingress

Example of system response:

NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                                      AGE
ingress               LoadBalancer   10.96.217.203   10.172.1.105   443:34616/TCP,10246:34498/TCP,53:34477/TCP   46d
ingress-error-pages   ClusterIP      None            <none>         80/TCP                                       46d
ingress-udp           LoadBalancer   10.96.190.235   10.172.1.105   53:35142/UDP                                 46d

Verify that the Ingress resources are present in the rook-ceph namespace:
```
kubectl -n rook-ceph get ingress
```

Verify that the OpenStack, Ceph, and OpenSDN endpoints are available through the Ingress IP address:

kubectl -n openstack get secret osh-dev-hidden -o jsonpath='{.data.ca_cert}' | base64 -d > ca.crt
ingresses=$(kubectl get ingress -A | awk '{print $4}' | grep -v HOSTS)
ingress_ip=$(kubectl -n openstack get svc ingress -o jsonpath='{.status.loadBalancer.ingress[*].ip}')

for i in $(echo $ingresses); do
    curl -s -o /dev/null --cacert ca.crt --max-time 5 --resolve $i:443:$ingress_ip https://$i/
    if [ $? -ne 0 ]; then
      echo "Host $i NOK"
    else
      echo "Host $i OK"
    fi
done

Example of system response:

Host barbican.<public_domain_name> OK
Host cinder.<public_domain_name> OK
Host cloudformation.<public_domain_name> OK
Host designate.<public_domain_name> OK
Host glance.<public_domain_name> OK
Host heat.<public_domain_name> OK
Host horizon.<public_domain_name> OK
Host keystone.<public_domain_name> OK
Host manila.<public_domain_name> OK
Host metadata.<public_domain_name> OK
Host neutron.<public_domain_name> OK
Host nova.<public_domain_name> OK
Host novncproxy.<public_domain_name> OK
Host octavia.<public_domain_name> OK
Host placement.<public_domain_name> OK
Host spiceproxy.<public_domain_name> OK
Host openstack-store.<public_domain_name> OK

Change the DNS server configuration. The exact steps depend on your DNS server setup. If you use CoreDNS to resolve MOSK cluster public endpoints as described in Configure DNS to access OpenStack, complete the following steps:
1. Obtain the CoreDNS HelmBundle:
```
kubectl -n osh-system get helmbundle coredns -o yaml > coredns.yaml
```
2. Update the public endpoint IP address to the Ingress service IP address:
```
sed -i 's/<APP_GATEWAY_IP>/<INGRESS_IP>/' coredns.yaml
kubectl apply -f coredns.yaml
```
3. Wait for the CoreDNS pods to restart.