Configure token-based SSH access to MOSK using Vault and Keycloak

This section describes the process of configuring token-based SSH access to MOSK cluster nodes using HashiCorp Vault and Keycloak.

Prerequisites

This blueprint requires an existing, production-ready corporate Vault deployment. A local Helm deployment can be used for simple testing only. For production environments, verify the following:

Vault is hosted on a dedicated, hardened cluster, for example, HashiCorp Vault HCP.
Vault is configured with high availability (HA) raft clustering, persistent volume claims (PVCs), and valid TLS certificates.
The vault CLI binary is installed on your local machine to be used for implementation of this blueprint.
You are authenticated to Vault with administrative privileges.
You have jq and kubectl installed on your local machine.

Step 1. Prepare the environment variables

In MOSK environments, Keycloak credentials and TLS CA certificates are stored in Kubernetes secrets within the kaas namespace. Extract these to your local machine that contains the tools described in requirements and has SSH access to the target MOSK cluster:

export KEYCLOAK_USER="keycloak"
export KEYCLOAK_PASS=$(kubectl -n kaas get secret iam-api-secrets \
    -o jsonpath='{.data.keycloak_password}' | base64 -d)

# Extract Keycloak CA for Vault TLS discovery
kubectl -n kaas get secret iam-api-secrets \
    -o jsonpath='{.data.keycloakCA\.pem}' \
    | base64 -d > /tmp/keycloakCA.pem
export KEYCLOAK_CA=$(cat /tmp/keycloakCA.pem)

# Discover Keycloak service address
export KEYCLOAK_IP=$(kubectl -n kaas get svc iam-keycloak-http \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export KEYCLOAK_ADDR="https://${KEYCLOAK_IP}"

Step 2. Initialize and configure the Keycloak server

Important

By default, Keycloak logs you into the master realm. You must switch to the iam realm before proceeding.

In the top-left corner of the Keycloak admin console, click the realm dropdown or navigate to Manage realms and select iam.

All subsequent client, group, and user configurations must be performed within this realm.

Configure the Vault client

In the Keycloak admin console, navigate to Clients > Create client.
Configure the following attributes:
- Client ID: vault
- Client authentication: ON
- Valid Redirect URIs: add the following URIs as two separate entries by pasting the first URI, pressing Enter, then pasting the second URI:
  - https://<VAULT_DOMAIN>:8200/ui/vault/auth/oidc/oidc/callback
  - http://localhost:8250/oidc/callback
Save and copy the Client Secret from the Credentials tab, then export it in your shell session:
```
read -rsp "Keycloak client secret: " KEYCLOAK_CLIENT_SECRET; echo
export KEYCLOAK_CLIENT_SECRET
```
Note

Using read keeps the client secret out of the shell history file.
Navigate to the Client scopes tab inside your vault client settings.

Caution

Do not use the global Client scopes menu on the left sidebar.
Click the vault-dedicated scope link in the list.
Click Configure a new mapper and select Group Membership from the list.

Note

By default, Keycloak does not include group memberships inside OIDC tokens. Configuring this mapper explicitly injects a groups claim into the ID token and UserInfo payload, enabling Vault to read Keycloak groups and evaluate identity policies.
Configure the mapper properties as follows and leave all other options unchanged:
- Name: groups
- Token Claim Name: groups
- Full group path: OFF
- Add to ID token: ON
- Add to userinfo: ON

Configure realm groups and users

Note

For demonstration purposes, this blueprint uses predefined test users and two specific groups (ssh-operators and default-users) to illustrate how to effectively split permissions using role-based access control (RBAC). In a real-world scenario, you can create and map your own custom users and groups in Keycloak tailored to your organizational needs.

To configure this demonstration setup, create the required groups and assign the existing users accordingly:

In the Keycloak admin console, navigate to Groups and click Create group. Create two groups: ssh-operators and default-users.
Open the default-users group and navigate to the Members tab.
Add all predefined users to this group, for example, operator, reader, serviceuser, stacklight, writer.
Open the ssh-operators group and navigate to the Members tab.
Add only the writer and operator users to this group. These users are granted SSH access privileges.
Navigate to Realm settings > User registration > Default groups and add default-users as the default group for the realm so that any new users automatically inherit it.

Enforce multi-factor authentication (MFA)

To enhance security, when logging in to Vault, operators must use an authenticator application, for example, FreeOTP or Google Authenticator.

To configure OTP enforcement:

In the left-hand menu of the Keycloak admin console, navigate to Authentication.
In the Required actions tab, ensure that Configure OTP is enabled.
Enforce MFA for a specific user:
1. Navigate to Users.
2. Select the target user, for example, operator.
3. Add Configure OTP to their Required user actions.
4. Click Save.
Ensure that Keycloak prompts for the OTP during login:
1. Navigate to Authentication > Flows > Browser.
2. Verify that the OTP step is set to Required or Conditional.

First-time login and subsequent authentications

During the initial login attempt using OIDC, Keycloak interrupts the authentication flow and prompts the user to register an authenticator application using a QR code.

On subsequent logins, once the session expires, Keycloak prompts the user to enter the generated 6-digit one-time code alongside their standard credentials.

Keycloak uses single sign-on (SSO) session cookies. After a successful login, Keycloak establishes an active session. Subsequent CLI or UI login attempts automatically authorize the user without prompting for an OTP until the session expires.

To force cookies to expire sooner and mandate re-authentication, in the Keycloak admin console, navigate to Realm settings > Sessions and adjust the SSO Session Idle and SSO Session Max values to a shorter duration.

Step 3. Configure SSH certificates in Vault

To configure Vault to act as a certificate authority (CA) for SSH certificates:

Enable the SSH secrets engine:

vault secrets enable -path=ssh-client-signer ssh

Generate a CA signing key:

vault write ssh-client-signer/config/ca generate_signing_key=true

Create the demo signing role:

vault write ssh-client-signer/roles/demo \
    allow_user_certificates=true \
    allowed_users="mcc-user" \
    allowed_extensions="permit-pty,permit-port-forwarding" \
    default_extensions="permit-pty=" \
    key_type="ca" \
    default_user="mcc-user" \
    ttl="5m0s" \
    max_ttl="10m0s"

Step 4. Configure OIDC integration and RBAC in Vault

Enable the OIDC method by injecting the Keycloak CA so that Vault trusts the HTTPS endpoint:

vault auth enable oidc
vault write auth/oidc/config \
    oidc_discovery_url="${KEYCLOAK_ADDR}/auth/realms/iam" \
    oidc_client_id="vault" \
    oidc_client_secret="<KEYCLOAK_CLIENT_SECRET>" \
    default_role="mosk-users" \
    oidc_discovery_ca_pem="${KEYCLOAK_CA}"

Create the ssh-allow ACL policy:

vault policy write ssh-allow -<<"EOH"
path "ssh-client-signer/roles" { capabilities = ["list"] }
path "ssh-client-signer/roles/*" { capabilities = ["read"] }
path "ssh-client-signer/issue/demo" { capabilities = ["create", "update"] }
path "ssh-client-signer/sign/demo" { capabilities = ["create", "update"] }
path "ssh-client-signer/config/ca" { capabilities = ["read"] }
EOH

Create the OIDC role that processes group claims:

export CB1="${VAULT_ADDR}/ui/vault/auth/oidc/oidc/callback"
export CB2="http://localhost:8250/oidc/callback"

vault write auth/oidc/role/mosk-users \
    bound_audiences="vault" \
    allowed_redirect_uris="${CB1},${CB2}" \
    user_claim="preferred_username" \
    groups_claim="groups" \
    token_policies="default" \
    role_type="oidc"

This OIDC role informs Vault how to extract identity information from the Keycloak token. It explicitly maps the preferred_username claim to the Vault user identity and maps the groups claim into the Vault internal group identity system.

Map Keycloak groups to Vault Identity. To apply Vault policies to Keycloak groups, map the external OIDC group (ssh-operators) to an internal Vault identity group:

Accessor (OIDC_ACCESSOR): Vault internal unique identifier for a specific authentication mount point.
Group alias: Links the external Keycloak group name to the Vault identity group, granting its members the mapped ssh-allow policy.

OIDC_ACCESSOR=$(vault auth list -format=json | jq -r '."oidc/".accessor')
GROUP_ID=$(vault write -format=json identity/group \
    name="vault-ssh-operators" type="external" policies="ssh-allow" \
    | jq -r '.data.id')
vault write identity/group-alias \
    name="ssh-operators" mount_accessor="$OIDC_ACCESSOR" \
    canonical_id="$GROUP_ID"

Step 5. Configure MOSK cluster nodes (SSHD)

Configure the target Linux servers (MOSK nodes) to trust the Vault SSH CA. Extract the key locally from Vault and distribute it using Secure Copy Protocol (SCP).

Note

To automate the distribution of the CA public key and the SSH daemon configuration across a large fleet of MOSK nodes, you can use the host operating system configuration framework. It is the operator’s responsibility to create a custom HostOSConfigurationModules object for this purpose. For reference, see HostOSConfiguration and HostOSConfigurationModules concepts.

Extract the CA public key from Vault:

vault read -field=public_key ssh-client-signer/config/ca \
    > trusted-user-ca-keys.pem

Copy the public key to the target MOSK cluster node using SCP:

scp trusted-user-ca-keys.pem mcc-user@<TARGET_HOST_IP>:/tmp/

Move the key into the SSH directory and set the following permissions:

sudo mv /tmp/trusted-user-ca-keys.pem /etc/ssh/trusted-user-ca-keys.pem
sudo chown root:root /etc/ssh/trusted-user-ca-keys.pem
sudo chmod 644 /etc/ssh/trusted-user-ca-keys.pem

Update the SSH daemon configuration in /etc/ssh/sshd_config to trust the CA public key:

echo "TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" \
    | sudo tee -a /etc/ssh/sshd_config

Restart the SSH daemon:
```
systemctl restart sshd
```

Step 6. Obtain SSH certificates and verify RBAC

Operators can request short-lived SSH certificates from Vault to securely access target nodes using the Vault CLI, the Vault UI, or the vault ssh wrapper. The first two methods produce a certificate file that you then inspect and use to log in, as described further in this section. The vault ssh wrapper establishes the SSH connection itself. This section also demonstrates how Keycloak group mappings enforce strict role-based access control (RBAC).

Sign an SSH certificate using the CLI

Log in using OIDC:
```
vault login -method=oidc
```
Generate a keypair and request a certificate:

Note

For demonstration purposes, this example generates a dedicated SSH key (vault_rsa) to avoid overwriting your default keys. Note the use of $HOME in the Vault command to resolve the path correctly.
```
ssh-keygen -t rsa -b 2048 -f ~/.ssh/vault_rsa -N ""
vault write -field=signed_key ssh-client-signer/sign/demo \
    public_key=@$HOME/.ssh/vault_rsa.pub \
    valid_principals="mcc-user" > ~/.ssh/vault_rsa-cert.pub
```
Important

The OpenSSH client strictly requires the certificate file to be named in the format <private_key_filename>-cert.pub and located in the same directory as the private key. For example, if your private key is vault_rsa, the certificate must be named vault_rsa-cert.pub. If they have different names, the SSH client fails to authenticate, even if you explicitly pass the -o CertificateFile= argument.

Sign an SSH certificate using the Vault UI

Create a dedicated keypair:

ssh-keygen -t rsa -b 2048 -f ~/.ssh/vault_rsa -N ""

In a browser, navigate to the Vault UI using https://<VAULT_DOMAIN>:8200/ui. Set the authentication Method to OIDC and click Sign in with OIDC Provider.
Sign the public key:
1. Navigate to Secrets > ssh-client-signer > Roles.
2. Select the demo role.
3. Configure the form parameters:
  - Public Key: paste the contents of your local ~/.ssh/vault_rsa.pub file.
  - Valid Principals: enter mcc-user.
4. Click Sign.
5. Capture the generated certificate block.
Save the copied certificate into ~/.ssh/vault_rsa-cert.pub alongside your private key.

Sign an SSH certificate using vault ssh

The vault ssh wrapper transparently handles key generation, signing, and SSH connection in a single step.

vault login -method=oidc
vault ssh -mode=ca -role=demo -mount-point=ssh-client-signer/ \
    mcc-user@<TARGET_HOST_IP>

Note

Under the hood, the vault ssh command generates an ephemeral in-memory keypair and writes temporary files to your local /tmp/ directory (following the pattern /tmp/vault_ssh_ca_<username>_<target_ip><random_suffix>). It executes the system ssh binary using these temporary credentials and automatically deletes them when the SSH session is terminated.

Inspect the signed certificate

Before connecting, inspect the signed certificate to understand exactly what parameters and security boundaries Vault embedded in it.

Run the following command against the newly created certificate:

ssh-keygen -L -f ~/.ssh/vault_rsa-cert.pub

The output displays several critical security parameters:

Valid Principals: The target operating system user names that you are permitted to log in with, for example, mcc-user. If you attempt to log in as root, the SSH daemon rejects the certificate.
Valid: The exact time boundaries for the certificate (TTL). Once expired (5 minutes in this demo), the certificate is no longer accepted.
Extensions: The specific SSH features authorized by Vault. This demo includes allowing a pseudo-terminal (permit-pty), but it can be restricted further, for example, denying port-forwarding.

Log in to the target MOSK node

Using the active signed certificate, initiate the SSH connection:

ssh -o IdentitiesOnly=yes -i ~/.ssh/vault_rsa \
    mcc-user@<TARGET_HOST_IP>

Use -o IdentitiesOnly=yes to prevent a running ssh-agent from offering extraneous keys that can trigger the Too many authentication failures error.

Verify access control: operator versus reader users

The group mapping configuration established between Keycloak and Vault ensures that permissions are granted based on team roles rather than individual accounts.

To verify access control:

Log in as the operator user and verify authorized access.

Keycloak validates the operator user credentials and transmits the user group memberships (including ssh-operators) inside the OIDC token. Vault evaluates the ssh-operators group alias, dynamically assigning the ssh-allow policy. As a result, the operator gains full access to the ssh-client-signer/ engine to issue SSH certificates.
Log in as the reader user and verify restricted access.

Keycloak authenticates the reader user by passing only the default group memberships (default-users). Because the default-users group is not mapped to any SSH signing policies in Vault, the user receives only the fallback default policy. The ssh-client-signer/ secret engine is hidden, and any attempt to request SSH certificates is rejected.

Step 7. Configure audit and logging

Comprehensive audit trails across all three architectural layers (Keycloak, Vault, and MOSK target nodes) are essential for compliance, security monitoring, and forensic analysis.

Keycloak authentication and MFA logs

Keycloak records all user authentication attempts, MFA challenges, and token issuance events. The following events are logged:

User logins
Failed password and MFA attempts
Token exchanges
Client authorization redirects

To enable audit logging, in the Keycloak admin console, navigate to Realm Settings > Events and enable Save Events and Admin Events.

You can route events to standard output or a dedicated syslog target and forward them to central SIEM systems, for example, OpenSearch or Logstash.

Audit logs for Vault certificate issuance

Vault provides detailed, non-repudiable audit logs for every secret access and SSH certificate signing request.

When enabling audit logging, ensure that the specified path is writable by the system user running the Vault process.

vault audit enable file file_path=/tmp/vault_audit.log

The following events are logged by Vault:

Authenticated user identity
Certificate TTL
Client IP address
Keycloak group mapping
Public key fingerprint
Requested principals

To verify detailed request payloads, inspect the vault_audit.log output:

{
  "time": "2026-09-07T11:50:52.37346354Z",
  "type": "request",
  "auth": {
    "display_name": "oidc-operator",
    "policies": ["default", "ssh-allow"]
  },
  "request": {
    "operation": "update",
    "path": "ssh-client-signer/sign/demo",
    "remote_address": "192.168.9.101"
  }
}

The example log above captures a request by the Keycloak identity oidc-operator (with the ssh-allow policy bound) connecting from 192.168.9.101 to issue an SSH certificate.

OpenSSH daemon audit logs on target nodes

OpenSSH daemons record certificate validation and user login events on target MOSK nodes locally. To record detailed certificate metadata, ensure that the LogLevel is set to VERBOSE or INFO in /etc/ssh/sshd_config.

On MOSK nodes, the OpenSSH service is accessed using ssh. To inspect the logs, run the following command:

sudo journalctl -u ssh -e | grep "Accepted publickey"

Example of system response extract:

Sep 07 11:43:02 mosk-worker-1 sshd[2330672]: Accepted publickey for
mcc-user from 192.168.9.15 port 43172 ssh2: RSA-CERT SHA256:sbF...
ID vault-oidc-operator-b1b... (serial 515525...) CA RSA SHA256:yS...

Field	Description
`mcc-user`	The target operating system principal the operator authenticated as.
`RSA-CERT`	Confirms that the login was authorized through an SSH certificate rather than a raw static key.
`ID vault-oidc-operator-...`	The Key ID injected by Vault. Provides an immediate audit trail back to the authentication method (`oidc`) and the specific Keycloak user such as `operator` who requested the certificate.
`serial ...`	The unique cryptographic serial number assigned to the ephemeral certificate by Vault.
`CA RSA SHA256:...`	The fingerprint of the Vault CA that signed the request, confirming the chain of trust.