I'm Gatan

Getting Started with Istio: Installing the Service Mesh & Running the Bookinfo Demo on Kubernetes

Mon, 14 Sep 2026 00:00:00 GMT

One of the most common next steps after setting up a Kubernetes cluster is adding a service mesh. Istio is the most widely adopted service mesh in the CNCF ecosystem — it gives you automatic mTLS encryption, traffic routing, observability, and resilience features without changing a single line of application code.

This guide walks through the full workflow:

Installing istioctl and the Istio control plane (default profile)
Enabling automatic Envoy sidecar injection
Deploying and exploring the Bookinfo sample application
Exposing the app on a bare-metal / single-node cluster (no cloud LoadBalancer required)
Visualizing your service mesh with the Kiali dashboard
Testing traffic routing with VirtualServices and DestinationRules

Official reference: Istio Installation Guide

Configuration Placeholders

Placeholder	Description	Example
`<NODE_IP>`	IP address of your Kubernetes node	`192.168.1.100`
`<NODE_PORT>`	NodePort assigned by Kubernetes	`32192`
`<ISTIO_VERSION>`	Istio release version downloaded	`1.31.0`

Environment Specifications

Component	Details
Kubernetes	Single-node cluster provisioned with kubeadm
CNI	Calico
Istio Version	1.31.x
Install Method	`istioctl` (CLI binary)
Profile	`default`

Note: Since this is a bare-metal single-node cluster without a cloud LoadBalancer (not GKE/EKS/AKS), services of type LoadBalancer created by the Istio IngressGateway will never receive an EXTERNAL-IP. This guide includes alternative access methods via port-forward and NodePort.

Prerequisites

Before starting, make sure the following are in place:

A single-node Kubernetes cluster in Ready state (see Kubernetes Single-Node Setup on AlmaLinux 9)
kubectl configured and able to access the cluster (kubectl get nodes shows Ready)
Sufficient resources on the node to run the Istio control plane + 4 Bookinfo microservices (at least 2 CPU / 4 GB RAM free is recommended)
Internet access from the node (to download the Istio binary and pull container images)

Step 1: Download and Install istioctl

1.1 Download the Istio Release

curl -L https://istio.io/downloadIstio | sh -

Output:

Downloading istio-<ISTIO_VERSION> from https://github.com/istio/istio/releases/download/<ISTIO_VERSION>/istio-<ISTIO_VERSION>-linux-amd64.tar.gz ...

Istio <ISTIO_VERSION> download complete!

The Istio release archive has been downloaded to the istio-<ISTIO_VERSION> directory.

To configure the istioctl client tool for your workstation,
add the /home/user/istio-<ISTIO_VERSION>/bin directory to your environment path variable with:
         export PATH="$PATH:/home/user/istio-<ISTIO_VERSION>/bin"

Begin the Istio pre-installation check by running:
         istioctl x precheck

Navigate into the extracted directory:

cd istio-*/

1.2 Add istioctl to PATH

export PATH=$PWD/bin:$PATH

# To make it permanent, append to your shell profile
echo 'export PATH=$HOME/istio-<ISTIO_VERSION>/bin:$PATH' >> ~/.bashrc

Tip: Replace <ISTIO_VERSION> with the actual version that was downloaded (e.g., 1.31.0).

1.3 Verify istioctl

istioctl version

Output:

Istio is not present in the cluster: no running Istio pods in namespace "istio-system"
1.31.0

The first line is expected — Istio hasn't been installed on the cluster yet. The important thing is that the istioctl client version is printed.

Step 2: Install Istio with the Default Profile

2.1 Pre-installation Check (optional but recommended)

istioctl x precheck

Output:

✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
  To get started, check out https://istio.io/latest/docs/setup/getting-started/.

Make sure there are no critical warnings or errors before proceeding.

2.2 Install the Default Profile

istioctl install --set profile=default -y

This command installs:

istiod (the control plane) in the istio-system namespace
istio-ingressgateway as the default ingress gateway

Output:

✔ Istio core installed ⛵️
✔ Istiod installed 🧠
✔ Ingress gateways installed 🛬
✔ Installation complete

Step 3: Verify the Installation

3.1 Check Pods in istio-system

kubectl get pods -n istio-system

Output:

NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-6dcdbcd758-z6mtb   1/1     Running   0          2m10s
istiod-84568cdf7c-tv9mt                 1/1     Running   0          2m24s

3.2 Check Control Plane Version

istioctl version

Output:

client version: 1.31.0
control plane version: 1.31.0
data plane version: 1.31.0 (1 proxies)

Now istioctl can read both the client and control plane versions.

3.3 Analyze Configuration (optional)

istioctl analyze -A

Use this to detect potential configuration issues across all namespaces before deploying any applications.

Step 4: Enable Automatic Sidecar Injection

Istio automatically injects an Envoy sidecar proxy into every pod deployed in namespaces labeled with istio-injection=enabled.

kubectl label namespace default istio-injection=enabled

Verify the label:

kubectl get namespace -L istio-injection

Output:

NAME              STATUS   AGE     ISTIO-INJECTION
default           Active   4h29m   enabled
istio-system      Active   3m26s
kube-system       Active   4h29m
...

Step 5: Deploy the Bookinfo Application

Bookinfo is Istio's official sample application, consisting of 4 microservices (productpage, details, reviews, ratings) with 3 different versions of the reviews service (v1, v2, v3) — perfect for demonstrating traffic routing capabilities.

5.1 Apply the Bookinfo Manifest

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

Make sure you are still inside the extracted Istio directory (istio-<ISTIO_VERSION>/) so that the samples/bookinfo/... path resolves correctly.

5.2 Verify Services and Pods

kubectl get services

Output:

NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
details       ClusterIP   10.102.162.188   <none>        9080/TCP   7s
kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP    4h30m
productpage   ClusterIP   10.111.175.168   <none>        9080/TCP   7s
ratings       ClusterIP   10.108.180.245   <none>        9080/TCP   7s
reviews       ClusterIP   10.99.206.200    <none>        9080/TCP   7s

kubectl get pods

Make sure each pod shows 2/2 in the READY column — the 2 indicates that the Envoy sidecar was successfully injected alongside the application container.

Output:

NAME                              READY   STATUS    RESTARTS   AGE
details-v1-65bd58d55f-svxqk       2/2     Running   0          7m32s
productpage-v1-7498b5bd44-4s76z   2/2     Running   0          7m32s
ratings-v1-9bff6cb79-7zpjv        2/2     Running   0          7m32s
reviews-v1-54465994f9-47h5d       2/2     Running   0          7m32s
reviews-v2-577579b4c5-6vvfl       2/2     Running   0          7m32s
reviews-v3-6d7944df59-tlr6r       2/2     Running   0          7m32s

If the READY column shows 1/1, the sidecar was not injected. Double-check the namespace label from Step 4, then restart the deployments with kubectl rollout restart deployment.

5.3 Test from Inside the Cluster

kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" \
  -c ratings -- curl -sS productpage:9080/productpage | grep -o "<title>.*</title>"

Output:

<title>Simple Bookstore App</title>

5.4 Create the Istio Gateway

kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Output:

gateway.networking.istio.io/bookinfo-gateway created
virtualservice.networking.istio.io/bookinfo created

Verify:

kubectl get gateway

Output:

NAME               AGE
bookinfo-gateway   24s

Step 6: Expose Bookinfo on a Single-Node Cluster

This is where the setup diverges from the official Istio documentation, which assumes your cluster has a cloud LoadBalancer. On a bare-metal single-node cluster, istio-ingressgateway will remain stuck in Pending:

kubectl get svc istio-ingressgateway -n istio-system

Output:

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                      AGE
istio-ingressgateway   LoadBalancer   10.100.98.219   <pending>     15021:32283/TCP,80:32192/TCP,443:32707/TCP   12m

Choose one of the options below to access Bookinfo from outside the cluster.

Option A — Port Forward (fastest for demos)

kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80

Keep this terminal running, then access the app from your browser or via curl:

http://<NODE_IP>:8080/productpage

If accessing from the server itself, use http://localhost:8080/productpage. If accessing from another machine (e.g., your laptop), make sure port 8080 is open in the firewall and use the server's IP address.

Option B — Patch to NodePort (more permanent)

kubectl patch svc istio-ingressgateway -n istio-system -p '{"spec": {"type": "NodePort"}}'

Check the NodePort assigned to port 80:

kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.spec.ports[?(@.port==80)].nodePort}'

Then access via:

http://<NODE_IP>:<NODE_PORT>/productpage

Once accessible, you should see the Bookinfo product page:

Step 7: Confirm the App is Working

Use curl to verify the application is accessible (example using port-forward from Option A):

curl -s "http://localhost:8080/productpage" | grep -o "<title>.*</title>"

Output:

<title>Simple Bookstore App</title>

Open the page in your browser and refresh several times — you'll notice the reviews section cycling through different versions in round-robin fashion: no stars, black stars, and red stars. This happens because no routing rules have been applied yet, so traffic is distributed equally across all three versions.

Step 8: Define Service Versions with DestinationRules

Before configuring traffic routing between reviews versions (e.g., request routing or traffic shifting), Istio needs to know which subsets/versions are available. This is done through DestinationRule resources.

kubectl apply -f samples/bookinfo/networking/destination-rule-all.yaml

Output:

destinationrule.networking.istio.io/productpage created
destinationrule.networking.istio.io/reviews created
destinationrule.networking.istio.io/ratings created
destinationrule.networking.istio.io/details created

Verify:

kubectl get destinationrules -o yaml

The default profile already enables auto mutual TLS, so you don't need to set up mTLS manually for this demo.

Step 9: Kiali Dashboard — Visualizing Your Service Mesh

Kiali is the official observability console for Istio. It provides a real-time visual representation of your service mesh topology, traffic flow, health status, and configuration validation. Istio ships Kiali as part of a bundle of observability addons alongside Prometheus (metrics), Grafana (dashboards), and Jaeger (distributed tracing).

9.1 Install the Observability Addons

Install all addons included in the Istio samples directory:

kubectl apply -f samples/addons

Output:

serviceaccount/grafana created
configmap/grafana created
service/grafana created
deployment.apps/grafana created
configmap/istio-grafana-dashboards created
configmap/istio-services-grafana-dashboards created
deployment.apps/jaeger created
service/tracing created
serviceaccount/kiali created
configmap/kiali created
clusterrole.rbac.authorization.k8s.io/kiali created
clusterrolebinding.rbac.authorization.k8s.io/kiali created
service/kiali created
deployment.apps/kiali created
serviceaccount/prometheus created
configmap/prometheus created
service/prometheus created
deployment.apps/prometheus created
...

9.2 Wait for Kiali to Be Ready

kubectl rollout status deployment/kiali -n istio-system

Output:

Waiting for deployment "kiali" rollout to finish: 0 of 1 updated replicas are available...
deployment "kiali" successfully rolled out

9.3 Access the Kiali Dashboard

To access the dashboard, patch the Kiali service to NodePort — the same approach used to expose the IngressGateway:

kubectl patch svc kiali -n istio-system -p '{"spec": {"type": "NodePort"}}'

Check the NodePort assigned to port 20001:

kubectl get svc kiali -n istio-system -o jsonpath='{.spec.ports[?(@.port==20001)].nodePort}'

Then open the dashboard in your browser at http://<NODE_IP>:<NODE_PORT>:

Navigate to the Graph section — Kiali displays a live visualization of your service mesh, showing the relationships between services in the Bookinfo application. It also provides filters to help you analyze traffic flow across the mesh.

Step 10: Test Traffic Routing with VirtualServices

Since the DestinationRules (with subsets v1/v2/v3) were applied in the previous step, you can now create a VirtualService to route 100% of traffic to a specific version. Let's route all reviews traffic to v2 (black stars):

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v2
EOF

Verify the routing rule:

kubectl get virtualservice reviews -o yaml

Refresh /productpage several times — now the reviews section will always show black stars (v2 fetches rating data but displays stars in black, unlike v3 which shows red stars). No more round-robin cycling between versions.

You can also confirm this in the Kiali graph — traffic to the reviews service now flows exclusively to the v2 subset, with no traffic going to v1 or v3:

Cleanup

Once you're done exploring, clean up all the resources:

# Remove Bookinfo and its gateway
samples/bookinfo/platform/kube/cleanup.sh

# (Optional) Revert services back to ClusterIP if you patched them to NodePort
kubectl patch svc istio-ingressgateway -n istio-system -p '{"spec": {"type": "ClusterIP"}}'

# Uninstall Istio from the cluster
istioctl uninstall --purge -y

# Delete the istio-system namespace
kubectl delete namespace istio-system

Troubleshooting

Symptom	Cause	Resolution
Sidecar not injected (pods show 1/1 instead of 2/2)	Namespace missing `istio-injection` label	Run `kubectl get namespace default --show-labels` and re-apply the label, then `kubectl rollout restart deployment -n default`
`istio-ingressgateway` stuck in `<pending>`	No cloud LoadBalancer on bare-metal	Use Option A (port-forward) or Option B (NodePort) from Step 6
`productpage` returns 503 / upstream connect error	Sidecar not fully ready or DestinationRules not applied	Check `kubectl get pods -n default` and inspect logs with `kubectl logs <pod> -c istio-proxy`
`istiod` in CrashLoopBackOff	Insufficient CPU/RAM on the node	Check `kubectl describe pod -n istio-system -l app=istiod` and `kubectl top nodes`

References

HashiCorp Vault + MariaDB: Eliminating Static Credentials with Dynamic Secrets

Wed, 19 Aug 2026 00:00:00 GMT

One of the most powerful — and underutilized — features of HashiCorp Vault is its Database Secrets Engine. Instead of embedding static database passwords in your .env files, application configs, or CI/CD pipelines, Vault generates short-lived, on-demand credentials that are automatically revoked when they expire.

This guide walks through the full setup:

Installing Vault in dev mode (local testing)
Running MariaDB via Docker Compose
Enabling and configuring the Database Secrets Engine
Creating db-admin and db-read-only roles
Generating and verifying dynamic credentials against MariaDB
Managing leases and manual revocation

Official reference: Vault MySQL/MariaDB Database Secrets Engine

Prerequisites

Vault CLI installed on your machine
Docker & Docker Compose (to run MariaDB)
A Unix-like terminal (Linux, macOS, or WSL on Windows)

Configuration Placeholders

The table below lists the placeholder variables used throughout this guide. Replace them with your environment's actual values:

Placeholder	Description	Example
`<MARIADB_HOST>`	IP or hostname of the MariaDB server	`192.168.1.100`
`<MARIADB_PORT>`	MariaDB port	`3306`
`<MARIADB_ROOT_PASSWORD>`	Root password for MariaDB	`YourStrongPassword`
`<VAULT_HOST>`	IP or hostname of the Vault server (non-dev mode)	`192.168.1.50`
`<VAULT_PORT>`	Vault API port	`8200`

Step 1: Install Vault (Dev Mode)

For local testing and exploration, you don't need a full production Vault setup (unsealing, persistent storage backends, etc.). The dev server starts pre-unsealed with an in-memory storage backend — perfect for learning.

macOS (Homebrew)

brew tap hashicorp/tap
brew install hashicorp/tap/vault

Linux (Debian/Ubuntu)

sudo apt update
sudo apt install -y gnupg wget lsb-release

wget -O- https://apt.releases.hashicorp.com/gpg \
  | gpg --dearmor \
  | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update
sudo apt install -y vault

Verify Installation

vault --version

Output:

Vault v2.0.4 (c9e9d1d4ddd4b55aae79a8949adffa9e96338720), built 2026-08-03T16:14:36Z

Start the Dev Server

Open a dedicated terminal window and run:

vault server -dev -dev-root-token-id="root"

Output:

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variables:

    $ export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: c+pRvn/2UC6tGpAH5ucQ9Qejufbxnpl/u0UuHTWnoKc=
Root Token: root

Development mode should NOT be used in production installations!

Vault will start listening on http://127.0.0.1:8200 with the root token root.

Warning: Dev mode stores all data in memory and resets on restart. Never use dev mode in production.

In a second terminal, export the required environment variables so the CLI knows where to connect:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'

Verify connectivity:

vault status

Output:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         2.0.4
Build Date      2026-08-03T16:14:36Z
Storage Type    inmem
Cluster Name    vault-cluster-b032d240
Cluster ID      3ae78de0-3009-c9b6-5993-caba26940ba5
HA Enabled      false

Expected output will show Sealed: false — confirming Vault is running and unsealed.

Alternative: Run Vault Without Dev Mode (via `config.hcl`)

Dev mode is great for quick exploration, but it stores everything in-memory (wiped on process exit) and starts fully unsealed automatically — which doesn't reflect how Vault behaves in a real environment. If you want a more realistic setup (persistent data, manual init/unseal), use a configuration file instead.

1. Create `config.hcl`

storage "file" {
  path = "/opt/vault/data"
}

listener "tcp" {
  address     = "<VAULT_HOST>:<VAULT_PORT>"
  tls_disable = true
}

api_addr = "http://<VAULT_HOST>:<VAULT_PORT>"
ui       = true

Configuration breakdown:

Field	Purpose
`storage "file"`	Persists all Vault data to disk. For HA environments, use `storage "raft"` instead
`listener "tcp" address`	Replace `<VAULT_HOST>` with the server's IP/hostname and `<VAULT_PORT>` with the desired port
`tls_disable = true`	Allows plain HTTP access without a certificate — testing/development only
`api_addr`	The address Vault advertises to clients and cluster peers; must match `listener`

Create the storage directory:

sudo mkdir -p /opt/vault/data
sudo chown -R $(whoami) /opt/vault/data

2. Start Vault with the Config File

vault server -config=config.hcl

Output:

==> Vault server configuration:

                     Cgo: disabled
              Go Version: go1.26.5
              Listener 1: tcp (addr: "<VAULT_HOST>:8200", cluster address: "<VAULT_HOST>:8201", tls: "disabled")
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: file
                 Version: Vault v2.0.4, built 2026-08-03T16:14:36Z

==> Vault server started! Log data will stream in below:

[INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
[INFO]  incrementing seal generation: generation=1
[INFO]  core: Initializing version history cache for core
[INFO]  events: Starting event system
[INFO]  core: security barrier not initialized
[INFO]  core: seal configuration missing, not initialized

In a new terminal, point the CLI at your Vault instance:

export VAULT_ADDR='http://<VAULT_HOST>:<VAULT_PORT>'

3. Initialize Vault

Unlike dev mode, a fresh Vault instance must be initialized before use. This process generates the unseal keys and the initial root token — store them securely, as they are only shown once:

vault operator init

Example output:

Unseal Key 1: xxxxx
Unseal Key 2: xxxxx
Unseal Key 3: xxxxx
Unseal Key 4: xxxxx
Unseal Key 5: xxxxx

Initial Root Token: hvs.xxxxxxxxxxxxxxxx

By default, Vault requires 3 out of 5 unseal keys (configurable via -key-shares and -key-threshold flags on vault operator init).

4. Unseal Vault

Run vault operator unseal once per key, up to the threshold (default: 3 times), providing a different key each time:

vault operator unseal   # Enter Unseal Key 1
vault operator unseal   # Enter Unseal Key 2
vault operator unseal   # Enter Unseal Key 3

Verify the status:

vault status

Confirm that Sealed shows false before proceeding.

5. Log In with the Root Token

vault login
# Enter the Initial Root Token from `vault operator init`

Once logged in, all subsequent steps — enabling the Database Secrets Engine, configuring MariaDB, creating roles, and generating credentials — are identical to the dev mode workflow below. Just make sure VAULT_ADDR points to http://<VAULT_HOST>:<VAULT_PORT> instead of 127.0.0.1:8200.

Security note: The setup above (tls_disable = true, single-node manual unseal) is appropriate only for testing and development on trusted/private networks. For production deployments, always enable TLS, consider auto-unseal (e.g., via cloud KMS), and restrict network access to the Vault port.

Step 2: Set Up MariaDB (Docker Compose)

Create a docker-compose.yml file to spin up a MariaDB instance:

services:
  database:
    container_name: mariadb
    image: mariadb:latest
    restart: always
    ports:
      - "<MARIADB_PORT>:3306"
    environment:
      - MARIADB_ROOT_PASSWORD=<MARIADB_ROOT_PASSWORD>
    volumes:
      - mariadb-data:/var/lib/mysql

volumes:
  mariadb-data:
    name: mariadb-data

Start the container:

docker compose up -d

Verify it's running:

docker ps

Output:

CONTAINER ID   IMAGE            COMMAND                  CREATED         STATUS         PORTS                    NAMES
de13ac6ba5cb   mariadb:latest   "docker-entrypoint.s…"   4 seconds ago   Up 3 seconds   0.0.0.0:3306->3306/tcp   mariadb

With this configuration, MariaDB is accessible at:

Parameter	Value
Host	`<MARIADB_HOST>` (or `127.0.0.1` if Vault runs on the same machine)
Port	`<MARIADB_PORT>`
User	`root`
Password	`<MARIADB_ROOT_PASSWORD>`

The root credentials are what Vault uses as its management account — Vault will authenticate as root to create and revoke dynamic user accounts on your behalf.

Step 3: Configure Vault Database Secrets Engine

Before proceeding, make sure that MariaDB at <MARIADB_HOST>:<MARIADB_PORT> is reachable from the machine where Vault is running.

3.1 Enable the Database Secrets Engine

vault secrets enable database

Output:

Success! Enabled the database secrets engine at: database/

3.2 Configure the Database Connection

Tell Vault how to connect to MariaDB. Vault uses the mysql-database-plugin, which is fully compatible with MariaDB:

vault write database/config/mariadb \
  plugin_name=mysql-database-plugin \
  connection_url="{{username}}:{{password}}@tcp(<MARIADB_HOST>:<MARIADB_PORT>)/" \
  allowed_roles="*" \
  username="root" \
  password="<MARIADB_ROOT_PASSWORD>"

Parameter breakdown:

Parameter	Purpose
`connection_url`	MariaDB address with `{{username}}` and `{{password}}` placeholders that Vault fills in automatically
`allowed_roles="*"`	Permits all roles defined under this connection to use it. For production, restrict to explicit role names (e.g., `"db-admin,db-read-only"`)
`username` / `password`	The management credentials Vault uses to create and drop dynamic users

3.3 Create an Admin Role

This role generates users with full ALL PRIVILEGES — suitable for administrative or migration workloads:

vault write database/roles/db-admin \
  db_name=mariadb \
  creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';" \
  creation_statements="GRANT ALL PRIVILEGES ON *.* TO '{{name}}'@'%';" \
  revocation_statements="DROP USER '{{name}}'@'%';" \
  default_ttl="1h" \
  max_ttl="1h"

3.4 Create a Read-Only Role

This role generates users with SELECT privileges only — ideal for reporting dashboards, data pipelines, or read replicas:

vault write database/roles/db-read-only \
  db_name=mariadb \
  creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';" \
  creation_statements="GRANT SELECT ON *.* TO '{{name}}'@'%';" \
  revocation_statements="DROP USER '{{name}}'@'%';" \
  default_ttl="1h" \
  max_ttl="1h"

Step 4: Generate Dynamic Credentials

With the roles in place, request credentials on demand. You no longer need to know or manage any static passwords.

Admin credentials:

vault read database/creds/db-admin

Read-only credentials:

vault read database/creds/db-read-only

Example output:

Key                Value
---                -----
lease_id           database/creds/db-read-only/2f6a614c-4aa2-7b19-24b9-ad944a8d4de6
lease_duration     1h
lease_renewable    true
password           yY-57n3X5UQhxnmFRP3f
username           v-root-db-read-onl-crBWVqVh2Hc1

Each invocation creates a brand-new MariaDB user with a randomly generated username and password. When the TTL expires, Vault automatically executes the revocation_statements to drop that user from MariaDB.

Step 5: Verify the Credentials

Use the freshly generated credentials to authenticate against MariaDB:

mariadb -h <MARIADB_HOST> \
  -u v-root-db-read-onl-crBWVqVh2Hc1 \
  -pyY-57n3X5UQhxnmFRP3f \
  -e "SHOW DATABASES;"

Expected behavior:

Credentials from db-admin can execute any SQL operation (DDL, DML, etc.)
Credentials from db-read-only: SELECT queries succeed; INSERT, DROP, etc., are rejected with an access denied error

If both behave as expected, the Vault and MariaDB integration is working correctly.

Step 6 (Optional): Lease Management & Manual Revocation

List Active Leases

vault list sys/leases/lookup/database/creds/db-read-only

Revoke a Specific Lease Early

vault lease revoke database/creds/db-read-only/<lease_id>

After revocation, Vault immediately executes the DROP USER statement on MariaDB — the credentials are invalidated instantly, even before the TTL would have expired.

Revoke All Leases for a Role

vault lease revoke -prefix database/creds/db-read-only/

This is useful in incident response scenarios where you want to instantly invalidate every active credential for a compromised role.

Why This Matters: Security Benefits

Traditional Approach	Vault Dynamic Secrets
Static password stored in `.env` or config	No static password exists anywhere
Password shared across multiple services	Each service/request gets a unique credential
Manual rotation (easy to forget)	Automatic revocation at TTL expiry
Audit trail unclear	Every credential tied to a Vault lease ID
Leaked password = permanent breach risk	Leaked credential expires in 1 hour or less

What's Next

Once you've verified the basic flow, consider exploring these advanced capabilities:

Root credential rotation — Let Vault manage and rotate the root password it uses:
```
vault write -force database/rotate-root/mariadb
```
After this command, only Vault knows the root password.
Static roles — For existing long-lived database users that need periodic password rotation rather than dynamic creation.
Vault Agent — A sidecar daemon that automatically fetches and renews credentials, injecting them into your application's environment or config files with zero code changes.
Vault API / SDK — Integrate dynamic credential retrieval directly into your application at startup time using the Vault HTTP API or language-specific SDKs (Go, Python, Java, etc.).

Conclusion

By integrating HashiCorp Vault's Database Secrets Engine with MariaDB, you fundamentally change how applications access your database:

No more static passwords — credentials are generated on-demand and never stored in plaintext
Automatic isolation — every consumer gets a unique credential, making audit logs precise and breach impact minimal
Zero-config revocation — credentials expire automatically; incident response is a single vault lease revoke command away

This pattern is the foundation of secrets management at scale and applies equally well to PostgreSQL, MySQL, Oracle, MongoDB, and other supported database engines.

Reference: Vault MySQL/MariaDB Database Secrets Engine — HashiCorp Docs

Zero Trust Kubernetes: Securing Cluster Access with Keycloak and Pinniped SSO

Fri, 24 Jul 2026 00:00:00 GMT

This guide details a battle-tested setup for implementing Zero Trust Kubernetes authentication in production — including common runtime pitfalls, configuration edge cases, and step-by-step troubleshooting solutions.

Background & Motivation

While many engineering teams run workloads on Kubernetes in production, cluster access is frequently managed using legacy authentication anti-patterns:

Shared, static kubeconfig files distributed via Slack, email, or internal repositories.
Static ServiceAccount tokens that are never rotated.
Over-privileged cluster admin roles with zero individual user audit logging.

In this tutorial, we will set up Keycloak as a centralized Identity Provider (IdP) and Pinniped as the authentication bridge between Keycloak and your Kubernetes cluster.

Key Outcomes

Self-service user login: Developers authenticate directly using their single sign-on (SSO) credentials via a browser window.
Short-lived sessions: User access automatically expires after 8 hours, eliminating static long-lived credentials.
Role-Based Access Control (RBAC): Permissions are dynamically mapped based on Keycloak user groups.
Stateless kubeconfig: Distribute a single, static kubeconfig template that contains zero credentials or secrets.

Prerequisites

Before starting, ensure you have the following prerequisites in place:

Kubernetes Cluster: A running Kubernetes cluster (e.g., set up via kubeadm or a managed provider).
Keycloak Instance: Keycloak running and accessible over HTTPS. If you don't have one set up yet, follow our guide on Deploying Keycloak 26 in Production on AlmaLinux 9.
Ingress Controller: Traefik installed as an Ingress Controller on your cluster. For installation details, see Installing ArgoCD & Traefik HTTPS Ingress — you only need to complete the Install Traefik via Helm section.
DNS Domain: A domain pointing to your cluster Ingress IP for exposing the Pinniped Supervisor (e.g., pinniped.example.com).

Configuration Placeholders

The table below lists the placeholder variables used throughout this guide. Replace them with your environment's specific values:

Placeholder	Description	Example
`<KEYCLOAK_URL>`	Keycloak domain FQDN	`sso.example.com`
`<REALM_NAME>`	Keycloak realm name	`internal`
`<PINNIPED_SUPERVISOR_URL>`	FQDN assigned to Pinniped Supervisor	`pinniped.example.com`
`<K8S_API_URL>`	Kubernetes API Server endpoint and port	`https://192.168.1.100:6443`
`<YOUR_EMAIL>`	ACME registration email for cert-manager	`admin@example.com`
`<USER_EMAIL>`	Test developer email address	`dev-user@example.com`

Understanding Pinniped

Pinniped is a CNCF Sandbox project designed specifically to make Kubernetes authentication secure, seamless, and infrastructure-agnostic.

Without Pinniped, every Kubernetes cluster must be individually configured with API server OIDC flags (--oidc-issuer-url, --oidc-client-id, etc.) — requiring control plane restarts and tricky multi-tenant maintenance. Pinniped solves this by acting as an intermediary authentication broker.

[ Keycloak ]  (Identity Provider)
      │
      │  OIDC Authorization Code Flow
      ▼
[ Pinniped Supervisor ]  (Central Broker / OIDC IdP)
      │
      │  Cluster-scoped Token Exchange
      ▼
[ Pinniped Concierge ]   (Per-cluster Agent)
      │
      │  Impersonate User & Groups
      ▼
[ kube-apiserver ]       (Cluster Control Plane)

Pinniped consists of two core components:

Supervisor: Installed once (either on a dedicated management cluster or your main cluster). It acts as an OIDC identity broker, authenticating users against Keycloak and issuing cluster-scoped identity tokens.
Concierge: Installed on every cluster you want to secure. It validates incoming tokens issued by the Supervisor and impersonates the identity when sending requests to kube-apiserver.

How Pinniped Kubeconfig Works

Traditional kubeconfig files contain embedded client certificates or static tokens. Pinniped changes this paradigm entirely:

Credential-less configuration: The generated kubeconfig file contains no user credentials, certificates, or tokens.
Safe distribution: The exact same kubeconfig file can be safely stored in public or team internal Git repositories.
Browser-based flow: When a user executes kubectl, the Pinniped CLI plugin opens a local browser window to authenticate with Keycloak. Upon successful login, short-lived credentials are cached locally.

Architecture Overview

┌──────────────────────────────────────────────┐
│           Keycloak (Identity Provider)       │
└──────────────────────┬───────────────────────┘
                       │ OIDC Auth Code Flow
                       ▼
┌──────────────────────────────────────────────┐
│             Pinniped Supervisor              │  ← TLS via cert-manager
│  - FederationDomain                          │
│  - OIDCIdentityProvider                      │
└──────────────────────┬───────────────────────┘
                       │ Cluster-Scoped Token Exchange
                       ▼
┌──────────────────────────────────────────────┐
│              Pinniped Concierge              │  ← Deployed per downstream cluster
│  - JWTAuthenticator                          │
└──────────────────────┬───────────────────────┘
                       │ User Impersonation Header
                       ▼
┌──────────────────────────────────────────────┐
│                kube-apiserver                │
└──────────────────────┬───────────────────────┘
                       │ Native RBAC Evaluation
                       │ (Group Claim: keycloak:<group>)
         ┌─────────────┼─────────────┐
         │             │             │
         ▼             ▼             ▼
  ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
  │  ns: dev    │ │  ns: prod   │ │ ns: staging │
  │ dev-team    │ │ ops-team    │ │ qa-team     │
  │ (Role: edit)│ │(Role: admin)│ │(Role: view) │
  └─────────────┘ └─────────────┘ └─────────────┘

CNCF Projects Involved

Project	CNCF Status	Role
Keycloak	Incubating	Identity Provider (SSO & User Directory)
Pinniped	Sandbox	Authentication Broker & Credential Distribution
cert-manager	Graduated	Automated TLS Certificate Management (Let's Encrypt)
Kubernetes	Graduated	Container Orchestration Engine

Part 1: Keycloak Configuration

1.1 Create the OIDC Client

Log in to your Keycloak Admin Console and select your realm (<REALM_NAME>).
Navigate to Clients → click Create client.
Configure the client parameters:

Field	Value
Client type	OpenID Connect
Client ID	`kubernetes`
Client authentication	Off (Public client)
Standard flow	On
Valid redirect URIs	`http://localhost:8000`, `http://localhost:18000`

Note: In Part 3, once the Pinniped Supervisor endpoint is configured, you must return here and add https://<PINNIPED_SUPERVISOR_URL>/* to Valid redirect URIs. Failing to do so will result in an Invalid parameter: redirect_uri error during browser authentication.

1.2 Add the Group Mapper

To enforce Kubernetes RBAC based on Keycloak user groups, the JWT token issued by Keycloak must contain a groups claim.

Under the kubernetes client, open the Client scopes tab.
Select the dedicated scope (e.g., kubernetes-dedicated).
Click Add mapper → By configuration → Group Membership.
Set the following fields:

Field	Value
Name	`groups`
Token Claim Name	`groups`
Full group path	Off
Add to ID token	On
Add to access token	On
Add to userinfo	On

Important: groups here is a Token Claim Name, not an OIDC scope name. If you attempt a manual OAuth request with -d "scope=openid groups", Keycloak will reject it with invalid_scope. Always request scope=openid — the groups claim will automatically be included in the token payload once this mapper is enabled.

You can verify the supported scopes on your realm endpoint:

curl -s https://<KEYCLOAK_URL>/realms/<REALM_NAME>/.well-known/openid-configuration | jq '.scopes_supported'

1.3 Create Groups and Assign Users

In Keycloak sidebar, go to Groups and create:
- dev-team
- ops-team
- qa-team
Go to Users, select a user, open the Groups tab, and click Join Group to assign them to their respective team.

1.4 Test User Setup Checklist

When creating test users in Keycloak, ensure all required user details are configured. Incomplete profiles will cause authentication failures:

Requirement	Consequence if omitted
First Name & Last Name set	Error: `Account is not fully set up`
Email verified = On (Details tab)	Error: `email_verified claim in upstream ID token has false value`
Password Temporary = Off (Credentials tab)	Error: `Account is not fully set up`
Required user actions cleared	Error: `Account is not fully set up`

Tip: If Keycloak returns Account is not fully set up despite meeting all criteria above, reset the user password manually under Credentials → Set password.

1.5 Optional: Test Token Generation via Direct Access Grants

Direct Access Grants (password grant type) are disabled by default in modern Keycloak versions for security reasons. To temporarily test token issuance via curl:

Open Client kubernetes → Settings → Capability config → enable Direct access grants → Save.
Execute the token request:

curl -s -X POST \
  https://<KEYCLOAK_URL>/realms/<REALM_NAME>/protocol/openid-connect/token \
  -d "client_id=kubernetes" \
  -d "username=<USER_EMAIL>" \
  -d "password=<PASSWORD>" \
  -d "grant_type=password" \
  -d "scope=openid" \
  | jq -r '.id_token' \
  | cut -d. -f2 \
  | base64 -d 2>/dev/null \
  | jq .

Verify that the groups array is present in the decoded token payload:

{
  "email": "dev-user@example.com",
  "email_verified": true,
  "groups": ["dev-team"],
  "iss": "https://sso.example.com/realms/internal"
}

Security Note: Disable Direct access grants after testing. Production authorization always uses the standard Authorization Code Flow via browser.

1.6 Configure 8-Hour Session Lifespan

In Keycloak Admin Console, open Realm Settings → Sessions tab:

Field	Lifespan
SSO Session Idle	`8 hours`
SSO Session Max	`8 hours`
Client Session Idle	`8 hours`
Client Session Max	`8 hours`

Next, select the Tokens tab:

Field	Lifespan
Access Token Lifespan	`5 minutes`
Refresh Token Max Reuse	`0`

When Keycloak's refresh token expires after 8 hours, Pinniped automatically prompts the user to re-authenticate via their browser on the next kubectl command execution.

Part 2: Installing cert-manager for Pinniped TLS

Pinniped Supervisor requires a trusted TLS certificate for its endpoint. By default, Pinniped generates a self-signed bootstrap certificate that external CLI clients will reject with: tls: failed to verify certificate: x509: certificate is valid for pinniped-supervisor-bootstrap-cert.

We install cert-manager to automatically obtain and renew Let's Encrypt TLS certificates as Kubernetes secrets.

2.1 Install cert-manager

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

# Monitor deployment status until all pods are Ready
kubectl get pods -n cert-manager --watch

2.2 Define ClusterIssuer and Certificate

Create cert-manager-pinniped.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL>
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            ingressClassName: traefik

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: pinniped-tls
  namespace: pinniped-supervisor
spec:
  secretName: pinniped-tls-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - <PINNIPED_SUPERVISOR_URL>

Apply the configuration:

kubectl apply -f cert-manager-pinniped.yaml

# Verify Certificate status (Wait for READY: True)
kubectl get certificate pinniped-tls -n pinniped-supervisor --watch

Part 3: Deploying Pinniped Supervisor

3.1 Install Pinniped Supervisor

kubectl apply -f https://get.pinniped.dev/latest/install-pinniped-supervisor.yaml

# Verify pods
kubectl get pods -n pinniped-supervisor

3.2 Expose Supervisor via Traefik (TLS Passthrough)

Because Pinniped Supervisor directly handles sensitive credential exchanges, it requires TLS Passthrough to perform end-to-end TLS termination directly inside the pod rather than at the Ingress controller layer.

Create a ClusterIP Service for the Supervisor:

# pinniped-supervisor-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: pinniped-supervisor
  namespace: pinniped-supervisor
spec:
  type: ClusterIP
  selector:
    app: pinniped-supervisor
  ports:
    - protocol: TCP
      port: 443
      targetPort: 8443

Create a Traefik IngressRouteTCP with passthrough: true:

# pinniped-supervisor-ingressroute.yaml
apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: pinniped-supervisor
  namespace: pinniped-supervisor
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI(`<PINNIPED_SUPERVISOR_URL>`)
      services:
        - name: pinniped-supervisor
          port: 443
  tls:
    passthrough: true

Apply both manifests:

kubectl apply -f pinniped-supervisor-service.yaml
kubectl apply -f pinniped-supervisor-ingressroute.yaml

3.3 Configure OIDCIdentityProvider

Create the custom OIDCIdentityProvider resource to link Pinniped Supervisor to Keycloak:

apiVersion: idp.supervisor.pinniped.dev/v1alpha1
kind: OIDCIdentityProvider
metadata:
  name: keycloak-idp
  namespace: pinniped-supervisor
spec:
  issuer: https://<KEYCLOAK_URL>/realms/<REALM_NAME>
  authorizationConfig:
    additionalScopes:
      - email
    allowPasswordGrant: false
  claims:
    username: email
    groups: groups
  client:
    secretName: keycloak-client-secret

Note: Do not add groups to additionalScopes. Requesting groups as an OIDC scope will trigger an invalid_scope error from Keycloak.

Now, create the corresponding client secret. Pinniped requires a custom secret type secrets.pinniped.dev/oidc-client:

apiVersion: v1
kind: Secret
metadata:
  name: keycloak-client-secret
  namespace: pinniped-supervisor
type: secrets.pinniped.dev/oidc-client
stringData:
  clientID: kubernetes
  clientSecret: ""

Caution: Do not use kubectl create secret generic. Generic secrets have type Opaque, which will cause Pinniped to throw the error referenced Secret has wrong type "Opaque". Always apply via YAML with type: secrets.pinniped.dev/oidc-client.

Verify IdP health:

kubectl describe OIDCIdentityProvider keycloak-idp -n pinniped-supervisor
# Look for: Phase: Ready

If status remains in Phase: Error, restart the Supervisor deployment to reload secrets:

kubectl rollout restart deployment/pinniped-supervisor -n pinniped-supervisor

3.4 Configure FederationDomain

The FederationDomain exposes the public OIDC issuer endpoint of the Supervisor, using the TLS certificate generated by cert-manager:

apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: FederationDomain
metadata:
  name: kubernetes-federation
  namespace: pinniped-supervisor
spec:
  issuer: https://<PINNIPED_SUPERVISOR_URL>/kubernetes-federation
  tls:
    secretName: pinniped-tls-cert
  identityProviders:
    - displayName: "Keycloak SSO"
      objectRef:
        apiGroup: idp.supervisor.pinniped.dev
        kind: OIDCIdentityProvider
        name: keycloak-idp
      transforms:
        expressions:
          - type: username/v1
            expression: '"keycloak:" + username'
          - type: groups/v1
            expression: 'groups.map(g, "keycloak:" + g)'

Verify FederationDomain status:

kubectl describe FederationDomain kubernetes-federation -n pinniped-supervisor | grep "Phase\|Message"
# Expected output: Phase: Ready

Remember to update Keycloak client settings (Part 1.1) by adding https://<PINNIPED_SUPERVISOR_URL>/* to the client's Valid redirect URIs.

Part 4: Deploying Pinniped Concierge

4.1 Install Concierge

Deploy the Pinniped Concierge agent on the target Kubernetes cluster:

kubectl apply -f https://get.pinniped.dev/latest/install-pinniped-concierge.yaml

4.2 Configure JWTAuthenticator

The Concierge uses a JWTAuthenticator to validate authentication tokens issued by the Supervisor.

Extract the active TLS Certificate Authority bundle from pinniped-tls-cert:

LE_CA=$(kubectl get secret pinniped-tls-cert \
  -n pinniped-supervisor \
  -o jsonpath='{.data.tls\.crt}')

Apply the JWTAuthenticator resource:

kubectl apply -f - <<EOF
apiVersion: authentication.concierge.pinniped.dev/v1alpha1
kind: JWTAuthenticator
metadata:
  name: supervisor-jwt-authenticator
  namespace: pinniped-concierge
spec:
  issuer: https://<PINNIPED_SUPERVISOR_URL>/kubernetes-federation
  audience: https://<K8S_API_URL>
  tls:
    certificateAuthorityData: ${LE_CA}
EOF

Verify Authenticator status:

kubectl describe JWTAuthenticator supervisor-jwt-authenticator -n pinniped-concierge | grep "Phase\|Message"
# Expected output: Phase: Ready

Part 5: Kubeconfig Generation & Developer Workflow

5.1 Install Pinniped CLI

Install the pinniped CLI on your workstation:

# macOS via Homebrew
brew install pinniped-cli

# Linux Binary
curl -Lo pinniped https://get.pinniped.dev/latest/pinniped-cli-linux-amd64
chmod +x pinniped
sudo mv pinniped /usr/local/bin/pinniped

5.2 Generate Kubeconfig File

As an administrator, generate the static kubeconfig template:

pinniped get kubeconfig \
  --kubeconfig ~/.kube/config \
  > kubeconfig-production.yaml

Note: In Pinniped CLI v0.47.0+, the --pinniped-namespace flag was deprecated in favor of automatic discovery.

This kubeconfig-production.yaml file contains no user secrets. Upload it to your team's internal documentation site or repository.

5.3 Workstation Setup for Developers

Developers set up cluster access on their machines in two steps:

# 1. Download generic kubeconfig from internal docs
curl -O https://internal-docs.example.com/kubernetes/kubeconfig-production.yaml

# 2. Install Pinniped CLI
brew install pinniped-cli

Path Troubleshooting: The generated kubeconfig invokes /usr/local/bin/pinniped. On Apple Silicon Macs, Homebrew installs binaries under /opt/homebrew/bin/pinniped. Create a symbolic link to prevent exec: fork/exec /usr/local/bin/pinniped: no such file or directory:
sudo ln -s $(which pinniped) /usr/local/bin/pinniped

5.4 First-Time Login Experience

When a developer runs kubectl using the Pinniped kubeconfig:

export KUBECONFIG=./kubeconfig-production.yaml
kubectl get pod -n dev

Pinniped automatically launches the system web browser pointing to Keycloak SSO.
The user enters their Keycloak credentials.
The browser displays "Login Succeeded".
Control returns to the terminal, and kubectl executes seamlessly.

5.5 Daily Session Flow

Morning (08:00):
  kubectl get pods ──▶ Browser opens ──▶ User authenticates
  └── Session active for 8 hours ✅

Afternoon (12:00):
  kubectl get pods ──▶ Seamless background token refresh ✅

End of Shift / Next Day (After 8 Hours):
  kubectl get pods ──▶ Refresh token expired
  └── Browser opens automatically for standard re-login ✅

To manually clear session credentials (e.g., when testing different users or groups):

# Linux
rm -rf ~/.config/pinniped/sessions.yaml

# macOS
rm -rf ~/Library/Application\ Support/pinniped/sessions.yaml

Part 6: Multi-Tenant RBAC & Workload Setup

Apply namespaces, RBAC bindings, and test pods across development, production, and staging environments:

# 06-rbac-and-pods.yaml
---
# NAMESPACE: dev — Granted to keycloak:dev-team (Role: edit)
apiVersion: v1
kind: Namespace
metadata:
  name: dev
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-team-edit
  namespace: dev
subjects:
  - kind: Group
    name: "keycloak:dev-team"
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Pod
metadata:
  name: demo-frontend
  namespace: dev
spec:
  containers:
  - name: frontend
    image: nginx:alpine
---
apiVersion: v1
kind: Pod
metadata:
  name: demo-backend
  namespace: dev
spec:
  containers:
  - name: backend
    image: hashicorp/http-echo:alpine
    args: ["-text=Backend Service running in DEV"]
---
apiVersion: v1
kind: Pod
metadata:
  name: demo-database
  namespace: dev
spec:
  containers:
  - name: database
    image: postgres:15-alpine
    env:
    - name: POSTGRES_PASSWORD
      value: "devpassword"

---
# NAMESPACE: prod — Granted to keycloak:ops-team (Role: admin)
apiVersion: v1
kind: Namespace
metadata:
  name: prod
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ops-team-admin
  namespace: prod
subjects:
  - kind: Group
    name: "keycloak:ops-team"
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Pod
metadata:
  name: demo-payment
  namespace: prod
spec:
  containers:
  - name: payment
    image: hashicorp/http-echo:alpine
    args: ["-text=Payment Service in PROD"]

Apply the resources:

kubectl apply -f 06-rbac-and-pods.yaml

Part 7: End-to-End Verification & CLI Testing

7.1 Verify Active User Identity

Run kubectl auth whoami to verify how the API server perceives your authenticated identity and groups:

$ kubectl auth whoami
ATTRIBUTE                                           VALUE
Username                                            keycloak:dev-user@example.com
Groups                                              [keycloak:dev-team system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=e5cfbbef0c518879defcaa995b2a4e578522c68d691d1ae9f910e759b5b7c5b7]

7.2 Access Control Verification

Test resource listing under allowed versus restricted namespaces:

Allowed Access (`dev` namespace)

$ kubectl get pod -n dev
NAME            READY   STATUS    RESTARTS   AGE
demo-backend    1/1     Running   0          9h
demo-database   1/1     Running   0          9h
demo-frontend   1/1     Running   0          9h

Forbidden Access (`prod` namespace for `dev-team` member)

$ kubectl get pod -n prod
Error from server (Forbidden): pods is forbidden: User "keycloak:dev-user@example.com" cannot list resource "pods" in API group "" in the namespace "prod"

Pre-flight Permission Check

$ kubectl auth can-i create deployments -n prod
no

Troubleshooting Guide

Symptom / Error	Root Cause	Resolution
`unauthorized_client: Client not allowed for direct access grants`	Direct access grants disabled in Keycloak settings	Enable temporarily under Client Settings tab if testing via `curl`.
`invalid_scope: Invalid scopes: openid groups`	`groups` passed as scope parameter	Omit `groups` from scope argument; standard `scope=openid` includes claim.
`Account is not fully set up`	Missing user profile data	Ensure First Name, Last Name, Email Verified=True, and clear temporary passwords.
`Secret has wrong type "Opaque"`	Created via `kubectl create secret generic`	Apply via YAML setting `type: secrets.pinniped.dev/oidc-client`.
`tls: certificate is valid for pinniped-supervisor-bootstrap-cert`	Using self-signed Supervisor certificate	Deploy cert-manager Certificate and reference TLS secret in `FederationDomain`.
`Invalid parameter: redirect_uri`	Callback URL missing in Keycloak	Add `https://<PINNIPED_SUPERVISOR_URL>/*` to Valid Redirect URIs in Keycloak.
`email_verified claim in upstream ID token has false value`	Unverified user email in Keycloak	Toggle Email Verified to `On` under Keycloak User details.
`fork/exec /usr/local/bin/pinniped: no such file or directory`	CLI installed outside standard path	Symlink binary: `sudo ln -s $(which pinniped) /usr/local/bin/pinniped`.
`could not complete Concierge credential exchange: authentication failed`	Outdated TLS CA in Concierge	Re-extract `tls.crt` from `pinniped-tls-cert` and update `JWTAuthenticator`.
`Error: unknown flag: --pinniped-namespace`	Deprecated CLI flag	Remove flag; Pinniped CLI auto-detects Concierge namespace in modern versions.

Production Lessons Learned

Group Mapper is Critical: Without adding the groups mapper under Client Scopes, authentication completes but RBAC bindings continuously fail.
Treat groups as Claims, Not Scopes: Passing groups as an OIDC scope parameter breaks Keycloak protocol validation.
Pinniped Supervisor Requires Valid Public TLS: Never expose Supervisor using self-signed bootstrap certs in production environments.
Stateless Kubeconfig Security: Since Pinniped kubeconfig files contain no secrets, credential leaks via repository commits are completely mitigated. Access revocation is performed instantly in Keycloak.

Architecture Summary

Aspect	Legacy Kubeconfig	Pinniped + Keycloak Zero Trust
Credentials	Static embedded tokens / certs	Short-lived OIDC tokens (8h max)
User Identification	Generic / Shared credentials	Per-user identity and group claims
Audit Trail	Anonymous / Shared admin logs	Distinct per-user audit trails in API server
Access Revocation	Manual cluster cert rotation	Single-click disable in Keycloak
Developer Onboarding	Admin generates custom `kubeconfig`	Self-service download of static template

Official Documentation & References

Integrating Keycloak with GitLab — Single Sign-On via OpenID Connect

Mon, 20 Jul 2026 00:00:00 GMT

This guide walks you through integrating GitLab Self-Managed (Omnibus) with Keycloak as a Single Sign-On (SSO) Identity Provider using the OpenID Connect (OIDC) protocol — so your users can log in to GitLab through Keycloak.

This document focuses purely on the integration itself. For the Keycloak installation, refer to the separate guide: Deploying Keycloak 26 in Production on AlmaLinux 9.

Placeholders used throughout this guide: Wherever you see values like sso.example.com, gitlab.example.com, or YOUR_CLIENT_SECRET, replace them with your own actual values.

Placeholder	Description	Example
`sso.example.com`	Your Keycloak FQDN	`sso.company.net`
`gitlab.example.com`	Your GitLab FQDN	`gitlab.company.net`
`YOUR_CLIENT_SECRET`	The OIDC client secret from Keycloak	Copy from the Credentials tab
`YOUR_REALM`	Your Keycloak realm name	`internal`

Prerequisites
Create a Realm in Keycloak
Create a Client for GitLab
Retrieve the Client Secret
Verify the Realm Signature Algorithm
Configure GitLab (gitlab.rb)
Auto-Link Existing Accounts by Email
Reconfigure & Test Login
Troubleshooting

1. Prerequisites

Keycloak is up and running, accessible over HTTPS (e.g., https://sso.example.com). GitLab requires OIDC Identity Providers to be accessed over HTTPS — plain HTTP is not supported.
GitLab Omnibus (Self-Managed) is installed and accessible over HTTPS (e.g., https://gitlab.example.com).
Admin access to the Keycloak Admin Console.
Root/sudo access to the GitLab server for editing /etc/gitlab/gitlab.rb.

2. Create a Realm in Keycloak

Production application clients should not be placed in the master realm — that realm is recommended only for Keycloak's own superadmin accounts.

Log in to https://sso.example.com/admin/.
Click the realm dropdown in the top-left corner → Create realm.
Enter a Realm name, for example YOUR_REALM.
Click Create.

All subsequent steps are performed inside this realm.

[!NOTE] If you already have a realm set up from a previous integration (e.g., ArgoCD), you can reuse it. There is no need to create a new realm for each application — just create a new client within the existing realm.

3. Create a Client for GitLab

Inside your realm, navigate to Clients → Create client.
General Settings
- Client type: OpenID Connect
- Client ID: gitlab
- Click Next.
Capability config
- Client authentication: On (this makes the client confidential, generating a client secret)
- Authentication flow: check Standard flow only. Leave Direct access grants, Implicit flow, and other options unchecked — Standard flow (Authorization Code) is sufficient and the most secure choice for this use case.
- Click Next, then Save.
On the Settings page of the newly created client:
- Valid redirect URIs: https://gitlab.example.com/users/auth/openid_connect/callback
- Valid post logout redirect URIs: https://gitlab.example.com/*
- Web origins: https://gitlab.example.com
Click Save.

4. Retrieve the Client Secret

Still on the gitlab client, open the Credentials tab.
Copy the Client secret value — this will be used in the gitlab.rb configuration.

5. Verify the Realm Signature Algorithm

GitLab recommends tokens to be signed with a public-key algorithm (RS256) rather than a symmetric key (HS256), as it is more secure and easier to configure.

Navigate to Realm settings → Tokens tab.
Check the Default signature algorithm — ensure it is set to RS256 (this is Keycloak's default and typically does not need to be changed).

6. Configure GitLab (gitlab.rb)

Edit the GitLab configuration file:

sudo nano /etc/gitlab/gitlab.rb

Add the following block (replace identifier and secret with the Client ID and Client Secret from steps 3–4):

gitlab_rails['omniauth_allow_single_sign_on'] = ['openid_connect']
gitlab_rails['omniauth_block_auto_created_users'] = false

gitlab_rails['omniauth_providers'] = [
  {
    name: "openid_connect", # do not change — this is the internal provider name
    label: "Keycloak",      # the label displayed on the login button
    args: {
      name: "openid_connect",
      scope: ["openid", "profile", "email"],
      response_type: "code",
      issuer: "https://sso.example.com/realms/YOUR_REALM",
      client_auth_method: "query",
      discovery: true,
      uid_field: "preferred_username",
      pkce: true,
      client_options: {
        identifier: "gitlab",
        secret: "YOUR_CLIENT_SECRET",
        redirect_uri: "https://gitlab.example.com/users/auth/openid_connect/callback"
      }
    }
  }
]

Key options explained:

omniauth_allow_single_sign_on — enables automatic creation of new GitLab accounts (Just-In-Time provisioning) when a user logs in through Keycloak for the first time.
omniauth_block_auto_created_users — when false, automatically created accounts are immediately active without requiring admin approval. Set to true if you want a GitLab admin to manually approve every new SSO account.
discovery: true — GitLab automatically fetches all OIDC endpoints (authorization, token, userinfo, jwks) from https://sso.example.com/realms/YOUR_REALM/.well-known/openid-configuration, so there is no need to define them individually.
pkce: true — enables PKCE (Proof Key for Code Exchange) as an additional security layer on the Authorization Code flow.
uid_field: "preferred_username" — the field from the Keycloak token that GitLab uses as the unique user identifier.

7. Auto-Link Existing Accounts by Email

If there are existing GitLab users whose email addresses match those that will be used for Keycloak login, add the following line so that GitLab links to the existing account instead of creating a new one:

gitlab_rails['omniauth_auto_link_user'] = ["openid_connect"]

Auto-linking matches based on email addresses that are verified on both sides. Without this line, a user with a matching email will still receive a separate, brand-new GitLab account.

8. Reconfigure & Test Login

Apply the configuration changes:

sudo gitlab-ctl reconfigure

Once complete, open the GitLab login page:

https://gitlab.example.com/users/sign_in

A login button labeled Keycloak will appear below the standard login form. Click it to be redirected to the Keycloak login page, and after successful authentication, you will be redirected back to GitLab in a logged-in state.

[!IMPORTANT] After SSO is enabled, every sign-in attempt is redirected to the OmniAuth provider, which means the standard local GitLab login will no longer be directly available from the main page. Make sure at least one OmniAuth user has administrator privileges before enabling this in production. Local login can still be accessed manually via https://gitlab.example.com/users/sign_in?auto_sign_in=false as an emergency fallback.

9. Troubleshooting

Symptom	Likely Cause
Keycloak login button does not appear	`gitlab-ctl reconfigure` has not been run, or there is a syntax error in `gitlab.rb` — run `sudo gitlab-ctl reconfigure` and check the output for errors
`invalid_redirect_uri` error from Keycloak	The redirect URI in the Keycloak client does not exactly match the one sent by GitLab (`https://gitlab.example.com/users/auth/openid_connect/callback`) — pay attention to trailing slashes and the http/https scheme
Discovery failure / GitLab cannot fetch OIDC configuration	Ensure the GitLab server can resolve DNS and make HTTPS requests to `sso.example.com`; test with `curl https://sso.example.com/realms/YOUR_REALM/.well-known/openid-configuration` from the GitLab server
Login succeeds at Keycloak but fails returning to GitLab / signature invalid	Check that Default signature algorithm in Realm Settings > Tokens is set to RS256, not HS256
Same user logging in via SSO creates a new account instead of linking to the existing one	Add `gitlab_rails['omniauth_auto_link_user'] = ["openid_connect"]` as described in step 7, and ensure the email is identical and verified on both sides
GitLab refuses connection to Keycloak citing HTTPS	GitLab does not support OIDC Identity Providers over plain HTTP; ensure Keycloak is accessible over HTTPS (see the Keycloak deployment guide)

Summary of the flow: create a dedicated realm in Keycloak (or reuse an existing one) → register a gitlab client with Standard Flow only → copy the client secret → add the configuration to gitlab.rb with the matching issuer, identifier, and secret → run gitlab-ctl reconfigure → test login via the Keycloak button on the GitLab sign-in page.

Integrating Keycloak with ArgoCD — Single Sign-On via OpenID Connect

Sun, 19 Jul 2026 00:00:00 GMT

This guide walks you through integrating ArgoCD with Keycloak as a Single Sign-On (SSO) Identity Provider using ArgoCD's native OpenID Connect (OIDC) support — no Dex required. Along the way, we'll map Keycloak groups to ArgoCD RBAC roles for fine-grained access control.

This document focuses purely on the integration itself. For the Keycloak installation, refer to the separate guide: Deploying Keycloak 26 in Production on AlmaLinux 9.

Placeholders used throughout this guide: Wherever you see values like sso.example.com, argocd.example.com, or YOUR_CLIENT_SECRET, replace them with your own actual values.

Placeholder	Description	Example
`sso.example.com`	Your Keycloak FQDN	`sso.company.net`
`argocd.example.com`	Your ArgoCD FQDN	`argocd.company.net`
`YOUR_CLIENT_SECRET`	The OIDC client secret from Keycloak	Copy from the Credentials tab
`YOUR_REALM`	Your Keycloak realm name	`internal`

Prerequisites
Why Native OIDC Instead of Dex
Create a Client for ArgoCD in Keycloak
Add a Groups Mapper
Create Groups in Keycloak
Store the Client Secret in a Kubernetes Secret
Configure argocd-cm
Configure argocd-rbac-cm
Restart & Test Login
Break-Glass Account (Local Admin)
Offboarding
Troubleshooting

1. Prerequisites

Keycloak is up and running, accessible over HTTPS (e.g., https://sso.example.com), with a realm already created (see the GitLab integration guide for realm creation steps).
ArgoCD is installed on your Kubernetes cluster and accessible over HTTPS (e.g., https://argocd.example.com).
Admin access to the Keycloak Admin Console.
kubectl access with edit permissions on the namespace where ArgoCD is running (e.g., argocd).

2. Why Native OIDC Instead of Dex

ArgoCD supports two approaches for enabling SSO:

Dex — a built-in authentication proxy that can aggregate multiple identity providers (LDAP, GitHub, SAML, OIDC, etc.) into a single configuration.
Native OIDC (oidc.config directly in argocd-cm) — ArgoCD communicates directly with the Identity Provider without an intermediary.

Since Keycloak already supports OIDC natively and is the only Identity Provider in use, native OIDC is the simpler choice — there's no need to run an additional Dex component in the cluster, which means fewer moving parts and fewer points of failure. Dex only becomes relevant if you need to aggregate multiple Identity Providers simultaneously.

3. Create a Client for ArgoCD in Keycloak

Log in to https://sso.example.com/admin/ and switch to your realm (e.g., YOUR_REALM).
Navigate to Clients → Create client.
General Settings
- Client type: OpenID Connect
- Client ID: argocd
- Click Next.
Capability config
- Client authentication: On
- Authentication flow: check Standard flow only.
- Click Next, then Save.
On the Settings page:
- Valid redirect URIs: https://argocd.example.com/auth/callback
- Valid post logout redirect URIs: https://argocd.example.com/*
- Web origins: https://argocd.example.com
Click Save.
Open the Credentials tab and copy the Client secret — you'll need it in step 6.

4. Add a Groups Mapper

By default, the groups claim is not automatically included in Keycloak's OIDC tokens. ArgoCD needs this claim to match users to RBAC roles, so it must be added manually.

In your realm, navigate to Client scopes → Create client scope.
- Name: groups
- Type: Default (not Optional — this ensures the groups claim is always included in tokens without the client needing to explicitly request it)
- Click Save.

Enter the newly created groups client scope → Mappers tab → Add mapper → By configuration → Group Membership.
- Name: groups
- Token Claim Name: groups
- Full group path: Off (so the output is a plain group name like argocd-admins rather than /argocd-admins with the full path)
- Add to ID token: On
- Add to access token: On
- Click Save.
Go back to the argocd client → Client scopes tab → verify that the groups scope is already assigned (since it was created as a Default scope, it should automatically appear for all clients).

5. Create Groups in Keycloak

Create the groups that will be mapped to ArgoCD roles. Here's an example with three access tiers:

Navigate to Groups → Create group, and create three groups:
- argocd-admins
- argocd-developers
- argocd-viewers

Assign users to the appropriate groups via Users → select a user → Groups tab → Join group.

6. Store the Client Secret in a Kubernetes Secret

The client secret should never be written as plain text in a ConfigMap. Instead, store it in the argocd-secret Secret that ArgoCD already uses:

kubectl -n argocd patch secret argocd-secret \
  -p '{"stringData": {"oidc.keycloak.clientSecret": "YOUR_CLIENT_SECRET"}}'

7. Configure argocd-cm

Edit the argocd-cm ConfigMap:

kubectl -n argocd edit configmap argocd-cm

Add or ensure the data section contains:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.example.com
  oidc.config: |
    name: Keycloak
    issuer: https://sso.example.com/realms/YOUR_REALM
    clientID: argocd
    clientSecret: $oidc.keycloak.clientSecret
    requestedScopes:
      - openid
      - profile
      - email
      - groups
    requestedIDTokenClaims:
      groups:
        essential: true

Key notes:

clientSecret: $oidc.keycloak.clientSecret — the $key syntax tells ArgoCD to read the value from the argocd-secret Secret (stored in step 6), rather than from plain text in the ConfigMap.
requestedScopes must include groups, matching the client scope created in step 4.

If ArgoCD is managed via a Helm chart (rather than manual kubectl edit), the same configuration can be placed in values.yaml:

configs:
  cm:
    url: https://argocd.example.com
    oidc.config: |
      name: Keycloak
      issuer: https://sso.example.com/realms/YOUR_REALM
      clientID: argocd
      clientSecret: $oidc.keycloak.clientSecret
      requestedScopes: ["openid", "profile", "email", "groups"]
      requestedIDTokenClaims:
        groups:
          essential: true

8. Configure argocd-rbac-cm

Edit the argocd-rbac-cm ConfigMap to map Keycloak groups to ArgoCD roles:

kubectl -n argocd edit configmap argocd-rbac-cm

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  scopes: '[groups]'
  policy.csv: |
    # "argocd-admins" group gets full admin access
    g, argocd-admins, role:admin

    # "argocd-developers" group can deploy to non-prod environments
    p, role:developer, applications, get, */*, allow
    p, role:developer, applications, sync, dev/*, allow
    p, role:developer, applications, sync, staging/*, allow
    p, role:developer, applications, create, dev/*, allow
    p, role:developer, applications, delete, dev/*, allow
    g, argocd-developers, role:developer

    # "argocd-viewers" group is read-only
    g, argocd-viewers, role:readonly

Important notes:

scopes: '[groups]' is required — this tells ArgoCD to read the groups claim from the token when evaluating policies. Without this line, the mapping g, argocd-admins, role:admin will never match, even if the groups claim is present in the token.
policy.default: role:readonly — the default role for users who successfully log in via SSO but don't belong to any group. You can set this to an empty string ("") if you want users without a group assignment to have no access at all.

9. Restart & Test Login

Apply the changes by restarting argocd-server:

kubectl -n argocd rollout restart deployment argocd-server

Wait for the rollout to complete:

kubectl -n argocd rollout status deployment argocd-server

Open https://argocd.example.com and click the "LOG IN VIA KEYCLOAK" button (the label matches the name defined in oidc.config). After successfully authenticating through Keycloak, verify that you have the expected access level based on the group assigned in step 5.

Optionally, verify the assigned role via the CLI (if the argocd CLI is installed):

argocd login argocd.example.com --sso
argocd account get-user-info

10. Break-Glass Account (Local Admin)

ArgoCD ships with a built-in local admin account (its initial password is auto-generated and stored in the argocd-initial-admin-secret Secret). Instead of deleting it, keep it as an emergency access path — similar to the break-glass pattern used in the GitLab integration.

Retrieve the initial password (if you haven't already):

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

Recommendations:

Log in once as admin, change the password to a very strong one, and store it in a password manager.
Do not delete this admin account — keep it active but guard the credentials carefully. Only use it if Keycloak goes down and no OIDC users can log in.
For extra protection, the admin account can be temporarily disabled via accounts.admin.enabled: "false" in argocd-cm after confirming that at least one OIDC group has a working role:admin assignment — but make sure you have a way to re-enable it (via kubectl edit directly against the cluster) before doing this.

11. Offboarding

Unlike GitLab (which requires two separate steps due to account linking), ArgoCD does not store local accounts for OIDC users — access is entirely derived from the groups claim in the token each time a user logs in or re-authenticates. The implications:

Remove the user from their group in Keycloak (or disable/delete the user) — ArgoCD access is automatically revoked the next time the token or session is refreshed, with no additional steps needed on the ArgoCD side.
Active sessions will continue to follow the validity of the already-issued token (requestedIDTokenClaims, Keycloak's default token expiry). If you need immediate revocation, revoke the user's session directly from Keycloak (Users → select user → Sessions tab → Sign out), rather than just removing them from the group.

This is one of the key advantages of group-based RBAC over account linking: there are no separate accounts to manually clean up on the ArgoCD side.

12. Troubleshooting

Symptom	Likely Cause
"LOG IN VIA KEYCLOAK" button doesn't appear	`argocd-cm` hasn't been applied correctly, or `argocd-server` hasn't been restarted after the ConfigMap change
Login succeeds in Keycloak but ArgoCD shows "permission denied" for all resources	The `groups` claim isn't reaching ArgoCD — recheck step 4 (the `groups` client scope must be `Default`, and the mapper must have `Add to ID token: On`), and make sure `scopes: '[groups]'` is present in `argocd-rbac-cm`
Group matches but the role doesn't behave as expected	Check the order and syntax of `policy.csv` — the `g, <group>, role:<role>` line must exactly match the group name in Keycloak (case-sensitive)
Client secret error during login	Ensure the key in the `argocd-secret` Secret is named exactly `oidc.keycloak.clientSecret`, matching what's referenced in `oidc.config`
`invalid_redirect_uri` error from Keycloak	The redirect URI in the Keycloak client must be exactly `https://argocd.example.com/auth/callback`
ConfigMap changes have no effect	`argocd-server` needs a manual restart after `kubectl edit configmap` — ArgoCD doesn't always auto-reload all config sections

Summary: Create an argocd client in Keycloak with Standard Flow only → add a groups client scope with a Group Membership mapper → create groups (argocd-admins, etc.) and assign users → store the client secret in the Kubernetes argocd-secret Secret → configure oidc.config in argocd-cm → map groups to roles in argocd-rbac-cm with scopes: '[groups]' → restart argocd-server → test login → keep the local admin account as a break-glass path → offboard users by simply removing them from their Keycloak group.

Setting Up ArgoCD with Traefik and HTTPS (Let's Encrypt) on a Public Domain

Thu, 16 Jul 2026 00:00:00 GMT

In the previous article, we set up ArgoCD with NGINX Ingress Controller and accessed it via a custom endpoint (sub-path). That setup works great for experimentation and lab environments — ArgoCD is accessed through the node's IP with a path like http://<IP>/argocd-lab, without a domain and without HTTPS.

But what if you have a domain?

Well, this article is the answer. We'll configure ArgoCD to be accessible through a public domain (e.g., argocd.example.com) with HTTPS using certificates automatically generated by Let's Encrypt — all handled by Traefik as the Ingress Controller, without needing to install cert-manager separately.

Intended for kubeadm bare-metal/on-prem clusters with a public domain whose DNS is already pointed to the node's IP.

⚠️ Important Note: Not Production-Ready Yet

[!WARNING] The configuration in this article is not suitable for production because several aspects are still insecure or suboptimal:

Aspect	Issue	Impact
Traefik runs as root (`runAsUser: 0`)	Container runs with full privileges	If the container is exploited, the attacker gains root access on the node
`hostNetwork: true`	Traefik pod shares the host's network namespace	Port conflicts, no network isolation, an attacker who enters the pod gets direct host network access
`allowPrivilegeEscalation: true`	Container can escalate privileges	Significantly expands the attack surface
`readOnlyRootFilesystem: false`	Container can write to any filesystem location	Attacker can modify binaries or configs inside the container
PersistentVolume `hostPath`	Certificate data is stored directly on the node's filesystem	No redundancy, data is lost if the node dies, not portable across nodes
Single replica Traefik	Only one ingress controller instance	Single point of failure — if the pod dies, all traffic to the cluster stops

💡 Recommendations for Production

Here are some recommendations if you plan to bring this setup to production:

1. Use Longhorn for Persistent Volumes

Instead of using hostPath which stores data directly on the node, use Longhorn — a distributed block storage solution for Kubernetes. Longhorn provides:

Data replication across nodes (data survives node failures)
Automatic snapshots & backups
Volume portability — pods can move between nodes without losing data
Built-in monitoring dashboard

2. Use MetalLB for Network Management

Instead of using hostNetwork: true (which bypasses network isolation), use MetalLB as a bare-metal LoadBalancer:

Provides proper external IPs for type: LoadBalancer Services
Eliminates the need for hostNetwork — Traefik can run with normal network isolation
Supports L2 (ARP) and BGP modes
Enables multiple Traefik replicas with the same IP (high availability)

3. Harden the Security Context

For production, improve Traefik's security context:

Use runAsNonRoot: true with NET_BIND_SERVICE capability
Set readOnlyRootFilesystem: true
Set allowPrivilegeEscalation: false
Drop all capabilities except the ones actually needed

A production-grade setup with Longhorn + MetalLB + hardened Traefik will be covered in a separate article. For now, let's focus on getting a functional setup first.

Prerequisites

A running Kubernetes cluster (kubeadm)
A public domain pointed to the node's IP (A record: argocd.example.com → <NODE-IP>)
Ports 80 and 443 on the node are open from the internet
kubectl and helm are configured

If Helm is not yet installed:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

Expected output:

version.BuildInfo{Version:"v4.2.3", GitCommit:"43e8b7feece8beb0fcba47059ec9b522fd929a64", GitTreeState:"clean", GoVersion:"go1.26.5", KubeClientVersion:"v1.36"}

1. Deploy ArgoCD

Download the official ArgoCD manifest.

wget https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Edit the manifest to disable ArgoCD's built-in TLS, since HTTPS will be handled entirely by Traefik in front of it.

nano install.yaml

Find the Deployment named argocd-server and change the args section to:

spec:
  containers:
  - args:
    - /usr/local/bin/argocd-server
    - --staticassets
    - /shared/app
    - --insecure

Why is --insecure safe here? This flag only makes ArgoCD serve HTTP from the pod side. Traefik sits in front and handles TLS termination — so external communication remains HTTPS-encrypted. Internal communication from Traefik to the ArgoCD pod goes over HTTP within the cluster, which is already isolated.

Create the namespace and deploy.

kubectl create namespace argocd
kubectl apply --server-side -f install.yaml -n argocd

Why use --server-side? Recent ArgoCD versions have very large CRDs, especially applicationsets.argoproj.io. By default, kubectl apply stores the entire manifest content as a kubectl.kubernetes.io/last-applied-configuration annotation on the CRD object itself. Kubernetes limits annotation size to a maximum of 256KB (262144 bytes) — and ArgoCD's CRDs exceed this limit, causing regular kubectl apply to fail with:
The CustomResourceDefinition "applicationsets.argoproj.io" is invalid:
metadata.annotations: Too long: may not be more than 262144 bytes
With --server-side, the apply process is handled on the API server side (not the client), so the last-applied-configuration annotation is not stored and the 256KB limit doesn't apply.

If a CRD was previously applied the regular way and a conflict occurs, add the --force-conflicts flag:
kubectl apply --server-side --force-conflicts -f install.yaml -n argocd

Wait until all pods are in Running status.

kubectl get pods -n argocd

2. Install Traefik via Helm

Add the official Traefik Helm repository.

helm repo add traefik https://traefik.github.io/charts
helm repo update

Expected output:

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "traefik" chart repository
Update Complete. ⎈Happy Helming!⎈

First, create a directory on the host and a PersistentVolume to persist Let's Encrypt certificates (acme.json).

Create the directory on the host node:

mkdir -p /data/traefik
chmod 700 /data/traefik

Create a pv.yaml file:

nano pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: traefik-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data/traefik
  persistentVolumeReclaimPolicy: Retain

kubectl apply -f pv.yaml

Create a traefik-values.yaml file to configure Traefik for bare-metal and Let's Encrypt.

nano traefik-values.yaml

# Use Deployment (not DaemonSet) because Traefik v3 does not support
# ACME/Let's Encrypt when running as a DaemonSet — each node would
# try to request certificates independently and conflict on acme.json
deployment:
  kind: Deployment
  replicas: 1

hostNetwork: true

service:
  type: ClusterIP

ports:
  web:
    port: 80
    hostPort: 80
    expose:
      default: true
  websecure:
    port: 443
    hostPort: 443
    expose:
      default: true

additionalArguments:
  - "--certificatesresolvers.letsencrypt.acme.email=email@example.com"
  - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
  - "--certificatesresolvers.letsencrypt.acme.tlsChallenge=true"

persistence:
  enabled: true
  storageClass: ""
  size: 128Mi

ingressClass:
  enabled: true
  isDefaultClass: false
  name: traefik

ingressRoute:
  dashboard:
    enabled: false

# ==============================================================================
# ROOT CONFIGURATION
# Forces Traefik to run as User 0 (Root) so it can freely use
# host ports 80/443 and manage the acme.json file
# ==============================================================================
podSecurityContext:
  runAsUser: 0
  runAsGroup: 0
  runAsNonRoot: false
  fsGroup: 0

securityContext:
  readOnlyRootFilesystem: false
  runAsNonRoot: false
  runAsUser: 0
  runAsGroup: 0
  allowPrivilegeEscalation: true
  capabilities:
    drop: []

Replace email@example.com with your active email address for Let's Encrypt notifications.

Breaking changes in Traefik v3 (Helm chart >= 30.x):

certResolvers at the values level is no longer recognized — ACME configuration is now passed directly via additionalArguments to the Traefik binary

tls inside ports.websecure is no longer allowed in the v3 schema

tlsStore.default.defaultCertResolver has been removed from the CRD — do not use it

Cert resolver is specified per-Ingress via annotation: traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt

Explanation of other configuration choices:

Deployment + hostNetwork: true — Traefik binds directly to host ports, the simplest solution for bare-metal without a LoadBalancer. Using Deployment (not DaemonSet) ensures only one replica manages ACME certificates

tlsChallenge=true — Traefik verifies the domain with Let's Encrypt via the TLS-ALPN-01 challenge directly on port 443, no need for port 80 for the challenge (but port 80 is still open for HTTP → HTTPS redirect)

persistence.enabled: true — certificates are stored on a PersistentVolume so they don't need to be re-requested from Let's Encrypt every time the pod restarts (Let's Encrypt has strict rate limits)

Install Traefik to the traefik namespace.

helm install traefik traefik/traefik \
  --namespace traefik \
  --create-namespace \
  --values traefik-values.yaml

Expected output:

NAME: traefik
LAST DEPLOYED: Thu Jul 16 21:08:26 2026
NAMESPACE: traefik
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
NOTES:
traefik with docker.io/traefik:v3.7.6 has been deployed successfully on traefik namespace!

Verify that the Traefik pod is Running and the IngressClass is registered.

kubectl get pods -n traefik
kubectl get ingressclass

Expected output:

NAME      CONTROLLER                      PARAMETERS   AGE
traefik   traefik.io/ingress-controller   <none>       30s

3. Create an Ingress for ArgoCD

Create an ingress.yaml file. Traefik will automatically request a certificate from Let's Encrypt based on the traefik.ingress.kubernetes.io/router.tls.certresolver annotation.

nano ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    # Use the certresolver configured in Traefik values
    traefik.ingress.kubernetes.io/router.tls.certresolver: letsencrypt
    # Redirect all HTTP to HTTPS
    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
    traefik.ingress.kubernetes.io/router.middlewares: argocd-redirect-https@kubernetescrd
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - argocd.example.com
  rules:
  - host: argocd.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 80

Also create a Middleware for HTTP to HTTPS redirect.

nano middleware.yaml

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
  namespace: argocd
spec:
  redirectScheme:
    scheme: https
    permanent: true

Apply both.

kubectl apply -f middleware.yaml
kubectl apply -f ingress.yaml

Verify the ingress is registered.

kubectl get ingress -n argocd

Expected output:

NAME                    CLASS     HOSTS                 ADDRESS          PORTS     AGE
argocd-server-ingress   traefik   argocd.example.com    192.168.205.71   80, 443   1m

Monitor the certificate issuance process from the Traefik logs. This usually takes 1-2 minutes.

kubectl logs -n traefik -l app.kubernetes.io/name=traefik --follow

The certificate has been successfully issued if you see logs like:

2026-07-16T14:41:28Z INF Creating in-cluster Provider client providerName=kubernetescrd
2026-07-16T14:41:28Z INF Starting provider *acme.Provider
2026-07-16T14:41:28Z INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=letsencrypt.acme
2026-07-16T14:45:02Z INF Register... providerName=letsencrypt.acme
2026-07-16T14:45:02Z INF Registering the account. email=email@example.com lib=lego
2026-07-16T14:45:02Z INF Obtaining bundled SAN certificate. domains=argocd.example.com lib=lego
2026-07-16T14:45:03Z INF Use solver. domain=argocd.example.com lib=lego type=tls-alpn-01
2026-07-16T14:45:03Z INF tlsalpn01: Trying to solve TLS-ALPN-01. domain=argocd.example.com lib=lego
2026-07-16T14:45:08Z INF The server validated our request. domain=argocd.example.com lib=lego
2026-07-16T14:45:08Z INF Validations succeeded; requesting certificates. domains=argocd.example.com lib=lego
2026-07-16T14:45:13Z INF Server responded with a certificate. domains=argocd.example.com lib=lego

4. Install the ArgoCD CLI

Download and install the ArgoCD CLI binary.

curl -SL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64
argocd version

5. Login and Password Setup

5.1 Retrieve the Initial Admin Password

argocd admin initial-password -n argocd

5.2 Login via CLI

Since we're already using HTTPS with a valid certificate from Let's Encrypt, login is done directly without the --insecure or --plaintext flags.

argocd login argocd.example.com

Username: admin
Password: <password-from-the-previous-step>

If the certificate hasn't finished being issued yet, you'll get a TLS error. Wait a few minutes and try again.

5.3 Change the Admin Password

argocd account update-password

6. Access ArgoCD via Browser

https://argocd.example.com

The browser shows the HTTPS padlock because the certificate from Let's Encrypt is valid. Accessing via http:// will automatically redirect to https://.

7. Tuning the Refresh Time Interval

nano config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  timeout.reconciliation: 20s

kubectl apply -f config.yaml
kubectl describe configmap -n argocd argocd-cm
kubectl -n argocd rollout restart deploy argocd-repo-server

Troubleshooting

Certificate is not being issued

Make sure port 443 is open from the internet. Check the Traefik logs for error details:

kubectl logs -n traefik -l app.kubernetes.io/name=traefik | grep -i "acme\|cert\|error"

argocd login fails with a TLS error

The certificate hasn't finished being issued yet. Wait until the Obtained certificate log appears in Traefik, then try logging in again.

Traefik pod won't start

Make sure no other process is already using port 80 or 443 on the node. Check with:

ss -tlnp | grep -E ':80|:443'

Certificate expired and not auto-renewing

Make sure the PersistentVolume for Traefik still exists and is accessible:

kubectl get pvc -n traefik

Comparison: Traefik vs NGINX Ingress + cert-manager

Aspect	Traefik (this doc)	NGINX + cert-manager
Components	Traefik only	NGINX Ingress + cert-manager + ClusterIssuer
Let's Encrypt	Built-in, configured in values	Separate via cert-manager CRDs
Configuration	`traefik-values.yaml` + annotations	`ingress.yaml` + `clusterissuer.yaml`
Monitoring dashboard	Available (optional)	No built-in dashboard
Extra CRDs	`IngressRoute`, `Middleware`	`Certificate`, `ClusterIssuer`
Complexity	Simpler (fewer components)	More components but more flexible
Best suited for	Lab, staging, simple production	Production with many different certificates

References

This article is a continuation of the bare-metal Kubernetes installation series. Adjust domains, IPs, and version numbers to match your own environment.

Deploying Keycloak 26 in Production on AlmaLinux 9 with PostgreSQL, Nginx & Let's Encrypt

Mon, 13 Jul 2026 00:00:00 GMT

This guide covers a full production deployment of Keycloak 26.7.0 on AlmaLinux 9. We'll use OpenJDK 25 as the runtime, Percona Distribution for PostgreSQL 17 as the database, run Keycloak as a systemd service, and secure it behind an Nginx reverse proxy with a free Let's Encrypt TLS certificate.

Placeholders used throughout this guide: Wherever you see values like your-domain.example.com, YOUR_STRONG_DB_PASSWORD, or YOUR_ADMIN_PASSWORD, replace them with your own. A summary of all placeholders is provided below.

Placeholder	Description	Example
`your-domain.example.com`	Your Keycloak FQDN (must have a DNS A record pointing to your server's public IP)	`sso.example.com`
`YOUR_STRONG_DB_PASSWORD`	Password for the PostgreSQL `kc_user` database user	Use a password manager to generate one
`YOUR_ADMIN_USERNAME`	Keycloak admin console username	`admin`
`YOUR_ADMIN_PASSWORD`	Keycloak admin console password	Use a strong, unique password

Prerequisites

AlmaLinux 9 (fresh install recommended), minimum 2 vCPU / 4 GB RAM for small-to-medium workloads.
Root access or a user with sudo privileges.
A domain name (e.g., your-domain.example.com) with an A record pointing to your server's public IP.
Ports 80 and 443 open to the internet for Let's Encrypt validation and HTTPS traffic.

Component Versions

Component	Version
OS	AlmaLinux 9
Java	OpenJDK 25
Database	PostgreSQL 17 (Percona Distribution)
Keycloak	26.7.0
Web Server	Nginx (reverse proxy)
SSL	Let's Encrypt (Certbot)

1. System Preparation

Update existing packages and install basic utilities:

sudo dnf update -y
sudo dnf install -y wget curl unzip tar policycoreutils-python-utils

Optionally, set the hostname for log and certificate consistency:

# Replace with your actual domain
sudo hostnamectl set-hostname your-domain.example.com

2. Install OpenJDK 25

The default AlmaLinux 9 AppStream repository only provides up to OpenJDK 21. For OpenJDK 25, we'll download the official build directly.

2.1 Download & Extract

cd /opt
sudo wget https://download.java.net/java/GA/jdk25.0.2/b1e0dfa218384cb9959bdcb897162d4e/10/GPL/openjdk-25.0.2_linux-x64_bin.tar.gz

sudo tar -xzf openjdk-25.0.2_linux-x64_bin.tar.gz
sudo mv jdk-25.0.2 /opt/jdk-25
sudo rm openjdk-25.0.2_linux-x64_bin.tar.gz

Tip: The latest build download links are always available at https://jdk.java.net/25/ — check for the latest build number before installing, as the URL changes with each update release.

2.2 Register via `alternatives`

sudo alternatives --install /usr/bin/java java /opt/jdk-25/bin/java 2500
sudo alternatives --install /usr/bin/javac javac /opt/jdk-25/bin/javac 2500

sudo alternatives --set java /opt/jdk-25/bin/java
sudo alternatives --set javac /opt/jdk-25/bin/javac

2.3 Set `JAVA_HOME` System-Wide

sudo tee /etc/profile.d/java25.sh > /dev/null <<'EOF'
export JAVA_HOME=/opt/jdk-25
export PATH=$JAVA_HOME/bin:$PATH
EOF

source /etc/profile.d/java25.sh

2.4 Verify

java --version

Expected output:

openjdk 25.0.2 2026-01-20
OpenJDK Runtime Environment (build 25.0.2+10-69)
OpenJDK 64-Bit Server VM (build 25.0.2+10-69, mixed mode, sharing)

3. Install & Configure PostgreSQL 17 (Percona Distribution)

As an alternative to the community PostgreSQL (PGDG) packages, this guide uses Percona Distribution for PostgreSQL — a 100% upstream-compatible PostgreSQL build from Percona, bundled with operational extensions (pgBackRest, pg_stat_monitor, pgAudit, Patroni, etc.) useful for long-term production needs. Official reference: docs.percona.com/postgresql/17/yum.html.

The service structure, data paths, and binary names follow upstream PostgreSQL conventions (postgresql-17.service, /usr/pgsql-17/, /var/lib/pgsql/17/data/), so all post-installation steps (initialization, pg_hba.conf, systemd) are identical.

3.1 Install Dependencies & Percona Repository

sudo dnf -y install curl

# Install the Percona repository management tool
sudo dnf install -y https://repo.percona.com/yum/percona-release-latest.noarch.rpm

# Enable the Percona Distribution for PostgreSQL 17 repository
sudo percona-release setup ppg17

# Disable the default AppStream postgresql module to avoid conflicts
sudo dnf -qy module disable postgresql

3.2 Install Percona PostgreSQL Server

sudo dnf install -y percona-postgresql17-server percona-postgresql17-contrib

3.3 Initialize & Start the Service

sudo /usr/pgsql-17/bin/postgresql-17-setup initdb

sudo systemctl enable --now postgresql-17
sudo systemctl status postgresql-17

Verify that the installed build is from Percona:

psql --version

Expected output:

psql (PostgreSQL) 17.10 - Percona Server for PostgreSQL 17.10.2

3.4 Create Database & User for Keycloak

# ⚠️ Replace YOUR_STRONG_DB_PASSWORD with a secure password
sudo -u postgres psql <<'EOF'
CREATE DATABASE keycloak;
CREATE USER kc_user WITH ENCRYPTED PASSWORD 'YOUR_STRONG_DB_PASSWORD';
GRANT ALL PRIVILEGES ON DATABASE keycloak TO kc_user;
ALTER DATABASE keycloak OWNER TO kc_user;
EOF

3.5 Enable Password Authentication (scram-sha-256)

By default, PostgreSQL on RHEL-family systems uses peer authentication for local connections. Since Keycloak connects via TCP (127.0.0.1), we need to add a password authentication entry to pg_hba.conf:

sudo tee -a /var/lib/pgsql/17/data/pg_hba.conf > /dev/null <<'EOF'
host    keycloak    kc_user    127.0.0.1/32    scram-sha-256
EOF

sudo systemctl restart postgresql-17

4. Install Keycloak 26.7.0

4.1 Create a Dedicated System User

sudo groupadd keycloak
sudo useradd -r -g keycloak -d /opt/keycloak -s /sbin/nologin keycloak

4.2 Download & Extract Keycloak

cd /tmp
wget https://github.com/keycloak/keycloak/releases/download/26.7.0/keycloak-26.7.0.zip

sudo unzip keycloak-26.7.0.zip -d /opt/
sudo mv /opt/keycloak-26.7.0 /opt/keycloak
sudo chown -R keycloak:keycloak /opt/keycloak

5. Configure `keycloak.conf`

Edit the Keycloak configuration file:

sudo nano /opt/keycloak/conf/keycloak.conf

Paste the following configuration:

# ---------- Database ----------
db=postgres
db-username=kc_user
db-password=YOUR_STRONG_DB_PASSWORD          # ⚠️ Use the same password from step 3.4
db-url=jdbc:postgresql://127.0.0.1:5432/keycloak

# ---------- Hostname ----------
hostname=your-domain.example.com             # ⚠️ Replace with your actual domain
hostname-strict-backchannel=true

# ---------- Proxy (behind Nginx) ----------
proxy-headers=xforwarded
http-enabled=true
http-host=127.0.0.1
http-port=8080

# ---------- Production Features ----------
health-enabled=true
metrics-enabled=true

Since TLS termination is handled by Nginx in front, Keycloak only needs to listen for HTTP connections locally (127.0.0.1:8080) and is never directly exposed to the internet.

6. Build & Bootstrap Admin

6.1 Build Optimized Configuration

cd /opt/keycloak
sudo -u keycloak bin/kc.sh build

Expected output:

Updating the configuration and installing your custom providers, if any. Please wait.
...
Server configuration updated and persisted. Run the following command to review the configuration:

        kc.sh show-config

6.2 Create the Initial Admin Account

Since Keycloak 26.x, the --password flag on the bootstrap-admin command has been removed for security reasons (to avoid passwords being recorded in shell history or visible via ps aux). Use one of the following methods:

Method 1 — Interactive (recommended for a one-time manual setup):

# ⚠️ Replace YOUR_ADMIN_USERNAME with your desired admin username
sudo -u keycloak bin/kc.sh bootstrap-admin user --username YOUR_ADMIN_USERNAME

You will be prompted to type the password directly in the terminal. Choose a strong, unique password and store it securely.

Method 2 — Via Environment Variable (for automation/scripted setups):

# ⚠️ Replace the values below with your desired admin credentials
export KC_BOOTSTRAP_ADMIN_PASSWORD='YOUR_ADMIN_PASSWORD'

sudo -u keycloak -E bin/kc.sh bootstrap-admin user \
  --username YOUR_ADMIN_USERNAME \
  --password:env KC_BOOTSTRAP_ADMIN_PASSWORD

The -E flag on sudo is required to pass the KC_BOOTSTRAP_ADMIN_PASSWORD environment variable to the process running as the keycloak user. After the account is created, remove the variable from your shell:

unset KC_BOOTSTRAP_ADMIN_PASSWORD

7. Systemd Service

Create the unit file at /etc/systemd/system/keycloak.service:

[Unit]
Description=Keycloak Identity and Access Management Server
After=network.target postgresql-17.service
Wants=postgresql-17.service

[Service]
Type=simple
User=keycloak
Group=keycloak
WorkingDirectory=/opt/keycloak
Environment="JAVA_HOME=/opt/jdk-25"
Environment="JAVA_OPTS=-Xms512m -Xmx2048m"
ExecStart=/opt/keycloak/bin/kc.sh start --optimized
ExecStop=/opt/keycloak/bin/kc.sh stop
Restart=on-failure
RestartSec=10
TimeoutStartSec=300

# Basic hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/opt/keycloak

[Install]
WantedBy=multi-user.target

Why PrivateTmp=true? This directive is essential when used alongside ProtectSystem=strict. Without it, Keycloak (via Quarkus/Vert.x) will fail to start with a Read-only file system error when attempting to write temporary cache files to /tmp, because ProtectSystem=strict locks the entire filesystem except paths listed in ReadWritePaths. PrivateTmp=true gives the service its own private, writable /tmp without weakening the overall hardening posture.

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now keycloak
sudo systemctl status keycloak

Check logs if anything goes wrong:

sudo journalctl -u keycloak -f

8. Reverse Proxy with Nginx + SSL (Let's Encrypt)

8.1 Install Nginx & Certbot

sudo dnf install -y epel-release
sudo dnf install -y nginx certbot python3-certbot-nginx

sudo systemctl enable --now nginx

8.2 Create the Virtual Host Configuration

# ⚠️ Replace your-domain.example.com with your actual domain (in both the filename and content)
sudo tee /etc/nginx/conf.d/your-domain.example.com.conf > /dev/null <<'EOF'
server {
    listen 80;
    server_name your-domain.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;
    }
}
EOF

sudo nginx -t
sudo systemctl reload nginx

8.3 Issue the TLS Certificate

# ⚠️ Replace your-domain.example.com with your actual domain
sudo certbot --nginx -d your-domain.example.com

Certbot will automatically add the listen 443 ssl block and redirect HTTP traffic to HTTPS. Certificate renewal runs automatically via the systemd timer (certbot-renew.timer), which you can verify with:

systemctl list-timers | grep certbot

9. Firewall & SELinux

9.1 Firewalld

AlmaLinux 9 runs firewalld by default, so all port management in this guide is done through firewall-cmd rather than disabling the firewall.

Check the active zone and firewalld status:

sudo systemctl status firewalld
sudo firewall-cmd --get-default-zone
sudo firewall-cmd --list-all

Open HTTP and HTTPS services (required for Let's Encrypt validation and traffic to Nginx):

sudo firewall-cmd --permanent --add-service=http
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

Verify allowed services:

sudo firewall-cmd --list-services

Ports intentionally NOT opened in firewalld:

8080 (Keycloak internal) — Only bound to 127.0.0.1 and accessed locally by Nginx. It must not be exposed to the public internet.
5432 (PostgreSQL) — Only accessed locally by Keycloak. No external access is needed.

Since neither service is added with --add-port, firewalld automatically rejects all inbound traffic to these ports. You can confirm this with:

sudo firewall-cmd --list-ports
# Expected: empty or not showing 8080/5432

Optional — Restrict Admin Console access to specific IPs/CIDRs. If you want to limit who can access /admin (e.g., only from your office network or VPN), use a rich rule on the firewalld zone facing the server's public IP:

# ⚠️ Replace 203.0.113.0/24 with your actual trusted network CIDR
sudo firewall-cmd --permanent --zone=public --add-rich-rule='
  rule family="ipv4" source address="203.0.113.0/24" service name="https" accept'

sudo firewall-cmd --permanent --zone=public --remove-service=https
sudo firewall-cmd --reload

Warning: The commands above change HTTPS from "open to all" to "only from defined source IPs." Only use this if all Keycloak users access from networks with predictable IPs (e.g., via VPN). If the CIDR is wrong, access will be cut off for everyone, including admins.

9.2 SELinux

AlmaLinux 9 runs SELinux in enforcing mode by default. To allow Nginx to proxy to the Keycloak backend (port 8080), enable the following boolean:

sudo setsebool -P httpd_can_network_connect on

If other access denials appear in the logs, investigate with:

sudo ausearch -m avc -ts recent

10. Verification

Open https://your-domain.example.com/admin/ in your browser and log in using the admin account created in step 6.2.

After a successful login, you should see the Keycloak Administration Console:

You can also verify the health endpoint (if health-enabled=true):

curl -s http://127.0.0.1:8080/health/ready

Expected response:

{"status": "UP", "checks": []}

Troubleshooting

Symptom	Likely Cause
`systemctl status keycloak` shows failure	Check `journalctl -u keycloak -f`; often caused by database connection not ready or port conflict
`password authentication failed` database error	Verify `pg_hba.conf` entries and make sure the password in `keycloak.conf` matches the one set in step 3.4
Nginx 502 Bad Gateway	Keycloak is not listening on `127.0.0.1:8080`; check `http-host`/`http-port` in `keycloak.conf` and the Keycloak service status
Redirect loop or "invalid redirect_uri" on client login	The redirect URI configured in Keycloak does not exactly match the one sent by the application (watch for trailing slashes)
Let's Encrypt certificate fails to issue	Ensure DNS for the domain has propagated and port 80 is open when `certbot` runs
Nginx cannot proxy despite correct configuration	Likely blocked by SELinux; run `setsebool -P httpd_can_network_connect on`

Summary: Update system → Install OpenJDK 25 manually → Install Percona Distribution for PostgreSQL 17 → Download & build Keycloak 26.7.0 → Run as a systemd service → Expose via Nginx with a Let's Encrypt certificate → Manage port access with firewall-cmd and adjust SELinux policies.

Mailpit: A Self-Hosted Mailtrap Alternative for Email Testing

Sun, 05 Jul 2026 00:00:00 GMT

If you frequently develop features that involve sending emails—such as password resets, notifications, invoices, and more—you are probably familiar with Mailtrap. It is an incredibly useful tool because it intercepts emails sent from your application without actually delivering them to real mailboxes, allowing you to safely inspect them during development.

However, Mailtrap's free plan has limitations (email volume, number of inboxes, retention, etc.). If you need a similar tool but want it self-hosted, completely free, and running entirely on your own local machine or staging server, Mailpit is an excellent alternative.

What is Mailpit?

Mailpit is an open-source, lightweight, and fast email testing tool with minimal dependencies. Conceptually, Mailpit plays two roles at once:

SMTP Server — Your application sends emails to Mailpit just like it would to a real SMTP provider, and Mailpit intercepts them.
Web UI & API — You can view intercepted emails through a modern web dashboard (supporting HTML, plain text, headers, raw source, and attachments) or fetch them via an API for automated testing.

Since it runs as a single binary or a Docker image and doesn't require an external database, Mailpit is extremely easy to run in local development or staging environments.

Why Choose Mailpit as a Mailtrap Alternative?

Feature	Mailtrap	Mailpit
Hosting	Cloud (SaaS)	Self-hosted (Docker)
Cost	Free with limits, paid for more	Free, limited only by your server resources
Data Privacy	Stored on third-party servers	Stored on your own server or volume
Setup	Sign up, create inbox, get credentials	`docker compose up` and it runs
Authentication	Built-in / Required	Configurable (explained below)

For development and internal staging needs, Mailpit is a perfect fit because it gives you full control over your data without any subscription costs.

Docker Compose Setup

Here is the docker-compose.yml configuration. It includes login protection for the Web UI and SMTP authentication (replicating Mailtrap's requirement for credentials):

services:
  mailpit:
    image: ghcr.io/axllent/mailpit:v1.30.1@sha256:f35e6c915ecec9c0bb1837ee176554c695f4558eda25a6d82c1d30ba2145f769
    container_name: mailpit
    restart: unless-stopped
    volumes:
      - ./data:/data
      # Mount credentials file to protect UI and SMTP
      - ./auth.txt:/auth.txt:ro
    ports:
      - "8025:8025" # Web UI Dashboard
      - "1025:1025" # SMTP Port
    environment:
      MP_MAX_MESSAGES: 5000
      MP_DATABASE: /data/mailpit.db

      # 1. Lock the Web UI Dashboard (requires login like Mailtrap)
      MP_UI_AUTH_FILE: /auth.txt

      # 2. Require valid SMTP authentication from the auth file
      MP_SMTP_AUTH_FILE: /auth.txt

      # MP_SMTP_AUTH_ACCEPT_ANY: 1  # <-- Disable to enforce strict authentication

      # Allow plain-text login from the application (for internal network without SSL/TLS)
      MP_SMTP_AUTH_ALLOW_INSECURE: 1

Environment Variables Breakdown

MP_MAX_MESSAGES — The maximum number of emails stored before older ones are automatically pruned.
MP_DATABASE — The path to the SQLite database file where Mailpit stores emails.
MP_UI_AUTH_FILE — Enables Basic Auth on the Web UI, pointing to the credentials file.
MP_SMTP_AUTH_FILE — Requires applications to authenticate with a valid username/password, pointing to the same credentials file.
MP_SMTP_AUTH_ALLOW_INSECURE — Allows SMTP authentication over plain-text (non-TLS) connections. This is suitable for internal testing environments, but should be avoided if Mailpit is exposed to the public internet.

Note: If you want SMTP authentication enabled but don't care about validation (e.g., local development where any credentials should work), you can set MP_SMTP_AUTH_ACCEPT_ANY: 1 instead. Mailpit will then accept any username and password combination.

Generating the Authentication File (`auth.txt`)

This is where people often run into issues. Mailpit does not read plain-text passwords directly from the auth.txt file. Instead, the passwords must be hashed (Mailpit supports formats like bcrypt, commonly used in standard htpasswd files).

The most practical way to generate this is using the htpasswd utility from the apache2-utils package (Debian/Ubuntu) or httpd-tools (RHEL/CentOS).

1. Install `htpasswd` (if not already installed)

# Ubuntu/Debian
sudo apt install apache2-utils

# RHEL/CentOS/Fedora
sudo dnf install httpd-tools

2. Generate the Credentials

Create a new auth.txt file with your first user using the -c (create) and -B (bcrypt) flags:

htpasswd -B -c auth.txt test@mailer.com

You will be prompted to enter and confirm a password:

New password: 
Re-type new password: 
Adding password for user test@mailer.com

If you want to add additional users (e.g., for team members), run the command again without the -c flag so you don't overwrite the existing file:

htpasswd -B auth.txt dev2@mailer.com

The resulting auth.txt file will look like this:

test@mailer.com:$2y$05$examplehashedpasswordstringxxxxxxxxxxxxxxxxxxxxxx
dev2@mailer.com:$2y$05$anotherexamplehashxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Save this file in the same directory as your docker-compose.yml, then start the container:

docker compose up -d

3. Using the Credentials

Log in to the Web UI at http://localhost:8025 using test@mailer.com and the password you created.
Configure SMTP in your application (e.g., in Laravel .env, Node.js, etc.):

MAIL_HOST=localhost
MAIL_PORT=1025
MAIL_USERNAME=test@mailer.com
MAIL_PASSWORD=your_configured_password
MAIL_ENCRYPTION=null

Since we enabled MP_SMTP_AUTH_ALLOW_INSECURE=1, the application can authenticate without TLS, which is perfect for local or testing setups.

Conclusion

With this setup, you now have a fully functional email testing environment running on your own infrastructure. It is secure, limit-free, and costs nothing. For small teams or personal projects, Mailpit is a incredibly solid replacement for Mailtrap.

If you plan to expose Mailpit to the public internet, make sure to add a reverse proxy with HTTPS (such as Nginx or Caddy) in front of port 8025, and avoid using MP_SMTP_AUTH_ALLOW_INSECURE in non-isolated networks.

Setting Up ArgoCD with NGINX Ingress on a Bare-Metal Kubernetes Cluster

Sun, 21 Jun 2026 00:00:00 GMT

This guide walks you through setting up ArgoCD — a declarative, GitOps-based continuous delivery tool — on a bare-metal Kubernetes cluster. By the end, you'll have ArgoCD fully accessible through a web browser via an NGINX Ingress Controller, and you'll deploy sample applications using both the CLI and the Web UI.

Note: This guide assumes you already have a running Kubernetes cluster (e.g., set up with kubeadm). If you don't have one yet, check out my previous post on Kubernetes Single-Node Setup.

Installing ArgoCD
Setting Up NGINX Ingress Controller
Creating an Ingress for ArgoCD
Installing the ArgoCD CLI
Initial Login and Password Setup
Accessing the ArgoCD Web UI
Tuning the Refresh Interval
Deploying an Application via CLI
Deploying an Application via Web UI
ArgoCD CLI Tips

1. Installing ArgoCD

1.1 Download the Installation Manifest

First, download the official ArgoCD installation manifest:

wget https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

1.2 Configure a Custom Root Path (Optional)

If you want to access ArgoCD through a sub-path (e.g., http://<your-ip>/argocd-lab) instead of using a dedicated domain, you'll need to modify the argocd-server Deployment in the downloaded install.yaml.

Find the Deployment resource named argocd-server and add the --rootpath, --basehref, and --insecure arguments:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/name: argocd-server
    app.kubernetes.io/part-of: argocd
  name: argocd-server
...
# Add these under spec.template.spec.containers[0].args
spec:
     containers:
     - args:
       - /usr/local/bin/argocd-server
       - --staticassets
       - /shared/app
       - --insecure
       - --rootpath
       - /argocd-lab
       - --basehref
       - /argocd-lab

Note: The path /argocd-lab is used in this guide as an example to avoid conflicts with ArgoCD's default /argocd path. Feel free to change it to whatever path suits your needs — just make sure it stays consistent across your Ingress configuration and CLI login commands. If you're using a dedicated domain or want ArgoCD on the root URL, you can skip the --rootpath and --basehref flags entirely.

1.3 Create the Namespace and Deploy

Create the argocd namespace and apply the manifest:

kubectl create namespace argocd

kubectl apply -f install.yaml -n argocd

2. Setting Up NGINX Ingress Controller

A Note on NGINX Ingress: We are using NGINX Ingress Controller here because it is very easy to set up for demonstration purposes. However, if you are planning to deploy this in a production environment, you should consider alternatives that provide better long-term security support and active maintenance (such as Traefik, HAProxy, Contour, etc.).

Important for bare-metal/on-prem clusters: Clusters bootstrapped with kubeadm don't come with a built-in Ingress Controller. Without one, any Ingress resource you create won't work — kubectl get ingress will show no ADDRESS, and you'll get connection refused errors when trying to access services.

2.1 Check If an Ingress Controller Exists

kubectl get ingressclass

If the output is No resources found, you need to install one.

2.2 Install NGINX Ingress Controller

Since we're on bare-metal (no cloud LoadBalancer), install the NGINX Ingress Controller using the bare-metal manifest:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/baremetal/deploy.yaml

2.3 Enable Host Networking

Edit the ingress-nginx-controller Deployment to enable hostNetwork, which allows the controller to bind directly to port 80/443 on the node — no external LoadBalancer or MetalLB required:

kubectl edit deployment ingress-nginx-controller -n ingress-nginx

Add hostNetwork: true under spec.template.spec, at the same level as containers:

spec:
  template:
    spec:
      hostNetwork: true
      containers:
      - name: controller
        ...

2.4 Verify the Installation

Wait a moment, then confirm the controller pod is running and the IngressClass is registered:

kubectl get pods -n ingress-nginx

kubectl get ingressclass

Expected output:

NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       30s

3. Creating an Ingress for ArgoCD

Create an ingress.yaml file to expose ArgoCD through the /argocd-lab path:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /argocd-lab
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 80

Apply it:

kubectl apply -f ingress.yaml

4. Installing the ArgoCD CLI

Download and install the ArgoCD CLI binary:

curl -SL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64

sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd

Clean up the downloaded file:

rm argocd-linux-amd64

Verify the installation:

argocd version

Expected output:

argocd: v3.4.4+443415b
  BuildDate: 2026-06-18T09:15:00Z
  GitCommit: 443415b5527ac55366e0760c93ef0e1abd0cf273
  GitTreeState: clean
  GoVersion: go1.26.0
  Compiler: gc
  Platform: linux/amd64

Note: You may see a fatal message about the server address being unspecified — this is normal before logging in.

5. Initial Login and Password Setup

5.1 Retrieve the Default Admin Password

ArgoCD generates a random initial password for the admin account during installation:

argocd admin initial-password -n argocd

Example output:

[PASSWORD]

This password must be only used for first time login. We strongly recommend
you update the password using `argocd account update-password`.

5.2 Log In via CLI

argocd login <your-ip>:80 --grpc-web-root-path=/argocd-lab --plaintext --insecure

When prompted, enter admin as the username and the password from the previous step.

5.3 Change the Admin Password

For security, update the default password immediately:

argocd account update-password

*** Enter password of currently logged in user (admin):
*** Enter new password for user admin:
*** Confirm new password for user admin:
Password updated
Context '<your-ip>:80/argocd-lab' updated

6. Accessing the ArgoCD Web UI

Open your browser and navigate to:

http://<your-ip>/argocd-lab

7. Tuning the Refresh Interval

By default, ArgoCD polls your Git repositories every 3 minutes to check for changes. If you want faster feedback (e.g., during development), you can lower this interval.

Create a config.yaml file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  timeout.reconciliation: 20s

Apply the configuration:

kubectl apply -f config.yaml

Verify the change:

kubectl describe configmap -n argocd argocd-cm

Restart the repo server to pick up the new setting:

kubectl -n argocd rollout restart deploy argocd-repo-server

Tip: A 20-second interval is great for development and testing, but for production environments you may want to keep it at the default 3 minutes (or higher) to reduce load on both ArgoCD and your Git server.

8. Deploying an Application via CLI

8.1 Create the Application

Create a sample application called simple-deploy pointing to a Git repository:

argocd app create simple-deploy \
  --repo https://github.com/maliqpotter/argo-simple-project.git \
  --path simple-demo \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default

application 'simple-deploy' created

8.2 Check the Application Status

argocd app get simple-deploy

The initial status will be OutOfSync because no Kubernetes resources have been created yet:

Name:               argocd/simple-deploy
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
Source:
- Repo:             https://github.com/maliqpotter/argo-simple-project.git
  Path:             simple-demo
Sync Status:        OutOfSync from  (1873ef1)
Health Status:      Missing

GROUP  KIND        NAMESPACE  NAME          STATUS     HEALTH   HOOK  MESSAGE
apps   Deployment  default    simple-nginx  OutOfSync  Missing

8.3 Sync (Deploy) the Application

Trigger a sync to deploy the application:

argocd app sync simple-deploy

Once the sync completes, verify the status:

argocd app get simple-deploy

Name:               argocd/simple-deploy
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
Source:
- Repo:             https://github.com/maliqpotter/argo-simple-project.git
  Path:             simple-demo
Sync Policy:        Manual
Sync Status:        Synced to  (1873ef1)
Health Status:      Healthy

GROUP  KIND        NAMESPACE  NAME          STATUS  HEALTH   HOOK  MESSAGE
apps   Deployment  default    simple-nginx  Synced  Healthy        deployment.apps/simple-nginx created

8.4 Verify the Resources

Check that the pods and deployment were created:

kubectl get pod -n default

NAME                             READY   STATUS    RESTARTS   AGE
simple-deploy-5d7468b6f8-75w68   1/1     Running   0          2m11s
simple-deploy-5d7468b6f8-mkkph   1/1     Running   0          2m11s
simple-deploy-5d7468b6f8-mkgdz   1/1     Running   0          2m11s

kubectl get deployment -n default

NAME            READY   UP-TO-DATE   AVAILABLE   AGE
simple-deploy   3/3     3            3           2m38s

9. Deploying an Application via Web UI

In addition to the CLI, you can create and deploy applications directly from the ArgoCD dashboard.

9.1 Open the Dashboard

After logging in, you'll see the ArgoCD dashboard showing your existing applications (like the simple-deploy app we just created via CLI).

Click the + NEW APP button to create a new application.

9.2 Fill in the General Settings

Under the GENERAL section, configure:

Application Name: nginx-with-ingress
Project Name: default
Sync Policy: Manual (or Automatic if you want auto-sync)

9.3 Configure the Source and Destination

Under the SOURCE section:

Repository URL: https://github.com/maliqpotter/argo-simple-project.git
Revision: HEAD (or the specific branch you want, e.g., main)
Path: nginx-with-ingress

Under the DESTINATION section:

Cluster URL: https://kubernetes.default.svc
Namespace: default

Then click the CREATE button at the top of the form.

9.4 Sync the Application

Once the nginx-with-ingress application appears on the dashboard with an OutOfSync status, click the SYNC button on the application card, then click SYNCHRONIZE to confirm.

After the sync completes, the application status will change to Synced and Healthy. You can click into the application to see a visual map of all the Kubernetes resources that were created (Deployments, ReplicaSets, Pods, etc.).

10. ArgoCD CLI Tips

10.1 Set Default Namespace

Since ArgoCD is installed in the argocd namespace, you can set it as your default to avoid typing -n argocd on every command:

Option A — Set the kubectl context:

kubectl config set-context --current --namespace=argocd

Option B — Use an environment variable:

export ARGOCD_OPTS='--port-forward-namespace argocd'

10.2 Enable Bash Completion

Speed up your CLI workflow with tab completion:

argocd completion bash | sudo tee /etc/bash_completion.d/argocd

source ~/.bashrc

References

This guide is based on a real-world deployment. Adjust IPs, paths, and version numbers to match your own environment.

Kubernetes Single-Node Setup on AlmaLinux 9.8 with Kubeadm, Containerd & Calico

Fri, 19 Jun 2026 00:00:00 GMT

This guide walks you through setting up a fully functional single-node Kubernetes cluster using Kubeadm, with containerd as the container runtime and Tigera Calico as the CNI plugin — all running on AlmaLinux 9.8 (Olive Jaguar).

By the end, you'll have a working Kubernetes node capable of scheduling and running pods, complete with networking provided by Calico.

Server Specifications

Component	Details
OS	AlmaLinux 9.8 (Olive Jaguar)
CPU	4 vCPU
RAM	10 GB
Container Runtime	containerd
CNI Plugin	Tigera Calico
Installer	kubeadm

System Preparation
Installing containerd
Configuring containerd for Kubernetes
Installing kubeadm, kubelet, kubectl
Initializing the Cluster with kubeadm
Configuring kubectl
Installing Tigera Calico
Verifying the Cluster

1. System Preparation

1.1 Set Hostname

hostnamectl set-hostname <your-hostname>

1.2 Disable Swap

Kubernetes requires swap to be disabled.

# Disable swap immediately
swapoff -a

# Disable swap permanently
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

1.3 Disable SELinux

# Disable temporarily
setenforce 0

# Disable permanently
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

1.4 Disable Firewall (or Configure Required Ports)

systemctl disable --now firewalld

Tip: If you prefer to keep the firewall active, open the following ports instead:

6443/tcp — Kubernetes API Server

2379-2380/tcp — etcd server client API

10250/tcp — kubelet API

10251/tcp — kube-scheduler

10252/tcp — kube-controller-manager

179/tcp — Calico BGP

4789/udp — Calico VXLAN (optional)

1.5 Load Required Kernel Modules

# Create module configuration
cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

# Load modules immediately
modprobe overlay
modprobe br_netfilter

1.6 Configure Sysctl Parameters

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply the configuration
sysctl --system

1.7 Verify Kernel Modules Are Loaded

lsmod | grep br_netfilter
lsmod | grep overlay

Expected output (values may vary):

br_netfilter           36864  0
bridge                421888  1 br_netfilter
overlay               237568  0

2. Installing containerd

Reference: Docker Engine on RHEL — docs.docker.com

Since we only need containerd (not the full Docker Engine), we'll use the Docker repository to install just the containerd.io package.

2.1 Remove Potentially Conflicting Packages

dnf remove -y docker \
    docker-client \
    docker-client-latest \
    docker-common \
    docker-latest \
    docker-latest-logrotate \
    docker-logrotate \
    docker-engine \
    podman \
    runc

2.2 Install DNF Plugin and Add Docker Repository

dnf -y install dnf-plugins-core
dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo

2.3 Install containerd.io

Only install the containerd.io package — no need for docker-ce or other Docker components.

dnf install -y containerd.io

2.4 Enable and Start containerd

systemctl enable --now containerd

2.5 Verify containerd

systemctl status containerd
containerd --version

3. Configuring containerd for Kubernetes

3.1 Generate Default Configuration

mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml

3.2 Enable SystemdCgroup

Kubernetes requires containerd to use systemd as the cgroup driver.

sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

Verify the change:

grep 'SystemdCgroup' /etc/containerd/config.toml
# Expected output: SystemdCgroup = true

3.3 Restart containerd

systemctl restart containerd
systemctl status containerd

4. Installing kubeadm, kubelet, kubectl

Reference: Installing kubeadm — kubernetes.io

4.1 Add the Kubernetes Repository

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

Note: Replace v1.36 with your desired Kubernetes version. Check the latest releases at kubernetes.io/releases.

4.2 Install kubelet, kubeadm, kubectl

dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

4.3 Enable kubelet

systemctl enable --now kubelet

Note: kubelet will enter a restart loop until kubeadm init completes — this is expected behavior.

4.4 Verify Versions

kubeadm version
kubectl version --client
kubelet --version

5. Initializing the Cluster with kubeadm

5.1 Initialize the Control Plane

kubeadm init \
  --pod-network-cidr=10.244.0.0/16 \
  --cri-socket unix:///run/containerd/containerd.sock

Important:

The --pod-network-cidr=10.244.0.0/16 flag is used here because some server networks (e.g., 192.168.x.x/24) may overlap with Calico's default CIDR (192.168.0.0/16). Using 10.244.0.0/16 avoids routing conflicts.

If your server has multiple network interfaces, add --apiserver-advertise-address=<your-server-ip> to specify which IP the API server should bind to.

Make sure to save the kubeadm join command from the output — you'll need it if you ever want to add worker nodes.

6. Configuring kubectl

6.1 Set Up kubeconfig for Root User

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

6.2 Verify the Node (NotReady Is Expected)

kubectl get nodes

Expected output:

NAME              STATUS     ROLES           AGE   VERSION
<your-hostname>   NotReady   control-plane   Xs    v1.36.x

The NotReady status is expected because the CNI plugin (Calico) hasn't been installed yet.

7. Installing Tigera Calico

Reference: Calico on single-host Kubernetes — docs.tigera.io

7.1 Install the Tigera Operator

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/tigera-operator.yaml

7.2 Create the Custom Resource for Calico

Since we're using CIDR 10.244.0.0/16 (instead of the default 192.168.0.0/16), download the manifest first, update the CIDR, then apply:

curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/custom-resources.yaml

# Replace Calico's default CIDR with ours
sed -i 's|192.168.0.0/16|10.244.0.0/16|g' custom-resources.yaml

# Verify the change
grep 'cidr' custom-resources.yaml

# Apply the configuration
kubectl create -f custom-resources.yaml

7.3 Wait for Calico to Deploy

watch kubectl get pods -n calico-system

Wait until all pods show Running status:

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-xxx                1/1     Running   0          Xm
calico-node-xxx                            1/1     Running   0          Xm
calico-typha-xxx                           1/1     Running   0          Xm
csi-node-driver-xxx                        2/2     Running   0          Xm

Press Ctrl+C to exit watch.

7.4 Untaint the Control Plane Node

By default, Kubernetes prevents workloads from being scheduled on control plane nodes. For a single-node setup, we need to remove this taint:

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

8. Verifying the Cluster

8.1 Check Node Status

kubectl get nodes -o wide

The node status should now show Ready:

NAME              STATUS   ROLES           AGE   VERSION   INTERNAL-IP     OS-IMAGE                      CONTAINER-RUNTIME
<your-hostname>   Ready    control-plane   Xm    v1.36.x   <your-ip>      AlmaLinux 9.8 (Olive Jaguar)  containerd://2.x.x

8.2 Check All System Pods

kubectl get pods -A

All pods across kube-system, calico-system, and tigera-operator namespaces should show Running status.

8.3 Test with a Simple Deployment

kubectl run nginx --image=nginx --port=80
kubectl get pods

Expected output:

NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          15s

Verify the pod details:

kubectl describe pod nginx

The pod should be scheduled on your node with a Calico-assigned IP from the 10.244.x.x range.

8.4 Check Cluster Information

kubectl cluster-info
kubectl version

Troubleshooting

Node remains NotReady after installing Calico

Inspect the node events and Calico logs:

kubectl describe node <your-hostname>
kubectl logs -n calico-system -l k8s-app=calico-node

Pod stuck in ContainerCreating

Check the pod events and kubelet logs:

kubectl describe pod <pod-name>
journalctl -u kubelet -f

Reset the cluster (start over)

If you need to completely reset and start fresh:

kubeadm reset -f
rm -rf /etc/cni/net.d
rm -rf $HOME/.kube
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm --clear 2>/dev/null || true

References

This guide is based on a real-world deployment. Adjust hostnames, IPs, and version numbers to match your own environment.

Deploying OpenStack with Ceph Tentacle: A 3-Node Converged Setup Guide

Thu, 18 Jun 2026 00:00:00 GMT

In this guide, I'll walk you through deploying a fully functional OpenStack cloud platform integrated with Ceph Tentacle as the distributed storage backend — all on a 3-node converged cluster running Ubuntu Noble 24.04.

This setup uses Kolla-Ansible for deploying OpenStack services as Docker containers, and cephadm for bootstrapping the Ceph cluster. By the end of this guide, you'll have a working private cloud with:

☁️ OpenStack — Keystone, Glance, Nova, Neutron, Cinder, Horizon
💾 Ceph — Distributed storage for images, volumes, VMs, and backups
🌐 Networking — Provider + tenant networks with floating IPs
🖥️ VM Instances — Running CirrOS test instances with SSH access

Topology & Specifications
Node Preparation
Deploy Ceph Tentacle with cephadm
Ceph Pool & Keyring Setup for OpenStack
Install Kolla-Ansible
Kolla-Ansible Configuration
External Ceph Integration with Kolla-Ansible
Deploy OpenStack
OpenStack Network Setup
Create VM Instance
Verify VM Connectivity

1. Topology & Specifications

Network Topology

                    ┌──────────────────┐
                    │   Proxmox Host   │
                    │  vmbr0 (eno1)    │
                    │ 192.168.205.253  │
                    └────────┬─────────┘
           ┌─────────────────┼─────────────────┐
           │                 │                 │
    ens18 (Mgmt)       ens18 (Mgmt)       ens18 (Mgmt)
  192.168.205.101    192.168.205.102    192.168.205.103
    ens19 (Provider)   ens19 (Provider)   ens19 (Provider)
    (no IP)            (no IP)            (no IP)
           │                 │                 │
  ┌─────────────────┐ ┌──────────────┐ ┌──────────────┐
  │  ops-ceph-ctrl  │ │ops-ceph-comp1│ │ops-ceph-comp2│
  │  (controller)   │ │  (compute)   │ │  (compute)   │
  │                 │ │              │ │              │
  │ /dev/sda (OS)   │ │ /dev/sda(OS) │ │ /dev/sda(OS) │
  │ /dev/sdb OSD.0  │ │ /dev/sdb OSD │ │ /dev/sdb OSD │
  │ /dev/sdc OSD.1  │ │ /dev/sdc OSD │ │ /dev/sdc OSD │
  │ /dev/sdd OSD.2  │ │ /dev/sdd OSD │ │ /dev/sdd OSD │
  └─────────────────┘ └──────────────┘ └──────────────┘

Node Specifications

Parameter	ops-ceph-ctrl	ops-ceph-comp1	ops-ceph-comp2
CPU	8 vCPU	6 vCPU	6 vCPU
RAM	16 GB	12 GB	12 GB
Disk OS	50 GB `/dev/sda`	50 GB `/dev/sda`	50 GB `/dev/sda`
Disk OSD	3x 25 GB (`/dev/sdb`,`sdc`,`sdd`)	3x 25 GB	3x 25 GB
NIC 1	`ens18` → 192.168.205.101/24	`ens18` → 192.168.205.102/24	`ens18` → 192.168.205.103/24
NIC 2	`ens19` → no IP	`ens19` → no IP	`ens19` → no IP
OS	Ubuntu Noble 24.04	Ubuntu Noble 24.04	Ubuntu Noble 24.04
Ceph Role	MON, MGR, OSD	OSD	OSD
OpenStack Role	Controller, Network	Compute	Compute

2. Node Preparation

Run the following steps on all nodes unless stated otherwise.

2.1 Set Hostname

# On ops-ceph-ctrl
hostnamectl set-hostname ops-ceph-ctrl

# On ops-ceph-comp1
hostnamectl set-hostname ops-ceph-comp1

# On ops-ceph-comp2
hostnamectl set-hostname ops-ceph-comp2

2.2 Configure /etc/hosts

sudo nano /etc/hosts

Add on all nodes:

192.168.205.101  ops-ceph-ctrl
192.168.205.102  ops-ceph-comp1
192.168.205.103  ops-ceph-comp2

2.3 Configure Netplan

sudo nano /etc/netplan/50-cloud-init.yaml

Example for ops-ceph-ctrl:

network:
  version: 2
  renderer: networkd
  ethernets:
    ens18:
      addresses:
        - 192.168.205.101/24
      routes:
        - to: default
          via: 192.168.205.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]
    ens19:
      dhcp4: false

Adjust IP for comp1 (192.168.205.102) and comp2 (192.168.205.103). ens19 only needs dhcp4: false without an IP — Neutron will use it as the provider interface.

sudo netplan apply

2.4 Install Dependencies

sudo apt update
sudo apt install -y git curl wget python3-dev python3-venv \
  libffi-dev gcc libssl-dev libdbus-glib-1-dev chrony sshpass nano

2.5 Configure Passwordless SSH (from ops-ceph-ctrl)

# Generate SSH key (ed25519)
ssh-keygen -t ed25519 -C "ops-ceph-ctrl"

# Distribute public key to all nodes including self
ssh-copy-id opsce@ops-ceph-ctrl
ssh-copy-id opsce@ops-ceph-comp1
ssh-copy-id opsce@ops-ceph-comp2

2.6 Set NOPASSWD sudo (all nodes)

echo "opsce ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/opsce

2.7 Disable Swap & Time Sync

sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab

sudo timedatectl set-timezone Asia/Jakarta
sudo systemctl enable --now chrony
chronyc tracking

2.8 Sysctl Configuration

sudo nano /etc/sysctl.conf

Add:

net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1

sudo sysctl -p

2.9 Install Docker

curl -sL https://get.docker.com | bash
sudo systemctl enable --now docker

3. Deploy Ceph Tentacle with cephadm

All steps are executed from ops-ceph-ctrl unless stated otherwise.

3.1 Install cephadm

curl --silent --remote-name --location \
  https://download.ceph.com/rpm-tentacle/el9/noarch/cephadm

chmod +x cephadm
sudo install -m 0755 cephadm /usr/local/sbin/cephadm

sudo cephadm add-repo --release tentacle
sudo apt update
sudo apt install -y cephadm

3.2 Bootstrap Ceph Cluster

sudo cephadm bootstrap \
  --mon-ip <your-controller-ip> \
  --cluster-network <your-subnet>/24 \
  --initial-dashboard-user <dashboard-user> \
  --initial-dashboard-password <dashboard-password> \
  --allow-overwrite

Verify:

sudo cephadm shell -- ceph -s

3.3 Install ceph-common

# On ctrl
sudo cephadm install ceph-common

# On comp1 and comp2
ssh -t ops-ceph-comp1 "sudo cephadm add-repo --release tentacle && sudo apt update && sudo apt install -y ceph-common"
ssh -t ops-ceph-comp2 "sudo cephadm add-repo --release tentacle && sudo apt update && sudo apt install -y ceph-common"

3.4 Copy cephadm SSH Key to All Nodes

sudo cephadm shell -- ceph cephadm get-pub-key | \
  sudo tee /etc/ceph/ceph.pub

# Must run as root
sudo su
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ops-ceph-comp1
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ops-ceph-comp2
exit

3.5 Add Nodes to Cluster

sudo cephadm shell -- ceph orch host add ops-ceph-comp1 192.168.205.102
sudo cephadm shell -- ceph orch host add ops-ceph-comp2 192.168.205.103

# Verify
sudo cephadm shell -- ceph orch host ls

3.6 Deploy OSD

# ops-ceph-ctrl
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-ctrl:/dev/sdb
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-ctrl:/dev/sdc
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-ctrl:/dev/sdd

# ops-ceph-comp1
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp1:/dev/sdb
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp1:/dev/sdc
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp1:/dev/sdd

# ops-ceph-comp2
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp2:/dev/sdb
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp2:/dev/sdc
sudo cephadm shell -- ceph orch daemon add osd ops-ceph-comp2:/dev/sdd

Tip: Use hostname:/dev/sdX format — not user@hostname:/dev/sdX. The latter causes a "host not found" error. Also, warnings like Not using image... not in the list of non-dangling images are cosmetic and safe to ignore.

3.7 Verify Cluster

sudo cephadm shell -- ceph -s
sudo cephadm shell -- ceph osd tree
sudo cephadm shell -- ceph df

Make sure all 9 OSDs (3 nodes × 3 disks) are up and in.

Here's what the Ceph Block Images dashboard looks like after a successful deployment:

And the Grafana monitoring dashboard showing cluster health, OSD performance, and resource utilization:

4. Ceph Pool & Keyring Setup for OpenStack

4.1 Create Pools

sudo cephadm shell -- bash -c "
  ceph osd pool create volumes
  ceph osd pool create images
  ceph osd pool create backups
  ceph osd pool create vms

  rbd pool init volumes
  rbd pool init images
  rbd pool init backups
  rbd pool init vms
"

4.2 Create Keyrings

sudo cephadm shell -- bash -c "
  ceph auth get-or-create client.glance \
    mon 'allow r' \
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=images' \
    -o /etc/ceph/ceph.client.glance.keyring

  ceph auth get-or-create client.cinder \
    mon 'allow r' \
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=images' \
    -o /etc/ceph/ceph.client.cinder.keyring

  ceph auth get-or-create client.nova \
    mon 'allow r' \
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' \
    -o /etc/ceph/ceph.client.nova.keyring

  ceph auth get-or-create client.cinder-backup \
    mon 'allow r' \
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups' \
    -o /etc/ceph/ceph.client.cinder-backup.keyring
"

4.3 Export Keyrings to Host

sudo cephadm shell -- ceph auth get client.glance | sudo tee /etc/ceph/ceph.client.glance.keyring
sudo cephadm shell -- ceph auth get client.cinder | sudo tee /etc/ceph/ceph.client.cinder.keyring
sudo cephadm shell -- ceph auth get client.nova | sudo tee /etc/ceph/ceph.client.nova.keyring
sudo cephadm shell -- ceph auth get client.cinder-backup | sudo tee /etc/ceph/ceph.client.cinder-backup.keyring

sudo chmod 640 /etc/ceph/ceph.client.*.keyring

4.4 Generate ceph.conf

sudo cephadm shell -- ceph config generate-minimal-conf | sudo tee /etc/ceph/ceph.conf

# Remove leading tabs to prevent kolla-ansible issues
sudo sed -i 's/^\t//' /etc/ceph/ceph.conf

cat /etc/ceph/ceph.conf

5. Install Kolla-Ansible

Run from ops-ceph-ctrl as user opsce.

5.1 Create Virtual Environment

python3 -m venv ~/kolla-venv
source ~/kolla-venv/bin/activate

# Auto-activate on login
echo 'source ~/kolla-venv/bin/activate' >> ~/.bashrc

5.2 Install Kolla-Ansible

pip install -U pip
pip install 'ansible-core>=2.16,<2.17'
pip install docker dbus-python

# Install kolla-ansible from master
pip install git+https://opendev.org/openstack/kolla-ansible@master

5.3 Setup Kolla Directories

sudo mkdir -p /etc/kolla
sudo chown $USER:$USER /etc/kolla

cp -r ~/kolla-venv/share/kolla-ansible/etc_examples/kolla/* /etc/kolla/
cp ~/kolla-venv/share/kolla-ansible/ansible/inventory/multinode ~/multinode

5.4 Install Ansible Galaxy Dependencies

kolla-ansible install-deps

6. Kolla-Ansible Configuration

6.1 Ansible Config

sudo mkdir -p /etc/ansible
sudo tee /etc/ansible/ansible.cfg <<EOF
[defaults]
host_key_checking = False
pipelining = True
forks = 100
interpreter_python = /usr/bin/python3
EOF

6.2 Multinode Inventory

nano ~/multinode

Update the top section:

[control]
ops-ceph-ctrl

[network]
ops-ceph-ctrl

[compute]
ops-ceph-comp1
ops-ceph-comp2

[monitoring]
ops-ceph-ctrl

[storage]
ops-ceph-ctrl
ops-ceph-comp1
ops-ceph-comp2

[deployment]
localhost       ansible_connection=local

Leave all [xxx:children] groups as default from the template.

Verify:

ansible -i ~/multinode all -m ping

6.3 Generate Passwords

kolla-genpwd

# Set admin password (replace with your own secure password)
sed -i 's/^keystone_admin_password:.*/keystone_admin_password: <your-password>/' /etc/kolla/passwords.yml

6.4 Configure globals.yml

nano /etc/kolla/globals.yml

# =====================
# BASE IMAGE
# =====================
kolla_base_distro: "ubuntu"

# =====================
# NETWORKING
# =====================
network_interface: "ens18"
neutron_external_interface: "ens19"
kolla_internal_vip_address: "192.168.205.100"

# =====================
# ENABLE SERVICES
# =====================
enable_openstack_core: "yes"
enable_cinder: "yes"
enable_cinder_backup: "yes"

# =====================
# CEPH INTEGRATION
# =====================
glance_backend_ceph: "yes"
cinder_backend_ceph: "yes"
nova_backend_ceph: "yes"

ceph_glance_user: "glance"
ceph_glance_pool_name: "images"
ceph_cinder_user: "cinder"
ceph_cinder_pool_name: "volumes"
ceph_cinder_backup_user: "cinder-backup"
ceph_cinder_backup_pool_name: "backups"
ceph_nova_user: "nova"
ceph_nova_pool_name: "vms"

kolla_internal_vip_address must be an unused IP in the ens18 subnet. This IP will automatically appear on the ens18 interface of ops-ceph-ctrl after deployment as the HAProxy VIP.

7. External Ceph Integration with Kolla-Ansible

7.1 Create Directories

mkdir -p /etc/kolla/config/glance
mkdir -p /etc/kolla/config/nova
mkdir -p /etc/kolla/config/cinder/cinder-volume
mkdir -p /etc/kolla/config/cinder/cinder-backup

7.2 Copy ceph.conf

cp /etc/ceph/ceph.conf /etc/kolla/config/glance/ceph.conf
cp /etc/ceph/ceph.conf /etc/kolla/config/nova/ceph.conf
cp /etc/ceph/ceph.conf /etc/kolla/config/cinder/ceph.conf

7.3 Copy Keyrings

# Glance
cp /etc/ceph/ceph.client.glance.keyring \
   /etc/kolla/config/glance/ceph.client.glance.keyring

# Nova (needs both nova AND cinder keyrings)
sudo cp /etc/ceph/ceph.client.nova.keyring \
   /etc/kolla/config/nova/ceph.client.nova.keyring
sudo cp /etc/ceph/ceph.client.cinder.keyring \
   /etc/kolla/config/nova/ceph.client.cinder.keyring

# Cinder Volume
sudo cp /etc/ceph/ceph.client.cinder.keyring \
   /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring

# Cinder Backup
sudo cp /etc/ceph/ceph.client.cinder.keyring \
   /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
sudo cp /etc/ceph/ceph.client.cinder-backup.keyring \
   /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring

7.4 Fix Ownership & Permissions

sudo chown -R opsce:opsce /etc/kolla/config
sudo chmod 640 /etc/kolla/config/cinder/cinder-backup/*.keyring
sudo chmod 640 /etc/kolla/config/cinder/cinder-volume/*.keyring
sudo chmod 640 /etc/kolla/config/nova/*.keyring
sudo chmod 640 /etc/kolla/config/glance/*.keyring

7.5 Verify File Structure

find /etc/kolla/config -type f -printf "%u %p\n" | sort

Expected output (all owned by opsce):

opsce /etc/kolla/config/cinder/ceph.conf
opsce /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring
opsce /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring
opsce /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring
opsce /etc/kolla/config/glance/ceph.conf
opsce /etc/kolla/config/glance/ceph.client.glance.keyring
opsce /etc/kolla/config/nova/ceph.conf
opsce /etc/kolla/config/nova/ceph.client.cinder.keyring
opsce /etc/kolla/config/nova/ceph.client.nova.keyring

8. Deploy OpenStack

8.1 Bootstrap Servers

kolla-ansible bootstrap-servers -i ~/multinode

Heads up: It's common for Ceph MON/MGR services to go inactive dead after this step, since kolla-ansible reconfigures Docker daemon settings. If this happens, a simple sudo reboot on affected nodes is the cleanest fix. After reboot, verify Ceph health with sudo ceph -s before proceeding.

8.2 Pre-checks

kolla-ansible prechecks -i ~/multinode

8.3 Deploy

kolla-ansible deploy -i ~/multinode

Expect this to take 20-60 minutes depending on your hardware and network speed. Occasional single-task failures on compute nodes are typically timing-related and generally resolve themselves — verify with openstack compute service list after deployment.

8.4 Post-Deploy

kolla-ansible post-deploy -i ~/multinode

8.5 Install OpenStack Client & Load Credentials

pip install python-openstackclient \
  -c https://releases.openstack.org/constraints/upper/2025.2

source /etc/kolla/admin-openrc.sh

8.6 Verify Services

openstack service list
openstack compute service list
openstack network agent list
openstack volume service list

All services should be up. Note that cinder-volume backend shows rbd-1, confirming successful Ceph integration.

Here's the actual output from a working deployment — all services, network agents, compute services, and volume services are confirmed up:

9. OpenStack Network Setup

9.1 Create Provider Network (External)

openstack network create \
  --share \
  --external \
  --provider-physical-network physnet1 \
  --provider-network-type flat \
  public-net

openstack subnet create \
  --network public-net \
  --subnet-range 17.19.21.0/24 \
  --gateway 17.19.21.254 \
  --allocation-pool start=17.19.21.100,end=17.19.21.200 \
  --dns-nameserver 8.8.8.8 \
  public-subnet

9.2 Create Private/Tenant Network

openstack network create private-net

openstack subnet create \
  --network private-net \
  --subnet-range 10.0.0.0/24 \
  --gateway 10.0.0.1 \
  --dns-nameserver 8.8.8.8 \
  private-subnet

9.3 Create Router

openstack router create main-router
openstack router set main-router --external-gateway public-net
openstack router add subnet main-router private-subnet

# Verify
openstack router show main-router
openstack port list --router main-router

Once the router is created, you can visualize the network topology in Horizon:

9.4 Create Security Group

openstack security group create security-group-net \
  --description 'Allow SSH and ICMP'

openstack security group rule create \
  --protocol icmp security-group-net

openstack security group rule create \
  --protocol tcp --ingress --dst-port 22 security-group-net

openstack security group rule list security-group-net

9.5 Create Floating IP

openstack floating ip create public-net
openstack floating ip list

Note: Floating IPs are only externally reachable if the provider network interface is bridged to your physical network. In isolated lab environments, you can still access VMs via their private IPs using the Neutron router namespace (see Section 11).

10. Create VM Instance

10.1 Upload Image

wget http://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img

openstack image create \
  --disk-format qcow2 \
  --container-format bare \
  --public \
  --file ./cirros-0.6.2-x86_64-disk.img \
  cirros-0.6.2

openstack image list

10.2 Create Keypair

openstack keypair create \
  --public-key ~/.ssh/id_ed25519.pub \
  controller-key

openstack keypair list

10.3 Create Flavor

openstack flavor create \
  --ram 512 \
  --disk 8 \
  --vcpus 1 \
  --public \
  c1-small

openstack flavor list

10.4 Create Instance

# First instance
openstack server create \
  --flavor c1-small \
  --image cirros-0.6.2 \
  --key-name controller-key \
  --security-group security-group-net \
  --network private-net \
  cirros-instance-test

# Second instance (for inter-VM connectivity test)
openstack server create \
  --flavor c1-small \
  --image cirros-0.6.2 \
  --key-name controller-key \
  --security-group security-group-net \
  --network private-net \
  cirros-instance-test-2

# Check status
openstack server list

Wait until status is ACTIVE.

10.5 Assign Floating IP

# List available floating IPs
openstack floating ip list

# Assign to each instance
openstack server add floating ip cirros-instance-test <floating-ip-1>
openstack server add floating ip cirros-instance-test-2 <floating-ip-2>

# Verify — instances should have 2 IPs (private + floating)
openstack server list

Here's what it looks like in Horizon — both instances are Active and Running with their private and floating IPs assigned:

And the detailed server list from the CLI, confirming Ceph-backed storage and network connectivity:

11. Verify VM Connectivity

11.1 Check Router Namespace

# List all namespaces
sudo ip netns list
# Output: qrouter-xxx (id: N), qdhcp-xxx (id: N)

11.2 Ping VM from Namespace

# Replace <router-id> with the ID from ip netns list output
sudo ip netns exec qrouter-<router-id> ping -c 4 10.0.0.71
sudo ip netns exec qrouter-<router-id> ping -c 4 10.0.0.104

11.3 SSH to VM from Namespace

# SSH using key (recommended)
sudo ip netns exec qrouter-<router-id> ssh -i ~/.ssh/id_ed25519 cirros@10.0.0.71

# Or SSH with password (cirros password: gocubsgo)
sudo ip netns exec qrouter-<router-id> ssh cirros@10.0.0.71

11.4 Test Inter-VM Connectivity

From inside cirros-instance-test, ping the other VM:

# Enter the first VM
sudo ip netns exec qrouter-<router-id> ssh cirros@10.0.0.71

# From inside the VM, ping the second VM
ping 10.0.0.104

Note: You cannot ping VMs directly from the controller host — there's no route to the tenant or provider networks from the host's default namespace. Always use sudo ip netns exec qrouter-<id> to access VM networks from the controller.

11.5 Verify OVS Bridge

sudo docker exec openvswitch_vswitchd ovs-vsctl show

Verify:

br-ex has port ens19
br-int has qr-xxx (router port) and tap-xxx (VM port)
All bridge controllers show is_connected: true

Troubleshooting

MON/MGR dies after bootstrap-servers

kolla-ansible reconfigures Docker settings which can prevent Ceph containers from restarting. A sudo reboot is the cleanest fix. Alternatively, manually start the services:

sudo systemctl start ceph-<fsid>@mon.<hostname>.service
sudo systemctl start ceph-<fsid>@mgr.<hostname>.<id>.service

Error "host not found" when adding OSD

Use hostname:/dev/sdX format — not user@hostname:/dev/sdX:

# CORRECT
sudo cephadm shell -- ceph orch daemon add osd <hostname>:/dev/sdb
# WRONG
sudo cephadm shell -- ceph orch daemon add osd user@<hostname>:/dev/sdb

Warning "Not using image... non-dangling images"

Cosmetic warning, not an error. Safe to ignore.

kolla-ansible: command not found

Ensure the Python virtual environment is activated:

source ~/kolla-venv/bin/activate
which kolla-ansible

Missing sudo password during bootstrap-servers

Ensure passwordless sudo is configured for your deploy user:

echo "<user> ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/<user>

ceph.conf with leading tabs

Kolla-ansible may fail to parse ceph.conf if it contains tabs. Strip them:

sudo sed -i 's/^\t//' /etc/kolla/config/*/ceph.conf

Cannot ping VM directly from controller

This is expected — the host has no route to tenant/provider networks. Use the Neutron router namespace:

sudo ip netns list
sudo ip netns exec qrouter-<id> ping -c 4 <vm-ip>

References

This guide is based on a real-world deployment. Adjust IPs, hostnames, credentials, and disk paths to match your own environment.

I'm Gatan

Getting Started with Istio: Installing the Service Mesh & Running the Bookinfo Demo on Kubernetes

Configuration Placeholders

Environment Specifications

Prerequisites

Step 1: Download and Install istioctl

1.1 Download the Istio Release

1.2 Add istioctl to PATH

1.3 Verify istioctl

Step 2: Install Istio with the Default Profile

2.1 Pre-installation Check (optional but recommended)

2.2 Install the Default Profile

Step 3: Verify the Installation

3.1 Check Pods in istio-system

3.2 Check Control Plane Version

3.3 Analyze Configuration (optional)

Step 4: Enable Automatic Sidecar Injection

Step 5: Deploy the Bookinfo Application

5.1 Apply the Bookinfo Manifest

5.2 Verify Services and Pods

5.3 Test from Inside the Cluster

5.4 Create the Istio Gateway

Step 6: Expose Bookinfo on a Single-Node Cluster

Option A — Port Forward (fastest for demos)

Option B — Patch to NodePort (more permanent)

Step 7: Confirm the App is Working

Step 8: Define Service Versions with DestinationRules

Step 9: Kiali Dashboard — Visualizing Your Service Mesh

9.1 Install the Observability Addons

9.2 Wait for Kiali to Be Ready

9.3 Access the Kiali Dashboard

Step 10: Test Traffic Routing with VirtualServices

Cleanup

Troubleshooting

References

HashiCorp Vault + MariaDB: Eliminating Static Credentials with Dynamic Secrets

Prerequisites

Configuration Placeholders

Step 1: Install Vault (Dev Mode)

macOS (Homebrew)

Linux (Debian/Ubuntu)

Verify Installation

Start the Dev Server

Alternative: Run Vault Without Dev Mode (via config.hcl)

1. Create config.hcl

2. Start Vault with the Config File

3. Initialize Vault

4. Unseal Vault

5. Log In with the Root Token

Step 2: Set Up MariaDB (Docker Compose)

Step 3: Configure Vault Database Secrets Engine

3.1 Enable the Database Secrets Engine

3.2 Configure the Database Connection

3.3 Create an Admin Role

3.4 Create a Read-Only Role

Step 4: Generate Dynamic Credentials

Step 5: Verify the Credentials

Step 6 (Optional): Lease Management & Manual Revocation

List Active Leases

Revoke a Specific Lease Early

Revoke All Leases for a Role

Why This Matters: Security Benefits

What's Next

Conclusion

Zero Trust Kubernetes: Securing Cluster Access with Keycloak and Pinniped SSO

Background & Motivation

Key Outcomes

Prerequisites

Configuration Placeholders

Understanding Pinniped

How Pinniped Kubeconfig Works

Architecture Overview

CNCF Projects Involved

Part 1: Keycloak Configuration

1.1 Create the OIDC Client

1.2 Add the Group Mapper

1.3 Create Groups and Assign Users

1.4 Test User Setup Checklist

1.5 Optional: Test Token Generation via Direct Access Grants

1.6 Configure 8-Hour Session Lifespan

Alternative: Run Vault Without Dev Mode (via `config.hcl`)

1. Create `config.hcl`

Allowed Access (`dev` namespace)

Forbidden Access (`prod` namespace for `dev-team` member)