This guide details a battle-tested setup for implementing Zero Trust Kubernetes authentication in production — including common runtime pitfalls, configuration edge cases, and step-by-step troubleshooting solutions.
While many engineering teams run workloads on Kubernetes in production, cluster access is frequently managed using legacy authentication anti-patterns:
Shared, static kubeconfig files distributed via Slack, email, or internal repositories.
Static ServiceAccount tokens that are never rotated.
Over-privileged cluster admin roles with zero individual user audit logging.
In this tutorial, we will set up Keycloak as a centralized Identity Provider (IdP) and Pinniped as the authentication bridge between Keycloak and your Kubernetes cluster.
Ingress Controller: Traefik installed as an Ingress Controller on your cluster. For installation details, see Installing ArgoCD & Traefik HTTPS Ingress — you only need to complete the Install Traefik via Helm section.
DNS Domain: A domain pointing to your cluster Ingress IP for exposing the Pinniped Supervisor (e.g., pinniped.example.com).
Pinniped is a CNCF Sandbox project designed specifically to make Kubernetes authentication secure, seamless, and infrastructure-agnostic.
Without Pinniped, every Kubernetes cluster must be individually configured with API server OIDC flags (--oidc-issuer-url, --oidc-client-id, etc.) — requiring control plane restarts and tricky multi-tenant maintenance. Pinniped solves this by acting as an intermediary authentication broker.
Supervisor: Installed once (either on a dedicated management cluster or your main cluster). It acts as an OIDC identity broker, authenticating users against Keycloak and issuing cluster-scoped identity tokens.
Concierge: Installed on every cluster you want to secure. It validates incoming tokens issued by the Supervisor and impersonates the identity when sending requests to kube-apiserver.
Traditional kubeconfig files contain embedded client certificates or static tokens. Pinniped changes this paradigm entirely:
Credential-less configuration: The generated kubeconfig file contains no user credentials, certificates, or tokens.
Safe distribution: The exact same kubeconfig file can be safely stored in public or team internal Git repositories.
Browser-based flow: When a user executes kubectl, the Pinniped CLI plugin opens a local browser window to authenticate with Keycloak. Upon successful login, short-lived credentials are cached locally.
Log in to your Keycloak Admin Console and select your realm (<REALM_NAME>).
Navigate to Clients → click Create client.
Configure the client parameters:
Field
Value
Client type
OpenID Connect
Client ID
kubernetes
Client authentication
Off (Public client)
Standard flow
On
Valid redirect URIs
http://localhost:8000, http://localhost:18000
Note: In Part 3, once the Pinniped Supervisor endpoint is configured, you must return here and add https://<PINNIPED_SUPERVISOR_URL>/* to Valid redirect URIs. Failing to do so will result in an Invalid parameter: redirect_uri error during browser authentication.
To enforce Kubernetes RBAC based on Keycloak user groups, the JWT token issued by Keycloak must contain a groups claim.
Under the kubernetes client, open the Client scopes tab.
Select the dedicated scope (e.g., kubernetes-dedicated).
Click Add mapper → By configuration → Group Membership.
Set the following fields:
Field
Value
Name
groups
Token Claim Name
groups
Full group path
Off
Add to ID token
On
Add to access token
On
Add to userinfo
On
Important: groups here is a Token Claim Name, not an OIDC scope name. If you attempt a manual OAuth request with -d "scope=openid groups", Keycloak will reject it with invalid_scope. Always request scope=openid — the groups claim will automatically be included in the token payload once this mapper is enabled.
You can verify the supported scopes on your realm endpoint:
When creating test users in Keycloak, ensure all required user details are configured. Incomplete profiles will cause authentication failures:
Requirement
Consequence if omitted
First Name & Last Name set
Error: Account is not fully set up
Email verified = On (Details tab)
Error: email_verified claim in upstream ID token has false value
Password Temporary = Off (Credentials tab)
Error: Account is not fully set up
Required user actions cleared
Error: Account is not fully set up
Tip: If Keycloak returns Account is not fully set up despite meeting all criteria above, reset the user password manually under Credentials → Set password.
1.5 Optional: Test Token Generation via Direct Access Grants#
Direct Access Grants (password grant type) are disabled by default in modern Keycloak versions for security reasons. To temporarily test token issuance via curl:
Open Client kubernetes → Settings → Capability config → enable Direct access grants → Save.
In Keycloak Admin Console, open Realm Settings → Sessions tab:
Field
Lifespan
SSO Session Idle
8 hours
SSO Session Max
8 hours
Client Session Idle
8 hours
Client Session Max
8 hours
Next, select the Tokens tab:
Field
Lifespan
Access Token Lifespan
5 minutes
Refresh Token Max Reuse
0
When Keycloak’s refresh token expires after 8 hours, Pinniped automatically prompts the user to re-authenticate via their browser on the next kubectl command execution.
Pinniped Supervisor requires a trusted TLS certificate for its endpoint. By default, Pinniped generates a self-signed bootstrap certificate that external CLI clients will reject with:
tls: failed to verify certificate: x509: certificate is valid for pinniped-supervisor-bootstrap-cert.
We install cert-manager to automatically obtain and renew Let’s Encrypt TLS certificates as Kubernetes secrets.
3.2 Expose Supervisor via Traefik (TLS Passthrough)#
Because Pinniped Supervisor directly handles sensitive credential exchanges, it requires TLS Passthrough to perform end-to-end TLS termination directly inside the pod rather than at the Ingress controller layer.
Create a ClusterIP Service for the Supervisor:
pinniped-supervisor-service.yaml
1
apiVersion: v1
2
kind: Service
3
metadata:
4
name: pinniped-supervisor
5
namespace: pinniped-supervisor
6
spec:
7
type: ClusterIP
8
selector:
9
app: pinniped-supervisor
10
ports:
11
- protocol: TCP
12
port: 443
13
targetPort: 8443
Create a Traefik IngressRouteTCP with passthrough: true:
Note: Do not add groups to additionalScopes. Requesting groups as an OIDC scope will trigger an invalid_scope error from Keycloak.
Now, create the corresponding client secret. Pinniped requires a custom secret type secrets.pinniped.dev/oidc-client:
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
name: keycloak-client-secret
5
namespace: pinniped-supervisor
6
type: secrets.pinniped.dev/oidc-client
7
stringData:
8
clientID: kubernetes
9
clientSecret: ""
Caution: Do not use kubectl create secret generic. Generic secrets have type Opaque, which will cause Pinniped to throw the error referenced Secret has wrong type "Opaque". Always apply via YAML with type: secrets.pinniped.dev/oidc-client.
Path Troubleshooting: The generated kubeconfig invokes /usr/local/bin/pinniped. On Apple Silicon Macs, Homebrew installs binaries under /opt/homebrew/bin/pinniped. Create a symbolic link to prevent exec: fork/exec /usr/local/bin/pinniped: no such file or directory:
Forbidden Access (prod namespace for dev-team member)#
Terminal window
1
$kubectlgetpod-nprod
2
Errorfromserver (Forbidden): pods is forbidden: User "keycloak:dev-user@example.com" cannot list resource "pods" in API group "" in the namespace "prod"
Group Mapper is Critical: Without adding the groups mapper under Client Scopes, authentication completes but RBAC bindings continuously fail.
Treat groups as Claims, Not Scopes: Passing groups as an OIDC scope parameter breaks Keycloak protocol validation.
Pinniped Supervisor Requires Valid Public TLS: Never expose Supervisor using self-signed bootstrap certs in production environments.
Stateless Kubeconfig Security: Since Pinniped kubeconfig files contain no secrets, credential leaks via repository commits are completely mitigated. Access revocation is performed instantly in Keycloak.