Skip to content

Use Kubernetes service accounts without stored keys

Let pods act as a bot with their service account token, with no API key stored in the cluster.

Updated View as Markdown

A pod can exchange its Kubernetes service account token for a short-lived API key that acts as a bot. Nothing secret is stored in the cluster: Kubernetes gives the pod a token for its service account, and the bot’s OIDC trust decides which service accounts may act as it. The cluster’s configuration doesn’t change.

You set this up once, signed in:

  1. Register your cluster as an OIDC provider of your organization.
  2. Add an OIDC trust entry to a bot for the pod’s service account.
  3. In the pod, mount a service account token for EZGH Cloud, exchange it for an API key, and use the key as usual.
Limit Value
OIDC providers per organization 50
Public keys uploaded per provider 1 to 10, at most 64 KiB
OIDC trust entries per bot 20
Keys from OIDC tokens a bot holds at once 500
Key lifetime 15 minutes to 12 hours; 1 hour by default

How it works

  1. Kubernetes mounts a projected service account token into the pod, for the audience ezghcloud.com. The token’s iss is the cluster’s issuer, and its sub is system:serviceaccount:<namespace>:<service account>. Kubernetes replaces the token in the file before it expires.
  2. The pod runs ezgh auth oidc-exchange with the token. EZGH Cloud checks the token’s signature against the cluster’s public keys, its audience and expiry, and the bot’s OIDC trust.
  3. The pod gets an API key that acts as the bot until it expires. To keep working, it exchanges the current token again for a new key.

The token is good only for the audience it was issued for: the cluster’s API server doesn’t accept a token for ezghcloud.com, so EZGH Cloud never receives a token that works in your cluster.

Find your cluster’s issuer

Every cluster names an issuer in its tokens. Read it from the cluster:

kubectl get --raw /.well-known/openid-configuration

Its issuer is the value to register, exactly as printed, including any trailing /. How EZGH Cloud gets the cluster’s public keys depends on whether the issuer is public:

Cluster Issuer Public keys
Amazon EKS https://oidc.eks.<region>.amazonaws.com/id/<ID> Read from the issuer
Google GKE https://container.googleapis.com/v1/projects/<project>/locations/<location>/clusters/<cluster> Read from the issuer
Azure AKS, with its OIDC issuer enabled Shown by az aks show --query oidcIssuerProfile.issuerUrl Read from the issuer
k3s, RKE2, kubeadm and other self-managed clusters https://kubernetes.default.svc.cluster.local by default Uploaded by you

An issuer whose keys EZGH Cloud reads must be an https URL that’s reachable from the internet. EZGH Cloud doesn’t connect to private addresses, or to host names such as *.local, *.internal, *.svc or *.cluster.local. For a cluster whose issuer isn’t public, upload its public keys instead.

Register the cluster as an OIDC provider

An OIDC provider is identified by its issuer, compared exactly with the iss of the tokens it issues, and accepts tokens for one of its audiences (1 to 10). Whether EZGH Cloud reads the provider’s keys from the issuer or you upload them is set when you create it. Neither the issuer nor that choice can change.

Managing providers needs oidcProviders.create, oidcProviders.list, oidcProviders.get, oidcProviders.update and oidcProviders.delete. Providers are also on the console’s OIDC providers page in IAM.

Clusters with a public issuer

EZGH Cloud reads the issuer’s discovery document (<issuer>/.well-known/openid-configuration) when you register it, and refuses an issuer it can’t read. On every exchange, it reads the issuer’s current keys, so the cluster’s key changes need nothing from you.

ezgh iam oidc-providers create prod-eks --issuer https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE

The audience defaults to ezghcloud.com. Use --audiences to accept others.

curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "prod-eks", "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE", "audiences": ["ezghcloud.com"]}'

Any other cluster: upload its public keys

Upload the cluster’s public keys, which it serves at /openid/v1/jwks. EZGH Cloud verifies tokens against the keys you uploaded and never connects to the issuer, so the issuer doesn’t need to be reachable, and the default https://kubernetes.default.svc.cluster.local works.

kubectl get --raw /openid/v1/jwks > cluster-jwks.json
ezgh iam oidc-providers create home-k3s --issuer https://kubernetes.default.svc.cluster.local --jwks-file cluster-jwks.json

--jwks-file - reads the keys from standard input. The audience defaults to ezghcloud.com. Use --audiences to accept others. ezgh iam oidc-providers get home-k3s shows the provider’s key IDs.

Send the keys as jwks:

curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "home-k3s", "issuer": "https://kubernetes.default.svc.cluster.local", "audiences": ["ezghcloud.com"], "jwks": '"$(kubectl get --raw /openid/v1/jwks)"'}'

The keys are a JSON Web Key Set, { "keys": [...] }, as the cluster serves it:

  • 1 to 10 keys with distinct kids, at most 64 KiB in all.
  • Each is a public signing key: RSA of at least 2,048 bits (RS256), or EC on the P-256 curve (ES256). use, when present, is sig.
  • Upload only public keys. A key with private parts is refused, and isn’t kept or recorded. EZGH Cloud keeps each key’s kty, kid, use, alg, n, e, crv, x and y and drops other members, such as x5c.
  • A key set that doesn’t fit is refused with 400 invalid_request, naming the problem and where it is, such as jwks.keys[1].n.

Several clusters with the same issuer. Self-managed clusters usually share the default issuer. Register each cluster as its own provider with its own keys: on an exchange, EZGH Cloud uses the provider that holds the key that signed the token (the kid in the token’s header). Two providers with the same issuer can’t hold a key with the same kid (409 conflict). An issuer has either one provider whose keys are read from the issuer, or any number with uploaded keys, never both (409 conflict).

Deleting a provider deletes every bot’s trust entries for it, and the keys they issued stop working at once.

When the cluster’s signing key changes

A provider with uploaded keys accepts only tokens signed by those keys. When the cluster’s service account signing key changes, update the provider’s keys, or new tokens are refused. While pods may still hold tokens signed by the old key, upload the old and new keys together: the cluster’s /openid/v1/jwks lists every key it currently accepts. Once the old key is retired, upload again to remove it.

kubectl get --raw /openid/v1/jwks > cluster-jwks.json
ezgh iam oidc-providers update home-k3s --jwks-file cluster-jwks.json

jwks replaces the provider’s keys:

curl -X PATCH https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers/$PROVIDER_ID \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jwks": '"$(kubectl get --raw /openid/v1/jwks)"'}'

The provider’s ETag covers its keys: send it as If-Match to update only if nobody changed the provider since you read it.

Removing a key refuses new exchanges for tokens it signed. Keys already issued keep working until they expire. A provider whose keys are read from the issuer has no keys to update (400 invalid_request).

Add OIDC trust to a bot

Create a bot with the policies the workload needs, then let the pod’s service account act as it. An OIDC trust entry names a provider and a subject pattern, matched against the token’s sub: for a service account, system:serviceaccount:<namespace>:<service account>. In the subject, * matches any run of characters; everything else matches itself, and the whole sub must match.

Seeing a bot’s OIDC trust needs bots.get; changing it needs bots.updateTrust on the bot. Whoever can change it can let a workload act as the bot, so treat it like bots.createApiKey.

ezgh iam bots create deployer
ezgh iam policies attach ProjectsFullAccess --bot deployer

# The deployer service account in the jobs namespace.
ezgh iam bots oidc-trust add deployer --provider home-k3s --subject system:serviceaccount:jobs:deployer

# Any service account in the jobs namespace.
ezgh iam bots oidc-trust add deployer --provider home-k3s --subject 'system:serviceaccount:jobs:*'

ezgh iam bots oidc-trust list deployer
ezgh iam bots oidc-trust remove deployer <entry-id>

Name the provider by its name, ID or slug: several providers can have the same issuer.

curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/bots/$BOT_ID/oidc-trust \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"providerId": "'"$PROVIDER_ID"'", "subject": "system:serviceaccount:jobs:deployer"}'

GET on the same URL lists the entries, and DELETE …/oidc-trust/<entry-id> removes one.

The subject is 1 to 256 characters, with at least one character before its first *. A trust entry for one provider doesn’t match tokens from another, so a service account with the same name in another cluster can’t act as the bot unless you add an entry for that cluster’s provider. Removing an entry ends the keys it issued at once.

Use it in a pod

Mount a projected service account token with the audience ezghcloud.com, and give its file to ezgh auth oidc-exchange with --token-file. The command needs no other credential and prints the key alone on standard output. Give the organization and the bot by ID or slug.

This CronJob runs as the deployer service account. Its init container installs ezgh into a shared volume; an image with ezgh already in it doesn’t need one.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: jobs
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly
  namespace: jobs
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: deployer
          restartPolicy: OnFailure
          initContainers:
            - name: install-ezgh
              image: alpine:3.20
              command:
                - sh
                - -c
                - apk add --no-cache curl && curl -fsSL https://get.ezghcloud.com | EZGH_INSTALL_DIR=/opt/ezgh sh
              volumeMounts:
                - name: ezgh-bin
                  mountPath: /opt/ezgh
          containers:
            - name: nightly
              image: alpine:3.20
              env:
                - name: EZGH_ORG
                  value: org_k3f9a0x2m7qp
              command:
                - sh
                - -c
                - |
                  set -e
                  export PATH="/opt/ezgh:$PATH"
                  EZGH_API_KEY=$(ezgh auth oidc-exchange --org "$EZGH_ORG" --bot bot_8k2l4m6n8p0q --token-file /var/run/secrets/ezgh/token)
                  export EZGH_API_KEY
                  ezgh projects list
              volumeMounts:
                - name: ezgh-token
                  mountPath: /var/run/secrets/ezgh
                  readOnly: true
                - name: ezgh-bin
                  mountPath: /opt/ezgh
          volumes:
            - name: ezgh-token
              projected:
                sources:
                  - serviceAccountToken:
                      path: token
                      audience: ezghcloud.com
                      expirationSeconds: 3600
            - name: ezgh-bin
              emptyDir: {}
  • audience must be one of the provider’s audiences. Don’t use the pod’s default token in /var/run/secrets/kubernetes.io/serviceaccount: its audience is the cluster’s API server, so it’s refused, and it shouldn’t leave the cluster.
  • expirationSeconds is how long the service account token lasts (Kubernetes’ minimum is 600). The API key’s lifetime is separate: use --expires-in (15m to 12h) for work that runs longer than an hour.
  • A long-running workload exchanges again before its key expires, reading the token file each time: Kubernetes keeps a current token there.
  • To pin the ezgh version, set EZGH_VERSION for the installer, such as EZGH_VERSION=0.19.0. --jwks-file needs 0.19.0 or later.

The key is an ordinary API key: send it as Authorization: Bearer to any EZGH Cloud API. Without the CLI, send the token to the exchange; it takes no other credential:

curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/bots/$BOT_ID/oidc-api-keys \
  -H "Content-Type: application/json" \
  -d '{"token": "'"$(cat /var/run/secrets/ezgh/token)"'", "expiresInSeconds": 3600}'

It answers 201 with { "apiKey": {…}, "token": "ezgh_…" }. expiresInSeconds is 900 to 43,200; name (1 to 100 characters) defaults to OIDC: <sub>.

Keys from OIDC tokens

  • A key acts as the bot, with exactly the bot’s access, and expires when its lifetime ends.
  • It stops working at once when its trust entry or OIDC provider is deleted, the bot is deleted, or the key is revoked.
  • It’s listed with the bot’s API keys; its sourceIdentity names the OIDC provider and the token’s sub. Revoke it like any key.
  • In Trails, the exchange is CreateOidcApiKey, and everything done with the key is the bot’s, with the service account (system:serviceaccount:…) as the source identity.

Troubleshooting

Every refused exchange gets the same 401 unauthenticated, “The OIDC token wasn’t accepted”, so it doesn’t reveal which organizations, bots or providers exist. Refused exchanges aren’t recorded in Trails. Check, in order:

  1. The organization and bot are right: IDs or slugs, not names.
  2. The token’s iss is exactly the issuer of one of your OIDC providers (ezgh iam oidc-providers list), including any trailing /.
  3. The token’s aud is one of that provider’s audiences. The projected volume’s audience sets it; the pod’s default token is always refused.
  4. For a provider with uploaded keys: the key that signed the token (the kid in its header) is one of the provider’s key IDs (ezgh iam oidc-providers get <provider>). After the cluster’s signing key changes, upload its current keys (kubectl get --raw /openid/v1/jwks). A token without a kid is refused.
  5. The token hasn’t expired. Read the token file at the time of the exchange rather than copying the token elsewhere.
  6. One of the bot’s entries (ezgh iam bots oidc-trust list <bot>) is for that provider and matches the token’s sub: the pod’s namespace and serviceAccountName, as system:serviceaccount:<namespace>:<service account>.

ezgh auth oidc-exchange prints the token’s iss, aud, sub, expiry and kid when the exchange is refused, so you can compare them with the provider and the bot’s entries.

  • 429 means too many exchanges; retry after the Retry-After seconds. Exchanges are limited per client address, so pods that leave the cluster through one address share a limit: exchange once per key, not per request.
  • 503 means EZGH Cloud couldn’t read the keys of a provider whose keys are read from the issuer; retry shortly. A provider with uploaded keys never answers 503.
  • 409 limit_exceeded means the bot already holds 500 keys from OIDC tokens; they expire on their own, or revoke some.
Navigation

Type to search…

↑↓ navigate↵ selectEsc close