A pod can exchange its Kubernetes service account token for a short-lived API key that acts as a bot. Nothing secret is stored in the cluster: Kubernetes gives the pod a token for its service account, and the bot’s OIDC trust decides which service accounts may act as it. The cluster’s configuration doesn’t change.
You set this up once, signed in:
- Register your cluster as an OIDC provider of your organization.
- Add an OIDC trust entry to a bot for the pod’s service account.
- In the pod, mount a service account token for EZGH Cloud, exchange it for an API key, and use the key as usual.
| Limit | Value |
|---|---|
| OIDC providers per organization | 50 |
| Public keys uploaded per provider | 1 to 10, at most 64 KiB |
| OIDC trust entries per bot | 20 |
| Keys from OIDC tokens a bot holds at once | 500 |
| Key lifetime | 15 minutes to 12 hours; 1 hour by default |
How it works
- Kubernetes mounts a projected service account token into the pod, for the audience
ezghcloud.com. The token’sissis the cluster’s issuer, and itssubissystem:serviceaccount:<namespace>:<service account>. Kubernetes replaces the token in the file before it expires. - The pod runs
ezgh auth oidc-exchangewith the token. EZGH Cloud checks the token’s signature against the cluster’s public keys, its audience and expiry, and the bot’s OIDC trust. - The pod gets an API key that acts as the bot until it expires. To keep working, it exchanges the current token again for a new key.
The token is good only for the audience it was issued for: the cluster’s API server doesn’t accept a token for ezghcloud.com, so EZGH Cloud never receives a token that works in your cluster.
Find your cluster’s issuer
Every cluster names an issuer in its tokens. Read it from the cluster:
kubectl get --raw /.well-known/openid-configurationIts issuer is the value to register, exactly as printed, including any trailing /. How EZGH Cloud gets the cluster’s public keys depends on whether the issuer is public:
| Cluster | Issuer | Public keys |
|---|---|---|
| Amazon EKS | https://oidc.eks.<region>.amazonaws.com/id/<ID> |
Read from the issuer |
| Google GKE | https://container.googleapis.com/v1/projects/<project>/locations/<location>/clusters/<cluster> |
Read from the issuer |
| Azure AKS, with its OIDC issuer enabled | Shown by az aks show --query oidcIssuerProfile.issuerUrl |
Read from the issuer |
| k3s, RKE2, kubeadm and other self-managed clusters | https://kubernetes.default.svc.cluster.local by default |
Uploaded by you |
An issuer whose keys EZGH Cloud reads must be an https URL that’s reachable from the internet. EZGH Cloud doesn’t connect to private addresses, or to host names such as *.local, *.internal, *.svc or *.cluster.local. For a cluster whose issuer isn’t public, upload its public keys instead.
Register the cluster as an OIDC provider
An OIDC provider is identified by its issuer, compared exactly with the iss of the tokens it issues, and accepts tokens for one of its audiences (1 to 10). Whether EZGH Cloud reads the provider’s keys from the issuer or you upload them is set when you create it. Neither the issuer nor that choice can change.
Managing providers needs oidcProviders.create, oidcProviders.list, oidcProviders.get, oidcProviders.update and oidcProviders.delete. Providers are also on the console’s OIDC providers page in IAM.
Clusters with a public issuer
EZGH Cloud reads the issuer’s discovery document (<issuer>/.well-known/openid-configuration) when you register it, and refuses an issuer it can’t read. On every exchange, it reads the issuer’s current keys, so the cluster’s key changes need nothing from you.
ezgh iam oidc-providers create prod-eks --issuer https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLEThe audience defaults to ezghcloud.com. Use --audiences to accept others.
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers \
-H "Authorization: Bearer $EZGH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "prod-eks", "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE", "audiences": ["ezghcloud.com"]}'Any other cluster: upload its public keys
Upload the cluster’s public keys, which it serves at /openid/v1/jwks. EZGH Cloud verifies tokens against the keys you uploaded and never connects to the issuer, so the issuer doesn’t need to be reachable, and the default https://kubernetes.default.svc.cluster.local works.
kubectl get --raw /openid/v1/jwks > cluster-jwks.json
ezgh iam oidc-providers create home-k3s --issuer https://kubernetes.default.svc.cluster.local --jwks-file cluster-jwks.json--jwks-file - reads the keys from standard input. The audience defaults to ezghcloud.com. Use --audiences to accept others. ezgh iam oidc-providers get home-k3s shows the provider’s key IDs.
Send the keys as jwks:
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers \
-H "Authorization: Bearer $EZGH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"name": "home-k3s", "issuer": "https://kubernetes.default.svc.cluster.local", "audiences": ["ezghcloud.com"], "jwks": '"$(kubectl get --raw /openid/v1/jwks)"'}'The keys are a JSON Web Key Set, { "keys": [...] }, as the cluster serves it:
- 1 to 10 keys with distinct
kids, at most 64 KiB in all. - Each is a public signing key: RSA of at least 2,048 bits (
RS256), or EC on theP-256curve (ES256).use, when present, issig. - Upload only public keys. A key with private parts is refused, and isn’t kept or recorded. EZGH Cloud keeps each key’s
kty,kid,use,alg,n,e,crv,xandyand drops other members, such asx5c. - A key set that doesn’t fit is refused with
400 invalid_request, naming the problem and where it is, such asjwks.keys[1].n.
Several clusters with the same issuer. Self-managed clusters usually share the default issuer. Register each cluster as its own provider with its own keys: on an exchange, EZGH Cloud uses the provider that holds the key that signed the token (the kid in the token’s header). Two providers with the same issuer can’t hold a key with the same kid (409 conflict). An issuer has either one provider whose keys are read from the issuer, or any number with uploaded keys, never both (409 conflict).
Deleting a provider deletes every bot’s trust entries for it, and the keys they issued stop working at once.
When the cluster’s signing key changes
A provider with uploaded keys accepts only tokens signed by those keys. When the cluster’s service account signing key changes, update the provider’s keys, or new tokens are refused. While pods may still hold tokens signed by the old key, upload the old and new keys together: the cluster’s /openid/v1/jwks lists every key it currently accepts. Once the old key is retired, upload again to remove it.
kubectl get --raw /openid/v1/jwks > cluster-jwks.json
ezgh iam oidc-providers update home-k3s --jwks-file cluster-jwks.jsonjwks replaces the provider’s keys:
curl -X PATCH https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers/$PROVIDER_ID \
-H "Authorization: Bearer $EZGH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"jwks": '"$(kubectl get --raw /openid/v1/jwks)"'}'The provider’s ETag covers its keys: send it as If-Match to update only if nobody changed the provider since you read it.
Removing a key refuses new exchanges for tokens it signed. Keys already issued keep working until they expire. A provider whose keys are read from the issuer has no keys to update (400 invalid_request).
Add OIDC trust to a bot
Create a bot with the policies the workload needs, then let the pod’s service account act as it. An OIDC trust entry names a provider and a subject pattern, matched against the token’s sub: for a service account, system:serviceaccount:<namespace>:<service account>. In the subject, * matches any run of characters; everything else matches itself, and the whole sub must match.
Seeing a bot’s OIDC trust needs bots.get; changing it needs bots.updateTrust on the bot. Whoever can change it can let a workload act as the bot, so treat it like bots.createApiKey.
ezgh iam bots create deployer
ezgh iam policies attach ProjectsFullAccess --bot deployer
# The deployer service account in the jobs namespace.
ezgh iam bots oidc-trust add deployer --provider home-k3s --subject system:serviceaccount:jobs:deployer
# Any service account in the jobs namespace.
ezgh iam bots oidc-trust add deployer --provider home-k3s --subject 'system:serviceaccount:jobs:*'
ezgh iam bots oidc-trust list deployer
ezgh iam bots oidc-trust remove deployer <entry-id>Name the provider by its name, ID or slug: several providers can have the same issuer.
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/bots/$BOT_ID/oidc-trust \
-H "Authorization: Bearer $EZGH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"providerId": "'"$PROVIDER_ID"'", "subject": "system:serviceaccount:jobs:deployer"}'GET on the same URL lists the entries, and DELETE …/oidc-trust/<entry-id> removes one.
The subject is 1 to 256 characters, with at least one character before its first *. A trust entry for one provider doesn’t match tokens from another, so a service account with the same name in another cluster can’t act as the bot unless you add an entry for that cluster’s provider. Removing an entry ends the keys it issued at once.
Use it in a pod
Mount a projected service account token with the audience ezghcloud.com, and give its file to ezgh auth oidc-exchange with --token-file. The command needs no other credential and prints the key alone on standard output. Give the organization and the bot by ID or slug.
This CronJob runs as the deployer service account. Its init container installs ezgh into a shared volume; an image with ezgh already in it doesn’t need one.
apiVersion: v1
kind: ServiceAccount
metadata:
name: deployer
namespace: jobs
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly
namespace: jobs
spec:
schedule: "0 3 * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: deployer
restartPolicy: OnFailure
initContainers:
- name: install-ezgh
image: alpine:3.20
command:
- sh
- -c
- apk add --no-cache curl && curl -fsSL https://get.ezghcloud.com | EZGH_INSTALL_DIR=/opt/ezgh sh
volumeMounts:
- name: ezgh-bin
mountPath: /opt/ezgh
containers:
- name: nightly
image: alpine:3.20
env:
- name: EZGH_ORG
value: org_k3f9a0x2m7qp
command:
- sh
- -c
- |
set -e
export PATH="/opt/ezgh:$PATH"
EZGH_API_KEY=$(ezgh auth oidc-exchange --org "$EZGH_ORG" --bot bot_8k2l4m6n8p0q --token-file /var/run/secrets/ezgh/token)
export EZGH_API_KEY
ezgh projects list
volumeMounts:
- name: ezgh-token
mountPath: /var/run/secrets/ezgh
readOnly: true
- name: ezgh-bin
mountPath: /opt/ezgh
volumes:
- name: ezgh-token
projected:
sources:
- serviceAccountToken:
path: token
audience: ezghcloud.com
expirationSeconds: 3600
- name: ezgh-bin
emptyDir: {}audiencemust be one of the provider’s audiences. Don’t use the pod’s default token in/var/run/secrets/kubernetes.io/serviceaccount: its audience is the cluster’s API server, so it’s refused, and it shouldn’t leave the cluster.expirationSecondsis how long the service account token lasts (Kubernetes’ minimum is 600). The API key’s lifetime is separate: use--expires-in(15mto12h) for work that runs longer than an hour.- A long-running workload exchanges again before its key expires, reading the token file each time: Kubernetes keeps a current token there.
- To pin the
ezghversion, setEZGH_VERSIONfor the installer, such asEZGH_VERSION=0.19.0.--jwks-fileneeds 0.19.0 or later.
The key is an ordinary API key: send it as Authorization: Bearer to any EZGH Cloud API. Without the CLI, send the token to the exchange; it takes no other credential:
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/bots/$BOT_ID/oidc-api-keys \
-H "Content-Type: application/json" \
-d '{"token": "'"$(cat /var/run/secrets/ezgh/token)"'", "expiresInSeconds": 3600}'It answers 201 with { "apiKey": {…}, "token": "ezgh_…" }. expiresInSeconds is 900 to 43,200; name (1 to 100 characters) defaults to OIDC: <sub>.
Keys from OIDC tokens
- A key acts as the bot, with exactly the bot’s access, and expires when its lifetime ends.
- It stops working at once when its trust entry or OIDC provider is deleted, the bot is deleted, or the key is revoked.
- It’s listed with the bot’s API keys; its
sourceIdentitynames the OIDC provider and the token’ssub. Revoke it like any key. - In Trails, the exchange is
CreateOidcApiKey, and everything done with the key is the bot’s, with the service account (system:serviceaccount:…) as the source identity.
Troubleshooting
Every refused exchange gets the same 401 unauthenticated, “The OIDC token wasn’t accepted”, so it doesn’t reveal which organizations, bots or providers exist. Refused exchanges aren’t recorded in Trails. Check, in order:
- The organization and bot are right: IDs or slugs, not names.
- The token’s
issis exactly the issuer of one of your OIDC providers (ezgh iam oidc-providers list), including any trailing/. - The token’s
audis one of that provider’s audiences. The projected volume’saudiencesets it; the pod’s default token is always refused. - For a provider with uploaded keys: the key that signed the token (the
kidin its header) is one of the provider’s key IDs (ezgh iam oidc-providers get <provider>). After the cluster’s signing key changes, upload its current keys (kubectl get --raw /openid/v1/jwks). A token without akidis refused. - The token hasn’t expired. Read the token file at the time of the exchange rather than copying the token elsewhere.
- One of the bot’s entries (
ezgh iam bots oidc-trust list <bot>) is for that provider and matches the token’ssub: the pod’s namespace andserviceAccountName, assystem:serviceaccount:<namespace>:<service account>.
ezgh auth oidc-exchange prints the token’s iss, aud, sub, expiry and kid when the exchange is refused, so you can compare them with the provider and the bot’s entries.
429means too many exchanges; retry after theRetry-Afterseconds. Exchanges are limited per client address, so pods that leave the cluster through one address share a limit: exchange once per key, not per request.503means EZGH Cloud couldn’t read the keys of a provider whose keys are read from the issuer; retry shortly. A provider with uploaded keys never answers503.409 limit_exceededmeans the bot already holds 500 keys from OIDC tokens; they expire on their own, or revoke some.