---
title: "Use Kubernetes service accounts without stored keys"
description: "Let pods act as a bot with their service account token, with no API key stored in the cluster."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Use Kubernetes service accounts without stored keys

A pod can exchange its Kubernetes service account token for a short-lived [API key](/iam/api-keys) that acts as a [bot](/iam/bots). Nothing secret is stored in the cluster: Kubernetes gives the pod a token for its service account, and the bot's **OIDC trust** decides which service accounts may act as it. The cluster's configuration doesn't change.

You set this up once, signed in:

1. Register your cluster as an **OIDC provider** of your organization.
2. Add an **OIDC trust** entry to a bot for the pod's service account.
3. In the pod, mount a service account token for EZGH Cloud, exchange it for an API key, and use the key as usual.

| Limit | Value |
| --- | --- |
| OIDC providers per organization | 50 |
| Public keys uploaded per provider | 1 to 10, at most 64 KiB |
| OIDC trust entries per bot | 20 |
| Keys from OIDC tokens a bot holds at once | 500 |
| Key lifetime | 15 minutes to 12 hours; 1 hour by default |

## How it works

1. Kubernetes mounts a **projected service account token** into the pod, for the audience `ezghcloud.com`. The token's `iss` is the cluster's issuer, and its `sub` is `system:serviceaccount:<namespace>:<service account>`. Kubernetes replaces the token in the file before it expires.
2. The pod runs `ezgh auth oidc-exchange` with the token. EZGH Cloud checks the token's signature against the cluster's public keys, its audience and expiry, and the bot's OIDC trust.
3. The pod gets an API key that acts as the bot until it expires. To keep working, it exchanges the current token again for a new key.

The token is good only for the audience it was issued for: the cluster's API server doesn't accept a token for `ezghcloud.com`, so EZGH Cloud never receives a token that works in your cluster.

## Find your cluster's issuer

Every cluster names an issuer in its tokens. Read it from the cluster:

```sh
kubectl get --raw /.well-known/openid-configuration
```

Its `issuer` is the value to register, exactly as printed, including any trailing `/`. How EZGH Cloud gets the cluster's public keys depends on whether the issuer is public:

| Cluster | Issuer | Public keys |
| --- | --- | --- |
| Amazon EKS | `https://oidc.eks.<region>.amazonaws.com/id/<ID>` | Read from the issuer |
| Google GKE | `https://container.googleapis.com/v1/projects/<project>/locations/<location>/clusters/<cluster>` | Read from the issuer |
| Azure AKS, with its OIDC issuer enabled | Shown by `az aks show --query oidcIssuerProfile.issuerUrl` | Read from the issuer |
| k3s, RKE2, kubeadm and other self-managed clusters | `https://kubernetes.default.svc.cluster.local` by default | Uploaded by you |

An issuer whose keys EZGH Cloud reads must be an `https` URL that's reachable from the internet. EZGH Cloud doesn't connect to private addresses, or to host names such as `*.local`, `*.internal`, `*.svc` or `*.cluster.local`. For a cluster whose issuer isn't public, upload its public keys instead.

## Register the cluster as an OIDC provider

An OIDC provider is identified by its **issuer**, compared exactly with the `iss` of the tokens it issues, and accepts tokens for one of its **audiences** (1 to 10). Whether EZGH Cloud reads the provider's keys from the issuer or you upload them is set when you create it. Neither the issuer nor that choice can change.

Managing providers needs `oidcProviders.create`, `oidcProviders.list`, `oidcProviders.get`, `oidcProviders.update` and `oidcProviders.delete`. Providers are also on the console's **OIDC providers** page in IAM.

### Clusters with a public issuer

EZGH Cloud reads the issuer's discovery document (`<issuer>/.well-known/openid-configuration`) when you register it, and refuses an issuer it can't read. On every exchange, it reads the issuer's current keys, so the cluster's key changes need nothing from you.

### CLI

```sh
ezgh iam oidc-providers create prod-eks --issuer https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE
```

The audience defaults to `ezghcloud.com`. Use `--audiences` to accept others.
### API

```sh
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "prod-eks", "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE", "audiences": ["ezghcloud.com"]}'
```

### Any other cluster: upload its public keys

Upload the cluster's public keys, which it serves at `/openid/v1/jwks`. EZGH Cloud verifies tokens against the keys you uploaded and never connects to the issuer, so the issuer doesn't need to be reachable, and the default `https://kubernetes.default.svc.cluster.local` works.

### CLI

```sh
kubectl get --raw /openid/v1/jwks > cluster-jwks.json
ezgh iam oidc-providers create home-k3s --issuer https://kubernetes.default.svc.cluster.local --jwks-file cluster-jwks.json
```

`--jwks-file -` reads the keys from standard input. The audience defaults to `ezghcloud.com`. Use `--audiences` to accept others. `ezgh iam oidc-providers get home-k3s` shows the provider's key IDs.
### API

Send the keys as `jwks`:

```sh
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "home-k3s", "issuer": "https://kubernetes.default.svc.cluster.local", "audiences": ["ezghcloud.com"], "jwks": '"$(kubectl get --raw /openid/v1/jwks)"'}'
```

The keys are a JSON Web Key Set, `{ "keys": [...] }`, as the cluster serves it:

- 1 to 10 keys with distinct `kid`s, at most 64 KiB in all.
- Each is a public signing key: RSA of at least 2,048 bits (`RS256`), or EC on the `P-256` curve (`ES256`). `use`, when present, is `sig`.
- Upload only public keys. A key with private parts is refused, and isn't kept or recorded. EZGH Cloud keeps each key's `kty`, `kid`, `use`, `alg`, `n`, `e`, `crv`, `x` and `y` and drops other members, such as `x5c`.
- A key set that doesn't fit is refused with `400 invalid_request`, naming the problem and where it is, such as `jwks.keys[1].n`.

**Several clusters with the same issuer.** Self-managed clusters usually share the default issuer. Register each cluster as its own provider with its own keys: on an exchange, EZGH Cloud uses the provider that holds the key that signed the token (the `kid` in the token's header). Two providers with the same issuer can't hold a key with the same `kid` (`409 conflict`). An issuer has either one provider whose keys are read from the issuer, or any number with uploaded keys, never both (`409 conflict`).

Deleting a provider deletes every bot's trust entries for it, and the keys they issued stop working at once.

## When the cluster's signing key changes

A provider with uploaded keys accepts only tokens signed by those keys. When the cluster's service account signing key changes, update the provider's keys, or new tokens are refused. While pods may still hold tokens signed by the old key, upload the old and new keys together: the cluster's `/openid/v1/jwks` lists every key it currently accepts. Once the old key is retired, upload again to remove it.

### CLI

```sh
kubectl get --raw /openid/v1/jwks > cluster-jwks.json
ezgh iam oidc-providers update home-k3s --jwks-file cluster-jwks.json
```
### API

`jwks` replaces the provider's keys:

```sh
curl -X PATCH https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/oidc-providers/$PROVIDER_ID \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"jwks": '"$(kubectl get --raw /openid/v1/jwks)"'}'
```

The provider's `ETag` covers its keys: send it as `If-Match` to update only if nobody changed the provider since you read it.

Removing a key refuses new exchanges for tokens it signed. Keys already issued keep working until they expire. A provider whose keys are read from the issuer has no keys to update (`400 invalid_request`).

## Add OIDC trust to a bot

Create a [bot](/iam/bots) with the policies the workload needs, then let the pod's service account act as it. An OIDC trust entry names a provider and a **subject** pattern, matched against the token's `sub`: for a service account, `system:serviceaccount:<namespace>:<service account>`. In the subject, `*` matches any run of characters; everything else matches itself, and the whole `sub` must match.

Seeing a bot's OIDC trust needs `bots.get`; changing it needs `bots.updateTrust` on the bot. Whoever can change it can let a workload act as the bot, so treat it like `bots.createApiKey`.

### CLI

```sh
ezgh iam bots create deployer
ezgh iam policies attach ProjectsFullAccess --bot deployer

# The deployer service account in the jobs namespace.
ezgh iam bots oidc-trust add deployer --provider home-k3s --subject system:serviceaccount:jobs:deployer

# Any service account in the jobs namespace.
ezgh iam bots oidc-trust add deployer --provider home-k3s --subject 'system:serviceaccount:jobs:*'

ezgh iam bots oidc-trust list deployer
ezgh iam bots oidc-trust remove deployer <entry-id>
```

Name the provider by its name, ID or slug: several providers can have the same issuer.
### API

```sh
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/bots/$BOT_ID/oidc-trust \
  -H "Authorization: Bearer $EZGH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"providerId": "'"$PROVIDER_ID"'", "subject": "system:serviceaccount:jobs:deployer"}'
```

`GET` on the same URL lists the entries, and `DELETE …/oidc-trust/<entry-id>` removes one.

The subject is 1 to 256 characters, with at least one character before its first `*`. A trust entry for one provider doesn't match tokens from another, so a service account with the same name in another cluster can't act as the bot unless you add an entry for that cluster's provider. Removing an entry ends the keys it issued at once.

## Use it in a pod

Mount a projected service account token with the audience `ezghcloud.com`, and give its file to `ezgh auth oidc-exchange` with `--token-file`. The command needs no other credential and prints the key alone on standard output. Give the organization and the bot by ID or slug.

This CronJob runs as the `deployer` service account. Its init container installs `ezgh` into a shared volume; an image with `ezgh` already in it doesn't need one.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: deployer
  namespace: jobs
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly
  namespace: jobs
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: deployer
          restartPolicy: OnFailure
          initContainers:
            - name: install-ezgh
              image: alpine:3.20
              command:
                - sh
                - -c
                - apk add --no-cache curl && curl -fsSL https://get.ezghcloud.com | EZGH_INSTALL_DIR=/opt/ezgh sh
              volumeMounts:
                - name: ezgh-bin
                  mountPath: /opt/ezgh
          containers:
            - name: nightly
              image: alpine:3.20
              env:
                - name: EZGH_ORG
                  value: org_k3f9a0x2m7qp
              command:
                - sh
                - -c
                - |
                  set -e
                  export PATH="/opt/ezgh:$PATH"
                  EZGH_API_KEY=$(ezgh auth oidc-exchange --org "$EZGH_ORG" --bot bot_8k2l4m6n8p0q --token-file /var/run/secrets/ezgh/token)
                  export EZGH_API_KEY
                  ezgh projects list
              volumeMounts:
                - name: ezgh-token
                  mountPath: /var/run/secrets/ezgh
                  readOnly: true
                - name: ezgh-bin
                  mountPath: /opt/ezgh
          volumes:
            - name: ezgh-token
              projected:
                sources:
                  - serviceAccountToken:
                      path: token
                      audience: ezghcloud.com
                      expirationSeconds: 3600
            - name: ezgh-bin
              emptyDir: {}
```

- `audience` must be one of the provider's audiences. Don't use the pod's default token in `/var/run/secrets/kubernetes.io/serviceaccount`: its audience is the cluster's API server, so it's refused, and it shouldn't leave the cluster.
- `expirationSeconds` is how long the service account token lasts (Kubernetes' minimum is 600). The API key's lifetime is separate: use `--expires-in` (`15m` to `12h`) for work that runs longer than an hour.
- A long-running workload exchanges again before its key expires, reading the token file each time: Kubernetes keeps a current token there.
- To pin the `ezgh` version, set `EZGH_VERSION` for the installer, such as `EZGH_VERSION=0.19.0`. `--jwks-file` needs 0.19.0 or later.

The key is an ordinary API key: send it as `Authorization: Bearer` to any EZGH Cloud API. Without the CLI, send the token to the exchange; it takes no other credential:

```sh
curl https://orgs.ezghcloud.com/v1/organizations/$ORG_ID/bots/$BOT_ID/oidc-api-keys \
  -H "Content-Type: application/json" \
  -d '{"token": "'"$(cat /var/run/secrets/ezgh/token)"'", "expiresInSeconds": 3600}'
```

It answers `201` with `{ "apiKey": {…}, "token": "ezgh_…" }`. `expiresInSeconds` is 900 to 43,200; `name` (1 to 100 characters) defaults to `OIDC: <sub>`.

## Keys from OIDC tokens

- A key acts as the bot, with exactly the bot's access, and expires when its lifetime ends.
- It stops working at once when its trust entry or OIDC provider is deleted, the bot is deleted, or the key is revoked.
- It's listed with the bot's [API keys](/iam/api-keys); its `sourceIdentity` names the OIDC provider and the token's `sub`. Revoke it like any key.
- In [Trails](/trails), the exchange is `CreateOidcApiKey`, and everything done with the key is the bot's, with the service account (`system:serviceaccount:…`) as the source identity.

## Troubleshooting

Every refused exchange gets the same `401 unauthenticated`, "The OIDC token wasn't accepted", so it doesn't reveal which organizations, bots or providers exist. Refused exchanges aren't recorded in Trails. Check, in order:

1. The organization and bot are right: IDs or slugs, not names.
2. The token's `iss` is exactly the issuer of one of your OIDC providers (`ezgh iam oidc-providers list`), including any trailing `/`.
3. The token's `aud` is one of that provider's audiences. The projected volume's `audience` sets it; the pod's default token is always refused.
4. For a provider with uploaded keys: the key that signed the token (the `kid` in its header) is one of the provider's key IDs (`ezgh iam oidc-providers get <provider>`). After the cluster's signing key changes, upload its current keys (`kubectl get --raw /openid/v1/jwks`). A token without a `kid` is refused.
5. The token hasn't expired. Read the token file at the time of the exchange rather than copying the token elsewhere.
6. One of the bot's entries (`ezgh iam bots oidc-trust list <bot>`) is for that provider and matches the token's `sub`: the pod's namespace and `serviceAccountName`, as `system:serviceaccount:<namespace>:<service account>`.

`ezgh auth oidc-exchange` prints the token's `iss`, `aud`, `sub`, expiry and `kid` when the exchange is refused, so you can compare them with the provider and the bot's entries.

- `429` means too many exchanges; retry after the `Retry-After` seconds. Exchanges are limited per client address, so pods that leave the cluster through one address share a limit: exchange once per key, not per request.
- `503` means EZGH Cloud couldn't read the keys of a provider whose keys are read from the issuer; retry shortly. A provider with uploaded keys never answers `503`.
- `409 limit_exceeded` means the bot already holds 500 keys from OIDC tokens; they expire on their own, or revoke some.

Source: https://docs.ezghcloud.com/iam/kubernetes/index.mdx
