---
title: "Kubernetes"
description: "Managed Kubernetes clusters in your VPC, with node groups of your own instances and every EZGH integration built in."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Kubernetes

EZGH Kubernetes runs a Kubernetes cluster's control plane for you. Your nodes are ordinary
[instances](/compute/instances) in your project, launched by node groups. Pods get addresses in your
VPC, `LoadBalancer` Services and Ingresses become EZGH [load balancers](/load-balancers),
PersistentVolumeClaims become [volumes](/compute/volumes) or [Files](/files) access points, pods act
as [bots](/iam/bots) with no stored key, and `kubectl` signs in with EZGH IAM. You install nothing in
the cluster for any of this.

- [kubectl access](/kubernetes/kubectl-access) — Sign in with EZGH IAM, access entries and the public endpoint.
- [Node groups](/kubernetes/node-groups) — Instance types, sizes, node bots, scaling and draining.
- [Load balancers](/kubernetes/load-balancers) — LoadBalancer Services, Ingress and the Gateway API.
- [Storage](/kubernetes/storage) — Block Storage volumes and Files volumes for PersistentVolumeClaims.
- [Pod identity](/kubernetes/pod-identity) — Pods that call EZGH APIs as a bot, with no stored key.
- [Secrets](/kubernetes/secrets) — Mount Secrets fields into pods.
- [Observability](/kubernetes/observability) — Control-plane logs and metrics, and node and workload telemetry in Obs.
- [Versions and upgrades](/kubernetes/upgrades) — Supported versions, patches, minor upgrades and end of support.

The Kubernetes API is at `https://kubernetes-api.ezghcloud.com`. In the console, open **Kubernetes**.

## Before you start

A cluster needs:

- A [VPC](/private-networks/vpcs) with an attached [Network Gateway](/private-networks/network-gateways).
- A private [subnet](/private-networks/vpcs) of it whose [route table](/private-networks/route-tables)
  sends `0.0.0.0/0` to that gateway. The cluster's endpoint takes 3 addresses in this subnet.
- A **cluster bot**: the [bot](/iam/bots) the cluster acts as when it creates load balancers, volumes
  and nodes in your project. Attach the managed policy `KubernetesClusterAccess` to it. Creating the
  cluster needs `bots.pass` on that bot.

What the cluster bot does shows in [Trails](/trails) as that bot.

## Create a cluster

### CLI

1. Create the cluster. It takes about 15 minutes, and `--wait` waits until it's `active`:

```sh
ezgh kubernetes clusters create prod --subnet sbn_2m9x0a7qpe4k --bot bot_8k2l4m6n8p0q --wait
```

2. Add a node group. Its bot needs the managed policy `KubernetesNodeAccess`, and you need
`bots.pass` on it ([Node groups](/kubernetes/node-groups)):

```sh
ezgh kubernetes node-groups create general --cluster kcl_3k9d0a1m2x7q \
  --instance-type gp-1.large --subnet sbn_2m9x0a7qpe4k --bot bot_5t7y9u1i3oqa \
  --min 2 --max 6 --wait
```

3. Point `kubectl` at the cluster, then list its nodes:

```sh
ezgh kubernetes update-kubeconfig kcl_3k9d0a1m2x7q
kubectl get nodes
```
### Console

1. Open **Kubernetes**, then **Create cluster**. Pick the version, the VPC and subnet, and the cluster
bot. **Create a bot** makes one with `KubernetesClusterAccess` attached.
2. On the cluster's **Node groups** tab, choose **Add node group**. **Create a bot for nodes** makes the
node bot with `KubernetesNodeAccess` attached and lets the cluster's bot use it.
3. Copy the `ezgh kubernetes update-kubeconfig` command from the cluster's **Overview**, run it, then
run `kubectl get nodes`.

In the API, [CreateCluster](/kubernetes-api/CreateCluster/) answers `202` with an operation. The
cluster exists from then on, `creating`, with its ID. Its `certificateAuthority` is `null` until it's
`active`. Clusters, node groups and add-ons are created, changed and deleted through operations; access
entries and pod identity associations change at once.

Whoever creates the cluster gets cluster admin access through an [access
entry](/kubernetes/kubectl-access#access-entries), unless they turn that off
(`bootstrapCreatorAccess: false`, `--no-creator-access`).

## Add-ons

Each cluster runs managed add-ons. EZGH installs, configures and upgrades them, and they upgrade with
the cluster unless you pin a version.

| Add-on | What it is | Default |
| --- | --- | --- |
| `vpc-cni` | Pod networking with VPC addresses, and NetworkPolicy | Installed; can't be removed |
| `kube-proxy` | Services | Installed; can't be removed |
| `coredns` | Cluster DNS | Installed; can't be removed |
| `pod-identity-agent` | [Pod identity](/kubernetes/pod-identity) | Installed; removable |
| `blockstore-csi` | [Block Storage volumes](/kubernetes/storage) | Installed; removable |
| `files-csi` | [Files volumes](/kubernetes/storage#files-volumes) | Optional |
| `secrets-store` | [Secrets](/kubernetes/secrets) mounted into pods | Optional |
| `obs-agent` | [Node and workload telemetry](/kubernetes/observability) | Optional |
| `metrics-server` | Resource metrics for `kubectl top` and autoscalers | Optional |

`GET /public/v1/addon-versions?clusterVersion=1.37` lists every add-on's versions for a cluster
version, which add-ons are installed by default and which can be removed (`ezgh kubernetes
addon-versions list --cluster-version 1.37`).

An add-on's `configuration` is a JSON object, checked against the schema of the add-on version it's
for; a configuration that doesn't fit is refused. EZGH manages the objects an add-on installs: a field
EZGH sets is overwritten if it's changed in the cluster.

```sh
ezgh kubernetes addons create metrics-server --cluster kcl_3k9d0a1m2x7q --wait
ezgh kubernetes addons update coredns --cluster kcl_3k9d0a1m2x7q --configuration-file coredns.json --wait
```

`--version` pins an add-on to a version; `--follow-cluster` unpins it, so it's upgraded with the cluster.

## Quotas and limits

| Quota | Default |
| --- | --- |
| Clusters per organization | 10 |
| Node groups per cluster | 10 |
| Nodes per cluster (the sum of its node groups' maximum sizes) | 300 |

These are [Service quotas](/quotas). Each node group's [autoscaling group](/compute/autoscaling) also
counts against Compute's autoscaling groups quota, and its nodes against Compute's instance and vCPU
quotas.

| Limit | |
| --- | --- |
| Nodes per node group | 100 |
| Pods per node | 110 |
| PersistentVolumes per node | The instance type's `limits.volumes`, less one for the root volume |

## Delete a cluster

Delete a cluster's node groups first: a cluster with node groups can't be deleted
(`409 cluster_has_node_groups`). Deleting the cluster then:

- deletes the load balancers it created for Services, Ingresses and Gateways, and its node firewall;
- keeps the volumes it created for PersistentVolumeClaims, tagged with the cluster. Delete the
  PersistentVolumeClaims first to delete their volumes with them.

```sh
ezgh kubernetes node-groups delete kng_7h3n2c5d8wqa --cluster kcl_3k9d0a1m2x7q --yes --wait
ezgh kubernetes clusters delete kcl_3k9d0a1m2x7q --yes --wait
```

Source: https://docs.ezghcloud.com/kubernetes/index.mdx
