---
title: "Autoscaling groups"
description: "Keep a number of identical instances running, add and remove them with load, replace unhealthy ones, and roll out a new launch spec in batches."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Autoscaling groups

An autoscaling group launches instances from one launch spec and keeps their number between a
minimum and a maximum size. It adds instances when a policy says the group is under load, removes
them when load stays low, replaces instances that stop working, and rolls a new launch spec out
across the group in batches. In the console, open **Compute**, then **Autoscaling groups**.

Groups cost nothing. Their instances are ordinary instances in the project and are billed as such.

## Launch spec

The launch spec is what each member is launched with: instance type, image, SSH keys, user data,
root volume, additional new volumes, network and firewalls, tags and bot. It takes the same values
as [launching an instance](/compute/instances#launch), except that the root volume can't be an
instance store disk and an existing volume, snapshot or network interface can't be named, since
each member needs its own. Every change to the launch spec creates a new version, and new members
always launch from the newest. An image alias is resolved when the launch spec is written and stays pinned: newer builds reach members only after another `set-launch-spec`, which resolves it again and starts a release.

User data is never returned: the group shows only that it is set, and keeps it when you change
other parts of the spec.

## Size and policies

| Setting | Meaning |
| --- | --- |
| Minimum, maximum | The group never has fewer than the minimum or more than the maximum. The maximum is at most 100 |
| Desired | The size the group is working toward, between the two. You can set it by hand; policies move it |
| Instance warmup | Seconds before a new member's metrics count in a policy. Default 300 |
| Scale-in cooldown | Seconds a lower size must hold before the group shrinks. Default 300 |
| Health check grace | Seconds after launch before a member is checked for health. Default 300 |

A policy tracks a target for one metric:

- **CPU**: the average CPU utilization of the group's members, as a percentage.
- **Memory**: the average memory utilization of the members, as a percentage. It needs members on a `linux/ezgh-*` [optimized image](/compute/optimized-images#observability-agent) with a launch-spec bot that may ingest metrics (billed as normal ingestion). Without them the policy never has a value and so holds scale-in.
- **PromQL**: a query over the project's [Observability](/observability) metrics that returns exactly one series. `__asg_instances__` in the query stands for the IDs of the group's warm members, for example `100 * avg(compute_instance_cpu_utilization_ratio{instance=~"__asg_instances__"})`. Aggregate so that one series comes back. A query about your own application can filter by its labels instead and leave it out.

Every 60 seconds the group evaluates each policy. A policy's proposed size is the number of warm
members multiplied by its value and divided by its target, rounded up and capped at the maximum.
Warm members are running, healthy and past their warmup. With several policies, the highest
proposal wins.

- **Scale out** happens at once when a proposal is above the current size.
- **Scale in** happens only when every policy has a value and proposes a smaller size, and has for
  the scale-in cooldown. It removes at most 10 percent of the group (at least one instance) at a
  time and waits out the cooldown again before the next step. A step never raises a desired size
  you set by hand.
- **No data means no change.** A policy with no value (no series, an Observability error or no warm
  member) proposes nothing. The group doesn't scale in while any policy has no value, and never
  scales to the minimum because of one; other policies can still scale it out.

The group retires the oldest launch spec version first, then the oldest instances. Retiring an
instance stops it, which asks the guest to shut down and powers it off after 120 seconds, and then
terminates it.

## Members

Every member carries the tags `ezgh:autoscaling-group=<group ID>` and
`ezgh:launch-spec-version=<version>`. They are reserved and can't be changed on the instance.

After its health check grace, a member that isn't running, or whose
[status checks](/compute/instances#status-checks) have failed for five minutes, is replaced. So is
a member you stop by hand. The group replaces one unhealthy member per evaluation. A member you terminate by hand is replaced at once, outside that limit and the pause below. When two or
more are unhealthy, replacement pauses unless more than half the group is healthy, and the group
reports the error `replacements_paused`.

## Autoscale a Node.js web server

This example runs a Node.js 22 web server on Ubuntu 24.04. A CPU policy keeps average CPU at 50
percent, and a latency policy keeps the server's 95th-percentile request time at 0.3 seconds.

1. **Write the user data.** The cloud-init config below downloads the official Node.js 22 tarball
   from nodejs.org and checks it against the published SHA-256 sums, writes the app, and runs it as
   a systemd service. Use the Node.js 22 version current on nodejs.org for `NODE_VERSION`. The app
   sends its request-duration histogram to Observability with `service.name=webapp`; the metric is
   queried as `http_server_request_duration_seconds` with the label `job`, which Observability sets from `service.name` (see [Metrics](/observability/metrics#names-and-labels)). Replace
   `<ingest key>` with a project [ingest key](/observability/ingest-keys) that allows
   `obs.metrics.ingest`. User data is readable from inside the instance, so use a key with only that
   action.

   ```yaml title="user-data.yaml"
   #cloud-config
   write_files:
     - path: /opt/webapp/package.json
       content: |
         {
           "name": "webapp",
           "private": true,
           "dependencies": {
             "@opentelemetry/auto-instrumentations-node": "^0.82.0"
           }
         }
     - path: /opt/webapp/server.js
       content: |
         const http = require("node:http");

         const server = http.createServer((req, res) => {
           res.end("hello\n");
         });
         server.listen(8080);

         // Scale-in stops the instance; systemd sends SIGTERM to the service.
         // Stop accepting connections and let in-flight requests finish.
         process.on("SIGTERM", () => {
           server.close(() => process.exit(0));
         });
     - path: /etc/webapp.env
       permissions: "0600"
       content: |
         OTEL_EXPORTER_OTLP_ENDPOINT=https://obs-ingest.ezghcloud.com
         OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
         OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20<ingest key>
         OTEL_METRICS_EXPORTER=otlp
         OTEL_LOGS_EXPORTER=none
         OTEL_TRACES_EXPORTER=none
         OTEL_METRIC_EXPORT_INTERVAL=15000
         OTEL_RESOURCE_ATTRIBUTES=service.name=webapp,ezgh.metadata.application=webapp,ezgh.metadata.environment=production
     - path: /etc/systemd/system/webapp.service
       content: |
         [Unit]
         Description=Web app
         After=network-online.target
         Wants=network-online.target

         [Service]
         WorkingDirectory=/opt/webapp
         EnvironmentFile=/etc/webapp.env
         ExecStart=/opt/node/bin/node --require @opentelemetry/auto-instrumentations-node/register server.js
         Restart=always
         KillSignal=SIGTERM
         TimeoutStopSec=90

         [Install]
         WantedBy=multi-user.target
   runcmd:
     - |
       set -e
       NODE_VERSION=v22.21.0
       cd /tmp
       curl -fsSLO "https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64.tar.xz"
       curl -fsSL "https://nodejs.org/dist/${NODE_VERSION}/SHASUMS256.txt" | grep "node-${NODE_VERSION}-linux-x64.tar.xz" | sha256sum -c -
       mkdir -p /opt/node
       tar -xJf "node-${NODE_VERSION}-linux-x64.tar.xz" -C /opt/node --strip-components=1
       cd /opt/webapp
       PATH=/opt/node/bin:$PATH npm install --omit=dev
       systemctl daemon-reload
       systemctl enable --now webapp
   ```

   The service stops within 90 seconds of `SIGTERM`, inside the 120 seconds a stop allows.

2. **Create the group.** Choose `<type>` from `ezgh compute instance-types list`. The `web` firewall replaces the project's default firewall, so it must allow TCP port 8080 from your clients and port 22 for SSH. `--ipv4` gives each member a public IPv4 address; without it members have only public IPv6. `--min 2` keeps
   two members running; `--instance-warmup 120` counts a member's metrics two minutes after launch.
   The query of a `--promql-policy` comes last, because it may contain commas. The query gives 0 while the app is idle, so the policy still has a value and the group can scale in.

   ```sh
   ezgh compute autoscaling-groups create webapp \
     --instance-type <type> --image linux/ubuntu:24.04 --ssh-key laptop \
     --user-data user-data.yaml --firewall web --ipv4 \
     --min 2 --max 10 --desired 2 \
     --cpu-target 50 \
     --promql-policy 'name=latency,target=0.3,query=(histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job="webapp"}[5m]))) > 0) or vector(0)' \
     --instance-warmup 120
   ```

   The group is created with an operation that ends when it is active. The launch spec can't be changed until then.

3. **Watch it.**

   ```sh
   ezgh compute autoscaling-groups get webapp
   ezgh compute autoscaling-groups members webapp
   ezgh compute autoscaling-groups activities webapp
   ```

   Activities list each launch, replacement, scale-in and release step, with its cause and
   outcome, for 30 days.

To change the size or policies later, use `update`. It merges policies by name, so `--cpu-target`
sets the policy named `cpu`, `--memory-target` the one named `memory`, `--promql-policy` adds or
replaces one by name, and `--remove-policy` drops one:

```sh
ezgh compute autoscaling-groups update webapp --min 3 --cpu-target 60
ezgh compute autoscaling-groups update webapp --remove-policy latency
```

## Roll out a new version

`set-launch-spec` changes the launch spec of the running group. The flags you give replace the same
settings of the current spec, and the rest stays, user data included. Use `--no-user-data` to clear
it.

```sh
ezgh compute autoscaling-groups set-launch-spec webapp --user-data user-data-v2.yaml
```

This creates a new version and starts a rolling release. Each step:

1. Launches a batch of new-version members above the desired size (a surge). The batch is the
   batch percent of the desired size, 25 percent by default, and at least one instance.
2. Waits until every member of the batch is running, healthy and past both its warmup and its
   health check grace.
3. Retires the same number of old members.

The release ends when no old member is left. If a surge launch is refused because of a quota or a lack
of capacity, the release continues one instance at a time instead of surging: it launches one new
member, and once the group is at full strength retires one old one. When the desired size is
already the minimum size, as in the example above, that isn't possible without dropping below the
minimum, and the release fails with the refusal's code as its reason (`quota_exceeded` or
`insufficient_capacity`).

A release stops with the status **Failed** when a member of the batch becomes unhealthy, stops or
disappears, even inside its health check grace, or still isn't healthy five minutes after its
warmup and health check grace end (reason `unhealthy`), or when three launches in a row fail (reason `launchFailed`), or in the quota and capacity case above. The
members of that batch are retired and the old members keep serving. The release shows its reason. Writing another launch spec replaces a release in progress; its batch in flight carries over as the new release's step in progress, so nothing old retires until it warms, and is replaced in turn.

To stop a release, cancel it. This retires the members launched for its current step and leaves the rest serving, so the group may run a mix of old and new versions:

```sh
ezgh compute autoscaling-groups cancel-release webapp
```

To go back, write the old spec again with `set-launch-spec`; that is a new version and a new
release. Use `--batch-percent` on `create` or `update` to change the batch size.

## Delete a group

```sh
ezgh compute autoscaling-groups delete webapp --yes
```

Deleting a group terminates its members and then removes the group.

## Limits

| | Limit |
| --- | --- |
| Maximum size of a group | 100 members |
| Groups | 20 per organization per region, a [service quota](/quotas) (`compute.autoscalingGroups`) you can request an increase of |
| Evaluation | Every 60 seconds |
| Activities kept | 30 days |

Members count against your instance, vCPU and volume quotas like any launch, and a launch over a
quota fails and is retried with a growing delay. Permissions are `compute.autoscalingGroups.list`,
`get`, `create`, `update` and `delete`; creating a group or writing a launch spec also needs the
permissions of the instance launch it describes, and a PromQL policy needs `obs.metrics.query` on
the project.

Source: https://docs.ezghcloud.com/compute/autoscaling/index.mdx
