Skip to content

Autoscaling groups

Keep a number of identical instances running, add and remove them with load, replace unhealthy ones, and roll out a new launch spec in batches.

Updated View as Markdown

An autoscaling group launches instances from one launch spec and keeps their number between a minimum and a maximum size. It adds instances when a policy says the group is under load, removes them when load stays low, replaces instances that stop working, and rolls a new launch spec out across the group in batches. In the console, open Compute, then Autoscaling groups.

Groups cost nothing. Their instances are ordinary instances in the project and are billed as such.

Launch spec

The launch spec is what each member is launched with: instance type, image, SSH keys, user data, root volume, additional new volumes, network and firewalls, tags and bot. It takes the same values as launching an instance, except that the root volume can’t be an instance store disk and an existing volume, snapshot or network interface can’t be named, since each member needs its own. Every change to the launch spec creates a new version, and new members always launch from the newest. An image alias is resolved when the launch spec is written and stays pinned: newer builds reach members only after another set-launch-spec, which resolves it again and starts a release.

User data is never returned: the group shows only that it is set, and keeps it when you change other parts of the spec.

Size and policies

Setting Meaning
Minimum, maximum The group never has fewer than the minimum or more than the maximum. The maximum is at most 100
Desired The size the group is working toward, between the two. You can set it by hand; policies move it
Instance warmup Seconds before a new member’s metrics count in a policy. Default 300
Scale-in cooldown Seconds a lower size must hold before the group shrinks. Default 300
Health check grace Seconds after launch before a member is checked for health. Default 300

A policy tracks a target for one metric:

  • CPU: the average CPU utilization of the group’s members, as a percentage.
  • Memory: the average memory utilization of the members, as a percentage. It needs members on a linux/ezgh-* optimized image with a launch-spec bot that may ingest metrics (billed as normal ingestion). Without them the policy never has a value and so holds scale-in.
  • PromQL: a query over the project’s Observability metrics that returns exactly one series. __asg_instances__ in the query stands for the IDs of the group’s warm members, for example 100 * avg(compute_instance_cpu_utilization_ratio{instance=~"__asg_instances__"}). Aggregate so that one series comes back. A query about your own application can filter by its labels instead and leave it out.

Every 60 seconds the group evaluates each policy. A policy’s proposed size is the number of warm members multiplied by its value and divided by its target, rounded up and capped at the maximum. Warm members are running, healthy and past their warmup. With several policies, the highest proposal wins.

  • Scale out happens at once when a proposal is above the current size.
  • Scale in happens only when every policy has a value and proposes a smaller size, and has for the scale-in cooldown. It removes at most 10 percent of the group (at least one instance) at a time and waits out the cooldown again before the next step. A step never raises a desired size you set by hand.
  • No data means no change. A policy with no value (no series, an Observability error or no warm member) proposes nothing. The group doesn’t scale in while any policy has no value, and never scales to the minimum because of one; other policies can still scale it out.

The group retires the oldest launch spec version first, then the oldest instances. Retiring an instance stops it, which asks the guest to shut down and powers it off after 120 seconds, and then terminates it.

Members

Every member carries the tags ezgh:autoscaling-group=<group ID> and ezgh:launch-spec-version=<version>. They are reserved and can’t be changed on the instance.

After its health check grace, a member that isn’t running, or whose status checks have failed for five minutes, is replaced. So is a member you stop by hand. The group replaces one unhealthy member per evaluation. A member you terminate by hand is replaced at once, outside that limit and the pause below. When two or more are unhealthy, replacement pauses unless more than half the group is healthy, and the group reports the error replacements_paused.

Autoscale a Node.js web server

This example runs a Node.js 22 web server on Ubuntu 24.04. A CPU policy keeps average CPU at 50 percent, and a latency policy keeps the server’s 95th-percentile request time at 0.3 seconds.

  1. Write the user data. The cloud-init config below downloads the official Node.js 22 tarball from nodejs.org and checks it against the published SHA-256 sums, writes the app, and runs it as a systemd service. Use the Node.js 22 version current on nodejs.org for NODE_VERSION. The app sends its request-duration histogram to Observability with service.name=webapp; the metric is queried as http_server_request_duration_seconds with the label job, which Observability sets from service.name (see Metrics). Replace <ingest key> with a project ingest key that allows obs.metrics.ingest. User data is readable from inside the instance, so use a key with only that action.

    user-data.yamlyaml
    #cloud-config
    write_files:
      - path: /opt/webapp/package.json
        content: |
          {
            "name": "webapp",
            "private": true,
            "dependencies": {
              "@opentelemetry/auto-instrumentations-node": "^0.82.0"
            }
          }
      - path: /opt/webapp/server.js
        content: |
          const http = require("node:http");
    
          const server = http.createServer((req, res) => {
            res.end("hello\n");
          });
          server.listen(8080);
    
          // Scale-in stops the instance; systemd sends SIGTERM to the service.
          // Stop accepting connections and let in-flight requests finish.
          process.on("SIGTERM", () => {
            server.close(() => process.exit(0));
          });
      - path: /etc/webapp.env
        permissions: "0600"
        content: |
          OTEL_EXPORTER_OTLP_ENDPOINT=https://obs-ingest.ezghcloud.com
          OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
          OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20<ingest key>
          OTEL_METRICS_EXPORTER=otlp
          OTEL_LOGS_EXPORTER=none
          OTEL_TRACES_EXPORTER=none
          OTEL_METRIC_EXPORT_INTERVAL=15000
          OTEL_RESOURCE_ATTRIBUTES=service.name=webapp,ezgh.metadata.application=webapp,ezgh.metadata.environment=production
      - path: /etc/systemd/system/webapp.service
        content: |
          [Unit]
          Description=Web app
          After=network-online.target
          Wants=network-online.target
    
          [Service]
          WorkingDirectory=/opt/webapp
          EnvironmentFile=/etc/webapp.env
          ExecStart=/opt/node/bin/node --require @opentelemetry/auto-instrumentations-node/register server.js
          Restart=always
          KillSignal=SIGTERM
          TimeoutStopSec=90
    
          [Install]
          WantedBy=multi-user.target
    runcmd:
      - |
        set -e
        NODE_VERSION=v22.21.0
        cd /tmp
        curl -fsSLO "https://nodejs.org/dist/${NODE_VERSION}/node-${NODE_VERSION}-linux-x64.tar.xz"
        curl -fsSL "https://nodejs.org/dist/${NODE_VERSION}/SHASUMS256.txt" | grep "node-${NODE_VERSION}-linux-x64.tar.xz" | sha256sum -c -
        mkdir -p /opt/node
        tar -xJf "node-${NODE_VERSION}-linux-x64.tar.xz" -C /opt/node --strip-components=1
        cd /opt/webapp
        PATH=/opt/node/bin:$PATH npm install --omit=dev
        systemctl daemon-reload
        systemctl enable --now webapp

    The service stops within 90 seconds of SIGTERM, inside the 120 seconds a stop allows.

  2. Create the group. Choose <type> from ezgh compute instance-types list. The web firewall replaces the project’s default firewall, so it must allow TCP port 8080 from your clients and port 22 for SSH. --ipv4 gives each member a public IPv4 address; without it members have only public IPv6. --min 2 keeps two members running; --instance-warmup 120 counts a member’s metrics two minutes after launch. The query of a --promql-policy comes last, because it may contain commas. The query gives 0 while the app is idle, so the policy still has a value and the group can scale in.

    ezgh compute autoscaling-groups create webapp \
      --instance-type <type> --image linux/ubuntu:24.04 --ssh-key laptop \
      --user-data user-data.yaml --firewall web --ipv4 \
      --min 2 --max 10 --desired 2 \
      --cpu-target 50 \
      --promql-policy 'name=latency,target=0.3,query=(histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job="webapp"}[5m]))) > 0) or vector(0)' \
      --instance-warmup 120

    The group is created with an operation that ends when it is active. The launch spec can’t be changed until then.

  3. Watch it.

    ezgh compute autoscaling-groups get webapp
    ezgh compute autoscaling-groups members webapp
    ezgh compute autoscaling-groups activities webapp

    Activities list each launch, replacement, scale-in and release step, with its cause and outcome, for 30 days.

To change the size or policies later, use update. It merges policies by name, so --cpu-target sets the policy named cpu, --memory-target the one named memory, --promql-policy adds or replaces one by name, and --remove-policy drops one:

ezgh compute autoscaling-groups update webapp --min 3 --cpu-target 60
ezgh compute autoscaling-groups update webapp --remove-policy latency

Roll out a new version

set-launch-spec changes the launch spec of the running group. The flags you give replace the same settings of the current spec, and the rest stays, user data included. Use --no-user-data to clear it.

ezgh compute autoscaling-groups set-launch-spec webapp --user-data user-data-v2.yaml

This creates a new version and starts a rolling release. Each step:

  1. Launches a batch of new-version members above the desired size (a surge). The batch is the batch percent of the desired size, 25 percent by default, and at least one instance.
  2. Waits until every member of the batch is running, healthy and past both its warmup and its health check grace.
  3. Retires the same number of old members.

The release ends when no old member is left. If a surge launch is refused because of a quota or a lack of capacity, the release continues one instance at a time instead of surging: it launches one new member, and once the group is at full strength retires one old one. When the desired size is already the minimum size, as in the example above, that isn’t possible without dropping below the minimum, and the release fails with the refusal’s code as its reason (quota_exceeded or insufficient_capacity).

A release stops with the status Failed when a member of the batch becomes unhealthy, stops or disappears, even inside its health check grace, or still isn’t healthy five minutes after its warmup and health check grace end (reason unhealthy), or when three launches in a row fail (reason launchFailed), or in the quota and capacity case above. The members of that batch are retired and the old members keep serving. The release shows its reason. Writing another launch spec replaces a release in progress; its batch in flight carries over as the new release’s step in progress, so nothing old retires until it warms, and is replaced in turn.

To stop a release, cancel it. This retires the members launched for its current step and leaves the rest serving, so the group may run a mix of old and new versions:

ezgh compute autoscaling-groups cancel-release webapp

To go back, write the old spec again with set-launch-spec; that is a new version and a new release. Use --batch-percent on create or update to change the batch size.

Delete a group

ezgh compute autoscaling-groups delete webapp --yes

Deleting a group terminates its members and then removes the group.

Limits

Limit
Maximum size of a group 100 members
Groups 20 per organization per region, a service quota (compute.autoscalingGroups) you can request an increase of
Evaluation Every 60 seconds
Activities kept 30 days

Members count against your instance, vCPU and volume quotas like any launch, and a launch over a quota fails and is retried with a growing delay. Permissions are compute.autoscalingGroups.list, get, create, update and delete; creating a group or writing a launch spec also needs the permissions of the instance launch it describes, and a PromQL policy needs obs.metrics.query on the project.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close