Trace sampling

Trace sampling keeps or drops whole traces by rules you set on a dataset. Observability waits for a trace’s spans to arrive from every service, decides on all of them together (tail sampling), and stores only the traces it keeps. Only kept spans are counted in the traces meters.

• Nothing changes where you send from. SDKs, Collectors and the EZGH Obs Agent send every span, as they do without sampling. There’s no sampler to run and no trace-ID routing to set up.
• Traces only. Logs and metrics are stored as they arrive.
• Request stats still count every span. Span metrics, the service catalog’s stats and the service map count every span, kept or dropped.
• Off by default. A dataset without a policy, or with sampling turned off, keeps every trace.

Each dataset (one environment of a project) has its own policy, so production can sample while staging keeps everything.

A policy

{
  "enabled": true,
  "decisionWait": "30s",
  "rules": [
    { "name": "errors",   "when": { "status": "error" },                         "keep": "all" },
    { "name": "slow",     "when": { "minDuration": "2s" },                       "keep": "all" },
    { "name": "health",   "when": { "root": true, "spanName": ["GET /healthz"] }, "keep": "none" },
    { "name": "vip",      "when": { "attributes": [{ "key": "tenant.tier", "scope": "resource", "equals": "vip" }] },
                          "keep": { "perSecond": 50 } },
    { "name": "checkout", "when": { "service": ["checkout"] },                   "keep": { "percent": 25 } }
  ],
  "default": { "percent": 5 }
}

This keeps every trace with an error and every trace of 2 seconds or more, drops health checks, keeps up to 50 traces a second from VIP tenants, a quarter of the checkout service’s traces, and 5% of everything else.

A rule’s name is 1 to 40 characters of a–z, 0–9, _ and -, starting with a letter or digit, and unique in the policy. default is reserved for the policy’s default. Stats are kept by rule name.

How a trace is decided

Rules are tried in order. The first rule that matches decides the whole trace: every span in it is kept, or none is. When no rule matches, default decides.

A rule matches when every condition in its when holds. when is required; {} matches every trace.

• service, spanName and attributes must hold on one span. "service": ["checkout"] with "spanName": ["POST /pay"] matches a trace with a checkout span named POST /pay, not one where checkout and POST /pay are different spans.
• status and minDuration are about the whole trace, with or without root.
• A trace whose root span hasn’t arrived doesn’t match a root rule.
• Lists hold 1 to 50 values of up to 255 bytes each, matched exactly and case-sensitively.

An attribute filter is { "key", "scope", "equals" } or { "key", "scope", "exists": true }. Unlike a span search’s filter, it has no op, and scope is required: record for the span’s own attributes, resource for its resource’s.

What a rule keeps

• percent is decided by trace ID, as in OpenTelemetry’s consistent probability sampling: the rightmost 56 bits of the trace ID are compared with a threshold for p. The same trace gets the same answer every time, in every dataset, and a trace kept at 5% is also kept at any higher percentage. Trace IDs must be random, as OpenTelemetry SDKs make them.
• perSecond falls through when it’s used up. Past n traces a second (with a burst of n), the rule doesn’t match and the next rule is tried, then the default. So a rule such as “keep errors, up to 100 a second” followed by a percent rule still keeps a share of errors beyond 100 a second.
• default can’t be perSecond.

Decision wait and incomplete traces

A trace is decided decisionWait after its first span arrives: arrival time, not the span’s timestamps. Set it to cover your longest normal trace plus your exporters’ batching delay (the OpenTelemetry SDK’s batch span processor exports every 5 seconds by default, and a Collector’s batch processor adds its own timeout).

• Spans that arrive after their trace is decided follow that decision for about 10 minutes: they’re kept or dropped at once. After that, a late span is decided on its own spans, so a late span of a dropped error trace can be kept by itself, or one of a kept trace dropped.
• Very large traces are decided early, when they reach 10,000 spans or 32 MiB. Their later spans follow the decision.
• Under heavy load, when too many spans are waiting, the oldest traces are decided early, under the same rules, on the spans that have arrived. The stats count them under (memory).
• Rules see only the spans that arrived in time. A minDuration rule measures those spans, and a status rule misses an error span that arrives after the decision.

A kept trace is searchable about decisionWait plus a few seconds after its first span arrives.

Changing the policy applies to traces decided from a few seconds after you save it. Turning sampling on applies to spans that arrive afterward, so a trace in flight at that moment can have some spans stored in full. Turning it off keeps every trace that arrives afterward; traces already waiting are decided under the last policy.

What sampling changes

So a service’s requests and errors on its service page count every request, while its traces list shows the kept ones. The console marks this with a Sampled badge.

Rarely, after an interruption on Observability’s side, a kept trace’s spans can be stored twice, as when an exporter retries; both copies are returned, with the same spanId. If Observability falls behind deciding, sends to a sampled dataset can be refused with 503 and a retry delay, and exporters retry them.

Billing and limits

• Only kept spans are metered. Dropped spans aren’t counted in the traces’ ingested bytes or in hot or cold stored bytes.
• Ingest rate limits count every span received, including the spans sampling then drops. The organization’s and the dataset’s rates apply before sampling.
• Span metrics stay free, and count every span.

Stats

For each dataset, Observability counts the traces and spans it received and kept over the last hour or the last 24 hours, and for each rule the traces it matched (decided) and kept.

{
  "window": "1h",
  "received": { "traces": 1203311, "spans": 18051976 },
  "kept":     { "traces": 113064,  "spans": 3618065 },
  "rules": [
    { "name": "errors",   "matched": { "traces": 8062 },   "kept": { "traces": 8062 } },
    { "name": "slow",     "matched": { "traces": 14560 },  "kept": { "traces": 14560 } },
    { "name": "health",   "matched": { "traces": 109742 }, "kept": { "traces": 0 } },
    { "name": "vip",      "matched": { "traces": 17942 },  "kept": { "traces": 17942 } },
    { "name": "checkout", "matched": { "traces": 108406 }, "kept": { "traces": 25270 } },
    { "name": "default",  "matched": { "traces": 944599 }, "kept": { "traces": 47230 } }
  ],
  "complete": true
}

• rules lists the policy’s rules in order, then default, then (memory) when traces were decided early under load, then rules that have since been renamed or removed but still have counts in the window.
• matched counts the traces a rule decided. A perSecond rule’s traces that fell through are counted under the rule that decided them.
• complete is false when some of the window’s counts weren’t recorded, so the counts are lower than what was sampled.
• Counts follow the traffic by about one decision wait.

In the console

Open Observability > Datasets, a dataset, then the Sampling tab.

• Set up sampling starts a policy. Turn Tail sampling on or off, set the Decision wait in seconds, add rules, reorder them with the arrows, edit or delete them, and choose the Default: Keep a percentage, Keep all or Drop. Save changes saves the whole policy; turning sampling off asks you to confirm.
• If someone else saved the policy since you opened it, saving says so and loads theirs, and your changes aren’t applied.
• Below the policy, a 1 hour / 24 hours switch shows the traces and spans received and kept, and each rule’s matched and kept traces with its kept share.
• Without obs.datasets.update, the tab shows the policy and stats read-only.

A Sampled badge marks traces that come from a sampled dataset: on the dataset’s page, on the Traces page (“Counts on this page are of stored traces”), and on a service’s page and the service map next to the time range. A service’s View error traces and a map node’s View traces carry the badge too, with “Charts count every request; traces shows the sampled ones”. With All environments, the badge shows when any environment of the project samples. It opens the dataset’s Sampling tab, or the Datasets list when several environments sample.

From the CLI and the API

ezgh obs datasets sampling get production --project web
ezgh obs datasets sampling get production -o json --project web > sampling.json
ezgh obs datasets sampling set production -f sampling.json --project web
ezgh obs datasets sampling stats production --window 24h --project web

Limits

Access and Trails

ObsFullAccess includes both; ObsReadOnlyAccess and ReadOnlyAccess include obs.datasets.get.

Changing a policy is recorded in Trails as UpdateDatasetSampling, with eventSource obs.ezghcloud.com. The event records whether sampling is on, the rule names and the new version, not the rules’ conditions, because their values can be your data. Viewing a policy and its stats isn’t recorded.