---
title: "Workflows in Observability"
description: "Send a project's workflow runs to its own Observability as traces and metrics, nest your own spans under them with traceparent, and query them with PromQL."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Workflows in Observability

Turn on **Workflows in Observability** for a project, and its workflow runs appear in the
project's own [Observability](/observability) with no code changes:

- **Traces:** each run is a trace, with spans for the run, its invocations, steps, sleeps,
  callbacks and child workflows. Your own OpenTelemetry spans can nest under the invocation that
  ran them, through [`traceparent`](#your-own-spans-traceparent).
- **Metrics:** counts and durations of runs, step attempts and invocations, queried with
  [PromQL](/observability/metrics).

It's **off by default**, and set per project. Runs and their steps never wait for Observability:
while Observability is slow or unavailable, telemetry arrives late and nothing else changes.

## Turn it on

The setting has two parts: whether it's on, and the Observability **environment** the telemetry
goes to, `production` unless you choose another. The environment chooses the project's
[dataset](/observability#datasets-are-environments), which is created on first use like any other.

### Console

Open **Workflows**, then **Settings** in the sidebar. In the **Observability** card, turn on
**Send runs to Observability**, set the environment and save. **Open Observability** goes to the
project's Observability.

While it's on, a run's page has **View trace**, which opens the run's trace in Observability.
### CLI

```sh
ezgh workflows settings observability set --enabled --environment production --project web
ezgh workflows settings observability get --project web
ezgh workflows settings observability set --enabled=false --project web
```

```
$ ezgh workflows settings observability get --project web
Enabled:      yes
Environment:  production
Updated:      2026-10-12T18:04:05.123Z
```

`set` changes only what you pass: `--enabled` alone keeps the environment, and `--environment`
alone keeps whether it's on. See
[`ezgh workflows settings observability`](/cli/reference/workflows-settings-observability).
### API

| Operation | Request |
| --- | --- |
| [GetObservabilitySettings](/workflows-api/GetObservabilitySettings/) | `GET /v1/organizations/<organization ID>/projects/<project ID>/settings/observability` |
| [UpdateObservabilitySettings](/workflows-api/UpdateObservabilitySettings/) | `PUT …/settings/observability` |

```json
{ "enabled": true, "environment": "production", "updatedAt": "2026-10-12T18:04:05.123Z" }
```

- A project that never saved the setting answers `{ "enabled": false, "environment": "production", "updatedAt": null }`.
- `PUT` takes `{ "enabled": true, "environment": "staging" }`. `enabled` is required;
`environment` is optional, and an omitted one keeps the stored environment. It takes an
`Idempotency-Key`.
- `environment` is trimmed and lowercased, then must match `^[a-z0-9][a-z0-9._-]{0,62}$`.
Otherwise, or with an unknown field, the answer is `400 invalid_request`.
- Without `obs.traces.ingest` or `obs.metrics.ingest` (see [Permissions](#permissions)), turning
it on is `403 access_denied`, naming the missing action.
- `409 settings_changed` means someone changed the setting at the same time. Read it and try
again.

A change takes effect within seconds, for runs already in progress too:

- **Turned on mid-run:** later spans arrive, with their parents' IDs, even though the spans that
  ended before were never sent. The trace shows the spans from then on.
- **Turned off:** nothing more is sent, and new invocations no longer carry `traceparent`. What
  was sent stays in Observability under the dataset's retention.

## Permissions

| Action | Allows | Checked against |
| --- | --- | --- |
| `workflows.settings.get` | View the project's Workflows settings | Project |
| `workflows.settings.update` | Change the project's Workflows settings | Project |

**Turning it on, or changing the environment while it's on,** sends telemetry into the project's
Observability, so it also needs `obs.traces.ingest` and `obs.metrics.ingest` on the project, as
creating an [ingest key](/observability/ingest-keys) does. Turning it off needs only
`workflows.settings.update`.

`WorkflowsFullAccess` allows both actions; `WorkflowsReadOnlyAccess` and `ReadOnlyAccess` allow
`workflows.settings.get`. Viewing the traces and metrics needs the usual Observability actions:
`obs.traces.search`, `obs.traces.getSpans` and `obs.metrics.query`.

Changing the setting is recorded in [Trails](/trails) as `UpdateObservabilitySettings`; reading it
isn't. When the telemetry creates a dataset, its `CreateDataset` event is a service event by
`workflows.ezghcloud.com`.

## Traces

A run's **trace ID is its run ID**: the UUID's 32 hex digits without hyphens. Search for it in the
Traces page, or read it with [Get a trace](/obs-api/GetTrace/).

```
workflow.run <workflow>         run created → ended, one per run attempt
├─ workflow.invocation          invocation created → completed, failed or abandoned
│    ├─ (your spans)            through traceparent
│    └─ workflow.step <name>    step started → completed, retrying or failed, one per attempt
├─ workflow.sleep <name>        sleep started → ended
├─ workflow.callback <name>     callback created → completed or timed out
└─ workflow.invoke <name>       child workflow started → ended
```

- **Retrying a run** starts a new `workflow.run` span in the same trace, and each step attempt
  has its own `workflow.step` span.
- **A child workflow** is a trace of its own (its own run ID). Its `workflow.run` span links to
  the parent's `workflow.invoke` span, and that span links to the child's.
- **Spans are sent when they end,** so a long run's steps appear before its `workflow.run` span.
- **Status:** a failed, timed-out or abandoned span, and a step attempt that's retried, has
  status `Error`, with `ezgh.workflows.error_code` and `ezgh.workflows.error_name`. A cancelled run's span is `Unset`,
  with `ezgh.workflows.status` `cancelled`. Error messages, inputs and outputs are never sent.

| Attribute | On | Value |
| --- | --- | --- |
| `ezgh.workflows.run_id` | Every span | The run ID |
| `ezgh.workflows.app` | Every span | The app's name |
| `ezgh.workflows.workflow` | Every span | The workflow's name |
| `ezgh.workflows.attempt` | Every span | The run attempt, or the step attempt on a step span |
| `ezgh.workflows.step` | Step, sleep, callback and invoke spans, and an invocation span that names a step | The step's name |
| `ezgh.workflows.invocation_id` | Invocation and step spans | The invocation |
| `ezgh.workflows.invocation_kind` | Invocation spans | `push` or `pull` |
| `ezgh.workflows.trigger` | Run spans | `api`, `schedule` or `invoke` |
| `ezgh.workflows.schedule_id` | Run spans of a schedule's runs | The schedule |
| `ezgh.workflows.status` | Run and invoke spans, and a timed-out callback's span | `completed`, `failed`, `cancelled` or `timedOut` |
| `ezgh.workflows.outcome` | Invocation and step spans | `completed`, `failed`, `retrying` (steps) or `abandoned` (invocations) |

The resource of every span and metric has `service.name` `ezgh.workflows`,
`ezgh.metadata.application` set to the app's name, and `ezgh.metadata.environment` set to the
setting's environment.

### Timing

Span times are when EZGH recorded each change, not when your code ran. The SDKs report a step's
result with the invocation's next checkpoint, so a step can end later in its span than it did in
your code, and a sleep or callback ends when the run resumes. Your own spans have exact times.

## Metrics

| OTLP metric | Type | PromQL names | Labels |
| --- | --- | --- | --- |
| `workflows.runs.started` | Sum | `workflows_runs_started_total` | `app`, `workflow`, `trigger` |
| `workflows.runs.ended` | Sum | `workflows_runs_ended_total` | `app`, `workflow`, `status` |
| `workflows.run.duration` | Histogram, seconds | `workflows_run_duration_seconds_bucket`, `…_sum`, `…_count` | `app`, `workflow`, `status` |
| `workflows.step.attempts` | Sum | `workflows_step_attempts_total` | `app`, `workflow`, `outcome` |
| `workflows.step.duration` | Histogram, seconds | `workflows_step_duration_seconds_bucket`, `…_sum`, `…_count` | `app`, `workflow`, `outcome` |
| `workflows.invocations` | Sum | `workflows_invocations_total` | `app`, `workflow`, `kind`, `outcome` |
| `workflows.invocation.duration` | Histogram, seconds | `workflows_invocation_duration_seconds_bucket`, `…_sum`, `…_count` | `app`, `workflow`, `kind`, `outcome` |

- `status` is `completed`, `failed`, `cancelled` or `timedOut`. A step's `outcome` is
  `completed`, `retrying` or `failed`; an invocation's is `completed`, `failed` or `abandoned`.
  `kind` is `push` or `pull`. `trigger` is `api`, `schedule` or `invoke`.
- Durations are of a run attempt, a step attempt, and an invocation from its creation. Histogram
  buckets run from 5 ms to 24 hours.
- Run IDs and step names are never labels, so the number of series depends only on your apps and
  workflows.
- Every series also has `environment`, `application` (the app) and `job` `ezgh.workflows`; see
  [Names and labels](/observability/metrics#names-and-labels). The sums and histograms are delta,
  so `rate`, `increase` and `histogram_quantile` are exact.

Failed runs a second, by workflow:

```promql
sum by (workflow) (rate(workflows_runs_ended_total{status="failed"}[5m]))
```

The 95th percentile of completed runs' durations, by workflow:

```promql
histogram_quantile(0.95,
  sum by (workflow, le) (rate(workflows_run_duration_seconds_bucket{status="completed"}[1h])))
```

Step retries a minute, by app:

```promql
sum by (app) (increase(workflows_step_attempts_total{outcome="retrying"}[1m]))
```

## Your own spans: traceparent

While the setting is on, each invocation carries a W3C `traceparent` naming its
`workflow.invocation` span: push requests have a `traceparent` header, and pulled invocations a
`traceparent` field. The SDKs expose it as `traceparent` on the workflow's context, an empty
string while the setting is off. Make it the parent of your spans to nest them under the
invocation. It needs SDK version 0.2.0 or later: TypeScript, Go and Python.

A push endpoint whose HTTP server is already instrumented with OpenTelemetry continues the trace
from the request's header by itself. Pull workers, and spans you start yourself, use the SDK's
value:

### TypeScript

```ts
import { context, propagation, trace } from "@opentelemetry/api"
import { workflow } from "@ezgh/workflows"

const tracer = trace.getTracer("billing")

export const charge = workflow("charge", async ({ input, step, traceparent }) => {
  // An empty traceparent leaves the context as it is.
  const parent = propagation.extract(context.active(), { traceparent })
  return step.run("charge-card", () =>
    tracer.startActiveSpan("charge-card", {}, parent, async (span) => {
      try {
        return await payments.charge(input.orderId)
      } finally {
        span.end()
      }
    }),
  )
})
```

`propagation.extract` uses the registered propagator: the OpenTelemetry Node.js SDK registers W3C
Trace Context by default.
### Go

```go
import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"

	workflows "github.com/ezgamehost/workflows-sdk-go"
)

var tracer = otel.Tracer("billing")

var charge = workflows.New("charge", func(ctx workflows.Context, in Order) (Receipt, error) {
	carrier := propagation.MapCarrier{"traceparent": ctx.Run().Traceparent}
	return workflows.Run(ctx, "charge-card", func(sc workflows.StepContext) (Receipt, error) {
		// An empty traceparent leaves the context as it is.
		spanCtx, span := tracer.Start(propagation.TraceContext{}.Extract(sc, carrier), "charge-card")
		defer span.End()
		return payments.Charge(spanCtx, in.OrderID)
	})
})
```
### Python

```python
from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

from ezgh_workflows import Context, workflow

tracer = trace.get_tracer("billing")


@workflow("charge")
async def charge(ctx: Context) -> dict:
    # An empty traceparent gives an empty context: the span starts a trace of its own.
    parent = TraceContextTextMapPropagator().extract({"traceparent": ctx.traceparent})

    def charge_card() -> dict:
        with tracer.start_as_current_span("charge-card", context=parent):
            return payments.charge(ctx.input["order_id"])

    return await ctx.step.run("charge-card", charge_card)
```

- Each invocation has its own `traceparent`. Code outside steps runs again on every invocation,
  so a span started there appears under each invocation that ran it.
- The SDKs don't depend on OpenTelemetry: install and configure it in your application, with an
  [ingest key](/observability/ingest-keys) sending to the same project and environment.
- While the setting is off, no `traceparent` is sent, so your sampler never waits for a parent
  that won't arrive.

## Billing and limits

The telemetry is ingested like anything you send: its bytes and storage count in the
[Observability meters](/observability/limits) of the environment's dataset, and it goes through
the same rate limits, [trace sampling](/observability/sampling) and
[transform rules](/observability/transforms). Workflows' own meters are unchanged.

## Delivery

Telemetry is delivered **at least once**. After a timeout, a span or metric point can be sent
again:

- A repeated span has the same trace and span IDs, so a trace can show the same span twice.
- A repeated metric point counts twice in `rate` and `increase`.

Telemetry older than 24 hours that couldn't be delivered, because Observability was unavailable or
the project's limits refused it, isn't sent.

Source: https://docs.ezghcloud.com/workflows/observability/index.mdx