Workflows in Observability

Turn on Workflows in Observability for a project, and its workflow runs appear in the project’s own Observability with no code changes:

• Traces: each run is a trace, with spans for the run, its invocations, steps, sleeps, callbacks and child workflows. Your own OpenTelemetry spans can nest under the invocation that ran them, through traceparent.
• Metrics: counts and durations of runs, step attempts and invocations, queried with PromQL.

It’s off by default, and set per project. Runs and their steps never wait for Observability: while Observability is slow or unavailable, telemetry arrives late and nothing else changes.

Turn it on

The setting has two parts: whether it’s on, and the Observability environment the telemetry goes to, production unless you choose another. The environment chooses the project’s dataset, which is created on first use like any other.

ezgh workflows settings observability set --enabled --environment production --project web
ezgh workflows settings observability get --project web
ezgh workflows settings observability set --enabled=false --project web

$ ezgh workflows settings observability get --project web
Enabled:      yes
Environment:  production
Updated:      2026-10-12T18:04:05.123Z

{ "enabled": true, "environment": "production", "updatedAt": "2026-10-12T18:04:05.123Z" }

A change takes effect within seconds, for runs already in progress too:

• Turned on mid-run: later spans arrive, with their parents’ IDs, even though the spans that ended before were never sent. The trace shows the spans from then on.
• Turned off: nothing more is sent, and new invocations no longer carry traceparent. What was sent stays in Observability under the dataset’s retention.

Permissions

Turning it on, or changing the environment while it’s on, sends telemetry into the project’s Observability, so it also needs obs.traces.ingest and obs.metrics.ingest on the project, as creating an ingest key does. Turning it off needs only workflows.settings.update.

WorkflowsFullAccess allows both actions; WorkflowsReadOnlyAccess and ReadOnlyAccess allow workflows.settings.get. Viewing the traces and metrics needs the usual Observability actions: obs.traces.search, obs.traces.getSpans and obs.metrics.query.

Changing the setting is recorded in Trails as UpdateObservabilitySettings; reading it isn’t. When the telemetry creates a dataset, its CreateDataset event is a service event by workflows.ezghcloud.com.

Traces

A run’s trace ID is its run ID: the UUID’s 32 hex digits without hyphens. Search for it in the Traces page, or read it with Get a trace.

workflow.run <workflow>         run created → ended, one per run attempt
├─ workflow.invocation          invocation created → completed, failed or abandoned
│    ├─ (your spans)            through traceparent
│    └─ workflow.step <name>    step started → completed, retrying or failed, one per attempt
├─ workflow.sleep <name>        sleep started → ended
├─ workflow.callback <name>     callback created → completed or timed out
└─ workflow.invoke <name>       child workflow started → ended

• Retrying a run starts a new workflow.run span in the same trace, and each step attempt has its own workflow.step span.
• A child workflow is a trace of its own (its own run ID). Its workflow.run span links to the parent’s workflow.invoke span, and that span links to the child’s.
• Spans are sent when they end, so a long run’s steps appear before its workflow.run span.
• Status: a failed, timed-out or abandoned span, and a step attempt that’s retried, has status Error, with ezgh.workflows.error_code and ezgh.workflows.error_name. A cancelled run’s span is Unset, with ezgh.workflows.status cancelled. Error messages, inputs and outputs are never sent.

The resource of every span and metric has service.name ezgh.workflows, ezgh.metadata.application set to the app’s name, and ezgh.metadata.environment set to the setting’s environment.

Timing

Span times are when EZGH recorded each change, not when your code ran. The SDKs report a step’s result with the invocation’s next checkpoint, so a step can end later in its span than it did in your code, and a sleep or callback ends when the run resumes. Your own spans have exact times.

Metrics

• status is completed, failed, cancelled or timedOut. A step’s outcome is completed, retrying or failed; an invocation’s is completed, failed or abandoned. kind is push or pull. trigger is api, schedule or invoke.
• Durations are of a run attempt, a step attempt, and an invocation from its creation. Histogram buckets run from 5 ms to 24 hours.
• Run IDs and step names are never labels, so the number of series depends only on your apps and workflows.
• Every series also has environment, application (the app) and job ezgh.workflows; see Names and labels. The sums and histograms are delta, so rate, increase and histogram_quantile are exact.

Failed runs a second, by workflow:

sum by (workflow) (rate(workflows_runs_ended_total{status="failed"}[5m]))

The 95th percentile of completed runs’ durations, by workflow:

histogram_quantile(0.95,
  sum by (workflow, le) (rate(workflows_run_duration_seconds_bucket{status="completed"}[1h])))

Step retries a minute, by app:

sum by (app) (increase(workflows_step_attempts_total{outcome="retrying"}[1m]))

Your own spans: traceparent

While the setting is on, each invocation carries a W3C traceparent naming its workflow.invocation span: push requests have a traceparent header, and pulled invocations a traceparent field. The SDKs expose it as traceparent on the workflow’s context, an empty string while the setting is off. Make it the parent of your spans to nest them under the invocation. It needs SDK version 0.2.0 or later: TypeScript, Go and Python.

A push endpoint whose HTTP server is already instrumented with OpenTelemetry continues the trace from the request’s header by itself. Pull workers, and spans you start yourself, use the SDK’s value:

import { context, propagation, trace } from "@opentelemetry/api"
import { workflow } from "@ezgh/workflows"

const tracer = trace.getTracer("billing")

export const charge = workflow("charge", async ({ input, step, traceparent }) => {
  // An empty traceparent leaves the context as it is.
  const parent = propagation.extract(context.active(), { traceparent })
  return step.run("charge-card", () =>
    tracer.startActiveSpan("charge-card", {}, parent, async (span) => {
      try {
        return await payments.charge(input.orderId)
      } finally {
        span.end()
      }
    }),
  )
})

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"

	workflows "github.com/ezgamehost/workflows-sdk-go"
)

var tracer = otel.Tracer("billing")

var charge = workflows.New("charge", func(ctx workflows.Context, in Order) (Receipt, error) {
	carrier := propagation.MapCarrier{"traceparent": ctx.Run().Traceparent}
	return workflows.Run(ctx, "charge-card", func(sc workflows.StepContext) (Receipt, error) {
		// An empty traceparent leaves the context as it is.
		spanCtx, span := tracer.Start(propagation.TraceContext{}.Extract(sc, carrier), "charge-card")
		defer span.End()
		return payments.Charge(spanCtx, in.OrderID)
	})
})

from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

from ezgh_workflows import Context, workflow

tracer = trace.get_tracer("billing")


@workflow("charge")
async def charge(ctx: Context) -> dict:
    # An empty traceparent gives an empty context: the span starts a trace of its own.
    parent = TraceContextTextMapPropagator().extract({"traceparent": ctx.traceparent})

    def charge_card() -> dict:
        with tracer.start_as_current_span("charge-card", context=parent):
            return payments.charge(ctx.input["order_id"])

    return await ctx.step.run("charge-card", charge_card)

• Each invocation has its own traceparent. Code outside steps runs again on every invocation, so a span started there appears under each invocation that ran it.
• The SDKs don’t depend on OpenTelemetry: install and configure it in your application, with an ingest key sending to the same project and environment.
• While the setting is off, no traceparent is sent, so your sampler never waits for a parent that won’t arrive.

Billing and limits

The telemetry is ingested like anything you send: its bytes and storage count in the Observability meters of the environment’s dataset, and it goes through the same rate limits, trace sampling and transform rules. Workflows’ own meters are unchanged.

Delivery

Telemetry is delivered at least once. After a timeout, a span or metric point can be sent again:

• A repeated span has the same trace and span IDs, so a trace can show the same span twice.
• A repeated metric point counts twice in rate and increase.

Telemetry older than 24 hours that couldn’t be delivered, because Observability was unavailable or the project’s limits refused it, isn’t sent.