Skip to content

Retries and idempotency

Step retry policies, timeouts, errors that stop or delay retries, and idempotency keys for at-least-once steps.

Updated View as Markdown

Retry policies

A step that throws (or returns an error, in Go) is retried with backoff. The default policy is 5 attempts, the first retry after 1 second, doubling each time up to 10 minutes apart.

Field Range
Attempts (maxAttempts) 1 to 100
First delay (initialMs) 0 or more
Longest delay (maxMs) Up to 24 hours
Multiplier (multiplier) 1 to 10

Set a policy on one step, or as the default for a workflow’s steps:

// One step: fields you leave out keep their defaults
await step.run("charge", () => charge(order), { retry: { maxAttempts: 3 } })

// Every step of a workflow
export const checkout = workflow("checkout", async ({ step }) => { /* … */ }, { retry: { maxAttempts: 3 } })
// A policy is used as given, so start from the default
p := workflows.DefaultRetry
p.MaxAttempts = 3
order, err := workflows.Run(ctx, "charge", charge, workflows.WithStepRetry(p))

// Every step of a workflow
var checkout = workflows.New("checkout", run, workflows.WithRetry(p))
# One step: fields you leave out keep their defaults
await ctx.step.run("charge", charge, retry={"maxAttempts": 3})

# Every step of a workflow
@workflow("checkout", retry={"maxAttempts": 3})
async def checkout(ctx: Context) -> None: ...

When a step has used up its attempts, its last error is thrown into the workflow at the step’s call. The workflow can catch it and carry on; if it doesn’t, the run fails. A retried attempt counts again whether it threw or its process crashed: an invocation lost while a step was running is a failed attempt of that step, with code workflows.invocation_lost.

Stop or delay retries

TypeScript Go Python
Fail the step at once throw new NonRetriableError(message) return workflows.NonRetriable(err) raise NonRetriableError(message)
Retry after a given delay (0 to 24 hours) throw new RetryAfterError(message, delayMs) return workflows.RetryAfter(err, d) raise RetryAfterError(message, delay_ms)

Use RetryAfter when the system you call tells you when to come back, such as an HTTP 429 with Retry-After. In TypeScript and Python, either error also works as another error’s cause.

Timeouts

A step can have a timeout, at most 24 hours: { timeoutMs } in TypeScript, WithStepTimeout in Go, timeout_ms= in Python, per step or as a workflow’s default. An attempt that runs too long fails with workflows.step_timeout, which is retried like any other failure. The step’s context is cancelled at the timeout: in TypeScript, pass its signal to fetch; in Go, use the StepContext.

A step can’t outlast its invocation. On a push app that’s your maxInvocationSeconds, so longer steps need a pull worker, whose invocations last up to the longest workflow timeout its workers announce, at most 24 hours.

Idempotency

Steps run at least once. If your process dies after a step ran but before its result was recorded, the step runs again. Make the calls inside steps idempotent where it matters: inside a step, the idempotency key <runId>:<stepId>:<attempt> identifies the attempt. Pass it to APIs that deduplicate requests, such as payment APIs.

await step.run("charge", ({ idempotencyKey }) => payments.charge(order, { idempotencyKey }))
_, err := workflows.Run(ctx, "charge", func(sc workflows.StepContext) (string, error) {
	return payments.Charge(sc, order, sc.IdempotencyKey())
})
await ctx.step.run("charge", lambda: payments.charge(order, idempotency_key=ctx.step.idempotency_key()))

Attempt numbers only grow, even when a run is retried, so a key is never issued twice.

Idempotent API calls

  • Every POST takes an Idempotency-Key header: a repeated request with the same key answers the first one’s stored result, with its status (such as 201) and the header Idempotent-Replayed: true. The SDK clients retry a 429, a 503 or a network error with the same key, except when creating an app or rotating its secret: a replay of those carries no secret.
  • A run key makes a start idempotent by meaning: starting a run with a runKey that exists answers the existing run (200) when the workflow and input are the same, and 409 run_key_conflict when they aren’t.
  • A request repeated while the first is still in progress answers 409 idempotency_in_progress; retry it after a moment.
Navigation

Type to search…

↑↓ navigate↵ selectEsc close