Pull workers

A pull app’s runs are executed by workers you run anywhere with outbound HTTPS: a server behind a firewall, a batch machine, your laptop. A worker long-polls EZGH for tasks with an API key, so it needs no public endpoint, and its steps can run for hours.

Task queues and partitions

A pull app has one task queue, named when you create it (default default, matching ^[a-z0-9][a-z0-9-]{0,62}$), and spread over partitions (default 4, 1 to 32). The queue can’t change. Partitions can grow later with UpdateApp but never shrink. Raise them for a queue with many tasks.

curl -X POST "https://workflows-api.ezghcloud.com/v1/organizations/$EZGH_ORG_ID/projects/$EZGH_PROJECT_ID/apps" \
  -H "Authorization: Bearer $EZGH_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "reports", "kind": "pull", "queue": "default", "partitions": 8}'

Run a worker

import { worker } from "@ezgh/workflows"

const w = worker({ apiKey, organizationId, projectId, app: appId, workflows: [resize], concurrency: 10 })
process.on("SIGTERM", () => void w.stop())
await w.run()

w, err := workflows.NewWorker(workflows.WorkerOptions{
	APIKey: apiKey, OrganizationID: orgID, ProjectID: projectID, App: appID,
	Workflows: []workflows.Workflow{resize}, MaxConcurrent: 10,
})
err = w.Run(ctx) // until ctx is done; then it lets running invocations finish

from ezgh_workflows import Worker

worker = Worker(
    api_key=api_key, organization_id=org_id, project_id=project_id, app=app_id,
    workflows=[resize], concurrency=10,
)
worker.run_sync()  # or: await worker.run(); worker.stop() from anywhere

• Polling: a worker holds one poll at a time, for up to 55 seconds, asking for at most 10 tasks and never more than it has room for. It polls again at once after an answer, and backs off on 429, 5xx and network errors after any Retry-After.
• Concurrency: each SDK runs up to 10 invocations at once by default (concurrency, MaxConcurrent).
• Heartbeats: while an invocation runs, the worker sends a heartbeat every 20 seconds by default. An invocation without a heartbeat for 60 seconds is given to another worker.
• Stopping: stopping a worker ends its poll at once and gives running invocations a grace period (30 seconds by default) to finish. Anything still running is given to another worker after its heartbeat timeout.
• A refused poll stops the worker: a 401 or 403 (the API key or its permissions), or another 4xx (the app or its queue).

Run as many workers as you like on the same queue. Each task goes to one worker.

Announcing workflows

Each poll announces the workflows the worker serves. The app’s workflows are those its workers announced in the last 10 minutes, so:

• A run of a pull app can start only for a workflow some worker announced in the last 10 minutes. Otherwise starting it is 409 workflow_not_announced.
• An announcement that changes the app’s workflows is recorded in Trails as AnnounceWorkflows.
• A worker serves at most 1,000 workflows.

Long steps

A pull invocation’s deadline is the longest timeout of the app’s announced workflows, at least 15 minutes and at most 24 hours. For a step longer than 15 minutes, give its workflow a timeout that covers it: workflow(name, fn, { timeoutMs }) in TypeScript, workflows.WithTimeout in Go, and @workflow(name, timeout_ms=…) in Python. Heartbeats keep the invocation alive meanwhile.

The worker’s API key

A worker needs workflows.workers.poll on the app, and workflows.apps.sync to announce its workflows: the WorkflowsWorkerAccess managed policy. Give the worker an API key for a bot with only that policy. Polls are checked on every request; only refused polls are recorded in Trails.

Local development

Each developer can run workflows on their own machine through a pull app of their own, with no tunnel. The SDKs’ dev mode uses a pull app named dev-<user> with the queue dev-<user>, from your operating system user name, so developers sharing a project never take each other’s runs:

Creating the app needs workflows.apps.create.