AI Center reads the OpenTelemetry GenAI traces your agents send to Observability and answers two kinds of question: how your agents are doing across all their runs, and what one agent saw, said and did in a single run. It needs nothing beyond sending the traces; see Send data.
- Overview: agent invocations, tokens (input, output, cache read and cache write), cache hit rate, tool calls per invocation and their failure rate, and p95 model-call latency, with tokens over time, the model mix, and tables of agents and tools. Filter by agent, model and tool. Every number opens the runs behind it.
- Runs: every run in a time range, with its agent, models, model and tool calls, tokens, duration and status. Filter by agent, model, tool and status, and sort by start time, tokens, duration or errors. The CLI and API also filter by provider, service, environment and conversation.
- A run: a transcript of the system, user and assistant messages, with each model call and tool call inline: its timing, tokens, finish reason, error, arguments and result. Waterfall shows the run as a trace instead, and the run links to its full trace and its conversation.
- Conversations: runs that share a
gen_ai.conversation.id, in order, with tokens and errors per turn. - Search: full text across prompts, completions, system instructions, tool arguments and tool results, together with the filters. Each matching run shows the first match.
Send data
OpenTelemetry GenAI traces with content capture on, and Mastra apps.
Data reference
The attributes read, content limits, generated metrics, retention, billing and permissions.
Concepts
| Term | Meaning |
|---|---|
| GenAI span | A span with a gen_ai.operation.name attribute |
| Model call | A GenAI span whose operation is chat, text_completion, generate_content or embeddings |
| Tool call | A GenAI span whose operation is execute_tool |
| Agent invocation | A GenAI span whose operation is invoke_agent |
| Run | One trace with at least one GenAI span. Its agent is the outermost invoke_agent span’s gen_ai.agent.name, else the earliest GenAI span’s |
| Conversation | The runs in a dataset that share a gen_ai.conversation.id, oldest first. A run without one isn’t in a conversation |
| Content | System instructions, input and output messages, tool definitions, tool call arguments and tool call results |
A filter matches a run when any of its GenAI spans has the value, so a sub-agent’s name finds the runs it took part in.
In the console
Open Observability, then AI Center: AI overview, Runs and Conversations. Each page reads one environment’s dataset over the time range you choose, and both are in the URL.
From the CLI
# Runs in the last day where the support agent was asked about a refund
ezgh obs ai runs search --dataset production --agent support --text refund --from 1d
# One run's steps, with its prompts and completions
ezgh obs ai runs get 5b8efff798038103d269b633813fc60c --dataset production --content
# Conversations in the last week
ezgh obs ai conversations search --dataset production --from 7dSee ezgh obs ai for every flag.
From the API
Each call is scoped to a dataset, under
/v1/organizations/:orgId/projects/:projectId/obs/datasets/:datasetId/genai/:
- Search runs that started between
fromandto, withfilters,sort,order, andtextto search content. - Search conversations, with the same filters.
- Get a run: its totals and every GenAI span’s metadata, its steps, oldest first, without content.
- Get a run’s content: each GenAI span’s content, as sent.
limit is 1 to 100, 25 by default. To read the next page of a search, send only cursor (the
previous page’s nextCursor) and limit. Without from and to, a run’s reads cover spans
that started in the last 7 days; the range can span at most 31 days. A search over the read budget
is refused with 422 query_too_expensive; narrow the time range or add filters.
Access
Run lists, run steps and the overview need the trace and metric read actions, which
ReadOnlyAccess includes. Reading prompts and completions, and searching them, also needs
obs.genai.readContent, which only ObsFullAccess and custom policies grant. See
Permissions.