Skip to content

AI Center

Your AI agents' tokens, cache hit rate, model calls and tool calls, and each run read as a transcript, built from the OpenTelemetry GenAI traces they send.

Updated View as Markdown

AI Center reads the OpenTelemetry GenAI traces your agents send to Observability and answers two kinds of question: how your agents are doing across all their runs, and what one agent saw, said and did in a single run. It needs nothing beyond sending the traces; see Send data.

  • Overview: agent invocations, tokens (input, output, cache read and cache write), cache hit rate, tool calls per invocation and their failure rate, and p95 model-call latency, with tokens over time, the model mix, and tables of agents and tools. Filter by agent, model and tool. Every number opens the runs behind it.
  • Runs: every run in a time range, with its agent, models, model and tool calls, tokens, duration and status. Filter by agent, model, tool and status, and sort by start time, tokens, duration or errors. The CLI and API also filter by provider, service, environment and conversation.
  • A run: a transcript of the system, user and assistant messages, with each model call and tool call inline: its timing, tokens, finish reason, error, arguments and result. Waterfall shows the run as a trace instead, and the run links to its full trace and its conversation.
  • Conversations: runs that share a gen_ai.conversation.id, in order, with tokens and errors per turn.
  • Search: full text across prompts, completions, system instructions, tool arguments and tool results, together with the filters. Each matching run shows the first match.

Concepts

Term Meaning
GenAI span A span with a gen_ai.operation.name attribute
Model call A GenAI span whose operation is chat, text_completion, generate_content or embeddings
Tool call A GenAI span whose operation is execute_tool
Agent invocation A GenAI span whose operation is invoke_agent
Run One trace with at least one GenAI span. Its agent is the outermost invoke_agent span’s gen_ai.agent.name, else the earliest GenAI span’s
Conversation The runs in a dataset that share a gen_ai.conversation.id, oldest first. A run without one isn’t in a conversation
Content System instructions, input and output messages, tool definitions, tool call arguments and tool call results

A filter matches a run when any of its GenAI spans has the value, so a sub-agent’s name finds the runs it took part in.

In the console

Open Observability, then AI Center: AI overview, Runs and Conversations. Each page reads one environment’s dataset over the time range you choose, and both are in the URL.

From the CLI

# Runs in the last day where the support agent was asked about a refund
ezgh obs ai runs search --dataset production --agent support --text refund --from 1d

# One run's steps, with its prompts and completions
ezgh obs ai runs get 5b8efff798038103d269b633813fc60c --dataset production --content

# Conversations in the last week
ezgh obs ai conversations search --dataset production --from 7d

See ezgh obs ai for every flag.

From the API

Each call is scoped to a dataset, under /v1/organizations/:orgId/projects/:projectId/obs/datasets/:datasetId/genai/:

  • Search runs that started between from and to, with filters, sort, order, and text to search content.
  • Search conversations, with the same filters.
  • Get a run: its totals and every GenAI span’s metadata, its steps, oldest first, without content.
  • Get a run’s content: each GenAI span’s content, as sent.

limit is 1 to 100, 25 by default. To read the next page of a search, send only cursor (the previous page’s nextCursor) and limit. Without from and to, a run’s reads cover spans that started in the last 7 days; the range can span at most 31 days. A search over the read budget is refused with 422 query_too_expensive; narrow the time range or add filters.

Access

Run lists, run steps and the overview need the trace and metric read actions, which ReadOnlyAccess includes. Reading prompts and completions, and searching them, also needs obs.genai.readContent, which only ObsFullAccess and custom policies grant. See Permissions.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close