---
title: "AI Center"
description: "Your AI agents' tokens, cache hit rate, model calls and tool calls, and each run read as a transcript, built from the OpenTelemetry GenAI traces they send."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Center

AI Center reads the [OpenTelemetry GenAI](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
traces your agents send to Observability and answers two kinds of question: how your agents are
doing across all their runs, and what one agent saw, said and did in a single run. It needs
nothing beyond sending the traces; see [Send data](/observability/ai-center/sending-data).

- **Overview:** agent invocations, tokens (input, output, cache read and cache write), cache hit
  rate, tool calls per invocation and their failure rate, and p95 model-call latency, with tokens
  over time, the model mix, and tables of agents and tools. Filter by agent, model and tool. Every
  number opens the runs behind it.
- **Runs:** every run in a time range, with its agent, models, model and tool calls, tokens,
  duration and status. Filter by agent, model, tool and status, and sort by start time, tokens,
  duration or errors. The CLI and API also filter by provider, service, environment and
  conversation.
- **A run:** a transcript of the system, user and assistant messages, with each model call and tool
  call inline: its timing, tokens, finish reason, error, arguments and result. **Waterfall**
  shows the run as a trace instead, and the run links to its full trace and its conversation.
- **Conversations:** runs that share a `gen_ai.conversation.id`, in order, with tokens and errors
  per turn.
- **Search:** full text across prompts, completions, system instructions, tool arguments and tool
  results, together with the filters. Each matching run shows the first match.

- [Send data](/observability/ai-center/sending-data) — OpenTelemetry GenAI traces with content capture on, and Mastra apps.
- [Data reference](/observability/ai-center/data-reference) — The attributes read, content limits, generated metrics, retention, billing and permissions.

## Concepts

| Term | Meaning |
| --- | --- |
| GenAI span | A span with a `gen_ai.operation.name` attribute |
| Model call | A GenAI span whose operation is `chat`, `text_completion`, `generate_content` or `embeddings` |
| Tool call | A GenAI span whose operation is `execute_tool` |
| Agent invocation | A GenAI span whose operation is `invoke_agent` |
| Run | One trace with at least one GenAI span. Its agent is the outermost `invoke_agent` span's `gen_ai.agent.name`, else the earliest GenAI span's |
| Conversation | The runs in a dataset that share a `gen_ai.conversation.id`, oldest first. A run without one isn't in a conversation |
| Content | System instructions, input and output messages, tool definitions, tool call arguments and tool call results |

A filter matches a run when any of its GenAI spans has the value, so a sub-agent's name finds the
runs it took part in.

## In the console

Open **Observability**, then **AI Center**: **AI overview**, **Runs** and **Conversations**. Each
page reads one environment's dataset over the time range you choose, and both are in the URL.

## From the CLI

```sh
# Runs in the last day where the support agent was asked about a refund
ezgh obs ai runs search --dataset production --agent support --text refund --from 1d

# One run's steps, with its prompts and completions
ezgh obs ai runs get 5b8efff798038103d269b633813fc60c --dataset production --content

# Conversations in the last week
ezgh obs ai conversations search --dataset production --from 7d
```

See [`ezgh obs ai`](/cli/reference/obs-ai) for every flag.

## From the API

Each call is scoped to a dataset, under
`/v1/organizations/:orgId/projects/:projectId/obs/datasets/:datasetId/genai/`:

- [Search runs](/obs-api/SearchGenAIRuns/) that started between `from` and `to`, with
  `filters`, `sort`, `order`, and `text` to search content.
- [Search conversations](/obs-api/SearchGenAIConversations/), with the same filters.
- [Get a run](/obs-api/GetGenAIRun/): its totals and every GenAI span's metadata, its steps,
  oldest first, without content.
- [Get a run's content](/obs-api/GetGenAIRunContent/): each GenAI span's content, as sent.

`limit` is 1 to 100, 25 by default. To read the next page of a search, send only `cursor` (the
previous page's `nextCursor`) and `limit`. Without `from` and `to`, a run's reads cover spans
that started in the last 7 days; the range can span at most 31 days. A search over the read budget
is refused with `422 query_too_expensive`; narrow the time range or add filters.

## Access

Run lists, run steps and the overview need the trace and metric read actions, which
`ReadOnlyAccess` includes. Reading prompts and completions, and searching them, also needs
`obs.genai.readContent`, which only `ObsFullAccess` and custom policies grant. See
[Permissions](/observability/ai-center/data-reference#permissions).

Source: https://docs.ezghcloud.com/observability/ai-center/index.mdx
