Attributes read
A span is a GenAI span when it has gen_ai.operation.name. AI Center reads these attributes from
it:
| Attribute | Used as |
|---|---|
gen_ai.operation.name |
The operation: chat, text_completion, generate_content and embeddings are model calls, execute_tool tool calls and invoke_agent agent invocations |
gen_ai.provider.name |
The provider |
gen_ai.response.model, else gen_ai.request.model |
The model |
gen_ai.agent.name |
The agent |
gen_ai.tool.name |
The tool |
gen_ai.tool.call.id |
Matches a tool call span to the tool call in the messages |
gen_ai.conversation.id |
The conversation |
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |
Input and output tokens |
gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens |
Cache read and cache write tokens |
gen_ai.response.finish_reasons |
The finish reasons |
error.type, and the span’s status |
An error: status ERROR or an error.type |
The service and environment come from the span’s resource, as for every span: service.name and
the dataset’s environment.
Content
| Attribute | Content |
|---|---|
gen_ai.system_instructions |
System instructions |
gen_ai.input.messages |
Input messages |
gen_ai.output.messages |
Output messages |
gen_ai.tool.definitions |
Tool definitions |
gen_ai.tool.call.arguments |
A tool call’s arguments |
gen_ai.tool.call.result |
A tool call’s result |
Content is read from the span’s attributes, and from the attributes of a
gen_ai.client.inference.operation.details span event; when both have the same attribute, the
span’s wins. Send each content attribute as a string: the JSON text the conventions describe, or any other
text. A string is kept as sent.
Content is stored apart from its span. In Traces, a GenAI span has no
content attributes, and its operation details events keep their other attributes. The span has
ezgh.genai.content_bytes instead: the size of its content as sent, in bytes. Observability sets
it; a value you send is replaced.
Text only
AI Center keeps text. In system instructions and input and output messages, a part whose type
is text, tool_call, tool_call_response or reasoning is kept as sent. Any other part, such
as an image, audio or a file (blob, file, uri), is replaced by a placeholder with its type,
MIME type (null when the part has none) and the size of its data as sent, in bytes (for a
blob, the length of its base64 text):
{ "type": "blob", "mime_type": "image/png", "bytes": 48213 }Limits
| Limit | Value |
|---|---|
| Content per span | 1 MiB, after media parts are replaced |
| Span size | 64 KiB, with its resource and scope, not counting its content |
A GenAI span’s content doesn’t count toward the span size limit; every other trace limit applies as usual, request sizes included.
Past 1 MiB, content is cut: fields are kept in the order of the table above, the field
that reaches the limit is cut at a character boundary (its JSON is then incomplete), and later
fields are left out. The span is stored, and its step in the run has truncated set, with the
size as sent in contentBytes.
Sampled datasets
In a dataset with trace sampling, content and runs are stored for kept traces only: a dropped trace’s run isn’t in AI Center. The generated metrics count every GenAI span received, kept or dropped, as span metrics do.
Generated metrics
Observability generates these series from GenAI spans, in each environment’s dataset, and reads them with PromQL like span metrics. They’re free: not counted in ingested or stored bytes, a dataset’s active series, or ingest rate limits. They power the overview, and work in dashboards and alert rules.
| Series | Type | Counts | Labels |
|---|---|---|---|
ezgh_genai_tokens_total |
Counter | Tokens in model calls | type (input, output, cache_read, cache_write), operation, provider, model, agent |
ezgh_genai_calls_total |
Counter | Model calls | status, operation, provider, model, agent |
ezgh_genai_call_duration_seconds_bucket, _sum, _count |
Histogram, in seconds | Model call durations | operation, provider, model, agent |
ezgh_genai_tool_calls_total |
Counter | Tool calls | status, tool, agent |
ezgh_genai_tool_duration_seconds_bucket, _sum, _count |
Histogram, in seconds | Tool call durations | tool, agent |
ezgh_genai_agent_invocations_total |
Counter | Agent invocations | status, agent |
ezgh_genai_agent_duration_seconds_bucket, _sum, _count |
Histogram, in seconds | Agent invocation durations | agent |
statusisokorerror.agentis the span’s owngen_ai.agent.name, absent when the span has none. Instrumentations that set it only oninvoke_agentspans have tool and model call series withoutagent.- Every series also has
job(the service key),environmentandapplication, as span metrics do. Histograms havele. - A
provider,model,agentortoolvalue over 128 bytes, or with control characters, is__other__.
| Histogram | Bucket bounds, in seconds |
|---|---|
| Model calls | 0.25, 0.5, 1, 2, 4, 8, 15, 30, 60, 120, +Inf |
| Tool calls | 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, +Inf |
| Agent invocations | 1, 2, 5, 10, 20, 30, 60, 120, 300, 600, +Inf |
| Cap | Value | Past it |
|---|---|---|
| Models, agents, tools | 200 distinct values each per environment, each hour | New values in that hour are counted as __other__ |
| Generated series | About 20,000 per environment, apart from span metrics’ | New series aren’t written |
The series are written and counted as span metrics are; see
Accuracy. Your own gen_ai.client.* metrics, if you send
them, are stored as sent.
# Tokens per hour by agent and type
sum by (agent, type) (increase(ezgh_genai_tokens_total{environment="production"}[1h]))
# Cache hit rate: cache read tokens over input tokens
sum(increase(ezgh_genai_tokens_total{environment="production", type="cache_read"}[1h]))
/ sum(increase(ezgh_genai_tokens_total{environment="production", type="input"}[1h]))
# p95 model call latency by model
histogram_quantile(0.95, sum by (model, le) (rate(ezgh_genai_call_duration_seconds_bucket{environment="production"}[5m])))
# Tool error ratio by tool
sum by (tool) (rate(ezgh_genai_tool_calls_total{environment="production", status="error"}[5m]))
/ sum by (tool) (rate(ezgh_genai_tool_calls_total{environment="production"}[5m]))Retention
Runs and their content follow the dataset’s traces: they’re kept for its retention class, in hot and cold storage, and leave with their spans. There’s no separate retention setting. The generated metrics follow the dataset’s metrics retention.
Billing
GenAI spans are traces. A GenAI span is metered as traces ingested bytes at its size as sent, content included, and its content’s storage is counted in traces stored bytes; see Pricing. AI Center has no meter of its own, and the generated metrics are free.
Permissions
| To | Needs |
|---|---|
| See the overview, and pick filter values on Runs and Conversations | obs.metrics.query |
| List and search runs and conversations | obs.traces.search |
| Read a run’s steps, and its waterfall | obs.traces.getSpans |
| Read a run’s content: prompts, completions, tool arguments and results | obs.genai.readContent |
Search content (text, --text, the Runs search box) |
obs.traces.search and obs.genai.readContent |
All are checked against the dataset. ReadOnlyAccess, ObsReadOnlyAccess and ObsFullAccess
include the first three. obs.genai.readContent has access level Write and is only in
ObsFullAccess; give it to others with a custom policy. Without it, the console shows a run’s
steps without their text, disables the Runs search box and names the permission. See
Policies.
Reads aren’t recorded in Trails, as for other trace reads.