Skip to content

Data reference

The gen_ai.* attributes AI Center reads, the content limit and media placeholders, the generated ezgh_genai_* metrics, retention, billing and permissions.

Updated View as Markdown

Attributes read

A span is a GenAI span when it has gen_ai.operation.name. AI Center reads these attributes from it:

Attribute Used as
gen_ai.operation.name The operation: chat, text_completion, generate_content and embeddings are model calls, execute_tool tool calls and invoke_agent agent invocations
gen_ai.provider.name The provider
gen_ai.response.model, else gen_ai.request.model The model
gen_ai.agent.name The agent
gen_ai.tool.name The tool
gen_ai.tool.call.id Matches a tool call span to the tool call in the messages
gen_ai.conversation.id The conversation
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens Input and output tokens
gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens Cache read and cache write tokens
gen_ai.response.finish_reasons The finish reasons
error.type, and the span’s status An error: status ERROR or an error.type

The service and environment come from the span’s resource, as for every span: service.name and the dataset’s environment.

Content

Attribute Content
gen_ai.system_instructions System instructions
gen_ai.input.messages Input messages
gen_ai.output.messages Output messages
gen_ai.tool.definitions Tool definitions
gen_ai.tool.call.arguments A tool call’s arguments
gen_ai.tool.call.result A tool call’s result

Content is read from the span’s attributes, and from the attributes of a gen_ai.client.inference.operation.details span event; when both have the same attribute, the span’s wins. Send each content attribute as a string: the JSON text the conventions describe, or any other text. A string is kept as sent.

Content is stored apart from its span. In Traces, a GenAI span has no content attributes, and its operation details events keep their other attributes. The span has ezgh.genai.content_bytes instead: the size of its content as sent, in bytes. Observability sets it; a value you send is replaced.

Text only

AI Center keeps text. In system instructions and input and output messages, a part whose type is text, tool_call, tool_call_response or reasoning is kept as sent. Any other part, such as an image, audio or a file (blob, file, uri), is replaced by a placeholder with its type, MIME type (null when the part has none) and the size of its data as sent, in bytes (for a blob, the length of its base64 text):

{ "type": "blob", "mime_type": "image/png", "bytes": 48213 }

Limits

Limit Value
Content per span 1 MiB, after media parts are replaced
Span size 64 KiB, with its resource and scope, not counting its content

A GenAI span’s content doesn’t count toward the span size limit; every other trace limit applies as usual, request sizes included.

Past 1 MiB, content is cut: fields are kept in the order of the table above, the field that reaches the limit is cut at a character boundary (its JSON is then incomplete), and later fields are left out. The span is stored, and its step in the run has truncated set, with the size as sent in contentBytes.

Sampled datasets

In a dataset with trace sampling, content and runs are stored for kept traces only: a dropped trace’s run isn’t in AI Center. The generated metrics count every GenAI span received, kept or dropped, as span metrics do.

Generated metrics

Observability generates these series from GenAI spans, in each environment’s dataset, and reads them with PromQL like span metrics. They’re free: not counted in ingested or stored bytes, a dataset’s active series, or ingest rate limits. They power the overview, and work in dashboards and alert rules.

Series Type Counts Labels
ezgh_genai_tokens_total Counter Tokens in model calls type (input, output, cache_read, cache_write), operation, provider, model, agent
ezgh_genai_calls_total Counter Model calls status, operation, provider, model, agent
ezgh_genai_call_duration_seconds_bucket, _sum, _count Histogram, in seconds Model call durations operation, provider, model, agent
ezgh_genai_tool_calls_total Counter Tool calls status, tool, agent
ezgh_genai_tool_duration_seconds_bucket, _sum, _count Histogram, in seconds Tool call durations tool, agent
ezgh_genai_agent_invocations_total Counter Agent invocations status, agent
ezgh_genai_agent_duration_seconds_bucket, _sum, _count Histogram, in seconds Agent invocation durations agent
  • status is ok or error.
  • agent is the span’s own gen_ai.agent.name, absent when the span has none. Instrumentations that set it only on invoke_agent spans have tool and model call series without agent.
  • Every series also has job (the service key), environment and application, as span metrics do. Histograms have le.
  • A provider, model, agent or tool value over 128 bytes, or with control characters, is __other__.
Histogram Bucket bounds, in seconds
Model calls 0.25, 0.5, 1, 2, 4, 8, 15, 30, 60, 120, +Inf
Tool calls 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, +Inf
Agent invocations 1, 2, 5, 10, 20, 30, 60, 120, 300, 600, +Inf
Cap Value Past it
Models, agents, tools 200 distinct values each per environment, each hour New values in that hour are counted as __other__
Generated series About 20,000 per environment, apart from span metrics’ New series aren’t written

The series are written and counted as span metrics are; see Accuracy. Your own gen_ai.client.* metrics, if you send them, are stored as sent.

# Tokens per hour by agent and type
sum by (agent, type) (increase(ezgh_genai_tokens_total{environment="production"}[1h]))

# Cache hit rate: cache read tokens over input tokens
sum(increase(ezgh_genai_tokens_total{environment="production", type="cache_read"}[1h]))
  / sum(increase(ezgh_genai_tokens_total{environment="production", type="input"}[1h]))

# p95 model call latency by model
histogram_quantile(0.95, sum by (model, le) (rate(ezgh_genai_call_duration_seconds_bucket{environment="production"}[5m])))

# Tool error ratio by tool
sum by (tool) (rate(ezgh_genai_tool_calls_total{environment="production", status="error"}[5m]))
  / sum by (tool) (rate(ezgh_genai_tool_calls_total{environment="production"}[5m]))

Retention

Runs and their content follow the dataset’s traces: they’re kept for its retention class, in hot and cold storage, and leave with their spans. There’s no separate retention setting. The generated metrics follow the dataset’s metrics retention.

Billing

GenAI spans are traces. A GenAI span is metered as traces ingested bytes at its size as sent, content included, and its content’s storage is counted in traces stored bytes; see Pricing. AI Center has no meter of its own, and the generated metrics are free.

Permissions

To Needs
See the overview, and pick filter values on Runs and Conversations obs.metrics.query
List and search runs and conversations obs.traces.search
Read a run’s steps, and its waterfall obs.traces.getSpans
Read a run’s content: prompts, completions, tool arguments and results obs.genai.readContent
Search content (text, --text, the Runs search box) obs.traces.search and obs.genai.readContent

All are checked against the dataset. ReadOnlyAccess, ObsReadOnlyAccess and ObsFullAccess include the first three. obs.genai.readContent has access level Write and is only in ObsFullAccess; give it to others with a custom policy. Without it, the console shows a run’s steps without their text, disables the Runs search box and names the permission. See Policies.

Reads aren’t recorded in Trails, as for other trace reads.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close