---
title: "Transform rules"
description: "Drop, redact, reshape and parse logs, spans and metric points as they arrive, before they're stored or billed, with OTTL rules set on a dataset."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Transform rules

Transform rules change your logs, spans and metric points **as they arrive, before they're stored
or billed**. They're written in the
[OpenTelemetry Transformation Language](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl)
(OTTL), the language of the OpenTelemetry Collector's `transform` and `filter` processors, and set
on a dataset with a section for each signal.

- **Drop** what isn't worth storing: debug logs, health-check spans, metrics you don't use.
- **Redact:** delete, hash or mask attributes and text in log bodies (emails, tokens, card
  numbers) before they're written anywhere.
- **Reshape:** set, rename, copy and delete attributes; set severity; rename metrics.
- **Parse log bodies:** JSON, `key=value` and regular expressions into attributes.

What to expect:

- **Nothing changes where you send from.** SDKs, Collectors and the EZGH Obs Agent send as they
  do now.
- **Rules aren't retroactive.** They apply to records that arrive after you save them. Records
  already stored keep what they were stored with, including values a new rule would redact, until
  their retention ends.
- **Only stored data is billed.** Dropped records and removed attributes aren't counted in the
  ingested or stored bytes.
- **Rate limits count bytes received.** Your organization's and dataset's
  [rates](/observability/limits#rates) are charged before rules run, so dropping records doesn't
  free up rate.
- **Off by default.** A dataset without rules, or with rules turned off, stores records as they
  arrive.

Each dataset (one environment of a project) has its own rules, so `production` can redact while
`staging` keeps everything.

## A rule document

```json
{
  "enabled": true,
  "logs": {
    "statements": [
      "replace_pattern(log.body, \"\\\\b\\\\d{13,16}\\\\b\", \"****\")",
      "merge_maps(log.attributes, ParseJSON(log.body), \"upsert\") where IsMatch(log.body, \"^\\\\{\")",
      "set(log.severity_number, SEVERITY_NUMBER_WARN) where log.attributes[\"level\"] == \"warn\"",
      "set(log.attributes[\"user.id\"], SHA256(log.attributes[\"user.id\"])) where log.attributes[\"user.id\"] != nil",
      "delete_key(log.attributes, \"user.email\")"
    ],
    "drop": [
      "log.severity_number < SEVERITY_NUMBER_INFO",
      "IsMatch(log.body, \"GET /healthz\")"
    ]
  },
  "traces": {
    "statements": ["delete_key(span.attributes, \"http.request.header.authorization\")"],
    "drop": ["span.name == \"GET /healthz\""]
  },
  "metrics": {
    "statements": ["set(metric.name, \"http.server.request.duration\") where metric.name == \"http_server_duration\""],
    "drop": ["IsMatch(metric.name, \"^go_gc_.*\")"]
  }
}
```

For logs, this masks 13- to 16-digit numbers in the body, then parses JSON bodies into attributes,
sets the severity of records whose `level` is `warn`, hashes the user ID and deletes the user's
email, then drops records below `INFO` and health-check lines. For traces it deletes the
`Authorization` header attribute and drops health-check spans; for metrics it renames a metric and
drops the Go garbage collector's.

| Field | Value |
| --- | --- |
| `enabled` | `true` to run the rules, `false` to keep them saved without running them. Required |
| `logs`, `traces`, `metrics` | A section for each signal, each optional |
| `statements` | OTTL statements that change records, run in order. Up to 50 |
| `drop` | OTTL conditions. A record is dropped when any of them holds. Up to 50 |

Each statement and condition is a JSON string, so its quotes and backslashes are escaped once more
in the document. The console's editor, and the recipes below, show them as plain OTTL, one per
line: the first statement above is `replace_pattern(log.body, "\\b\\d{13,16}\\b", "****")`.

## How rules run

### Contexts

Every path starts with its **context**, the kind of element it reads or changes:

| Prefix | Is | In |
| --- | --- | --- |
| `resource.` | A resource: `resource.attributes` | Every signal |
| `scope.` | An instrumentation scope: `scope.name`, `scope.version`, `scope.attributes` | Every signal |
| `log.` | A log record: `log.body`, `log.attributes`, `log.severity_number`, `log.severity_text`, `log.time` | Logs |
| `span.` | A span: `span.name`, `span.attributes`, `span.kind`, `span.status` | Traces |
| `spanevent.` | A span event: `spanevent.name`, `spanevent.attributes` | Traces |
| `metric.` | A metric: `metric.name`, `metric.unit`, `metric.description` | Metrics |
| `datapoint.` | A metric's data point: `datapoint.attributes` | Metrics |

Each statement and each drop condition runs in the context its own paths need: a statement with
`span.` paths runs once for every span, one with only `resource.` paths once for every resource. A
statement can read its element's parents, so this one runs for every log record:

```txt
set(log.attributes["region"], resource.attributes["cloud.region"])
```

### Statements, then drop conditions

- **Statements run first, in the order you write them**, then the drop conditions run on the
  result. A drop condition can use a field a statement parsed out of the body, or a severity a
  statement set.
- **Order matters**: a statement sees what earlier statements did. See
  [Redact before you parse](#redact-before-you-parse).
- **A drop condition removes the element of its context** when it holds: a resource (with all
  its records), a scope, a log record, a span (with its events and links), a span event, a metric
  (with all its points) or a data point. A scope, resource or metric left with nothing is removed
  too. A record is dropped when **any** condition holds.

### When a rule fails

- **A statement that fails is skipped**, and the record goes on through the rest of the rules and
  is stored. For example, `merge_maps(log.attributes, ParseJSON(log.body), "upsert")` on a body
  that's plain text adds nothing; the record is stored as it was. A `where` clause that checks the
  value first (`where IsMatch(log.body, "^\\{")`) keeps that from happening.
- **A drop condition that fails counts as false**, so an error never drops a record.
- **Some functions do nothing rather than fail.** `replace_pattern` on a body that isn't a string
  (a map, for example) leaves it unchanged.
- **So a redaction can miss.** If a statement that redacts a value fails on a record, or doesn't
  match it, the value is stored as sent. Try your rules on real samples with
  [Test](#in-the-console) before you rely on them, and prefer rules that can't fail, such as
  `delete_key`.
- **Rules that can't run don't let records through.** If a dataset's saved rules stop compiling
  (only possible when Observability upgrades OTTL), its records are rejected in
  `partial_success` with the reason `transform rules can't run; check them with GET …/transforms`,
  while records for other datasets in the same request are stored. This lasts until you fix the
  rules or turn them off. Testing or saving the rules shows which ones don't compile. Turning them
  off always saves, because rules that are off aren't compiled: the dataset's records are then
  stored as sent, without any of the rules.
- **Rules can't grow records without limit.** If a dataset's rules add more than the
  [growth limit](#limits) to a request (for example by repeating a value into itself), they stop,
  and that dataset's records in the request are rejected in `partial_success` with the reason
  `transform rules made the records too large`. They aren't stored, since their rules didn't
  finish. Other datasets in the same request are stored. Test shows the same error.

### What rules can't change

After a resource's rules run, its `ezgh.metadata.environment` attribute is set back to what was
sent (or removed again, if it wasn't sent). A rule can't move records to another dataset, whichever
function touched the attribute. Every other attribute can change, including `service.name`,
`service.namespace` and `ezgh.metadata.application`.

After the rules, invalid UTF-8 in strings is replaced with U+FFFD, and a value nested more than
1,000 levels deep (from `ParseJSON`, for example) is stored as its JSON text.

### After the rules

Everything after the rules sees the transformed records: record limits, the event-time window,
[trace sampling](/observability/sampling), storage, search, PromQL, span metrics and the service
catalog.

- **A record a rule dropped is accepted**, not rejected: it isn't listed in `partial_success`.
- **A record a rule pushed over a limit is rejected** in `partial_success` with its usual reason,
  as if it had been sent that way: a map with more than 256 attributes after `merge_maps`, for
  example, or an event time a rule moved outside the dataset's window.

## Recipes

Each recipe shows a signal's statements or drop conditions as plain OTTL, one per line, as the
console's editor shows them.

### Drop debug logs

```txt title="logs · drop"
log.severity_number > SEVERITY_NUMBER_UNSPECIFIED and log.severity_number < SEVERITY_NUMBER_INFO
```

Records sent without a severity have `SEVERITY_NUMBER_UNSPECIFIED` (0), so the condition
`log.severity_number < SEVERITY_NUMBER_INFO` alone drops them too.

### Drop health-check spans

```txt title="traces · drop"
span.name == "GET /healthz"
IsMatch(span.attributes["url.path"], "^/(healthz|readyz|livez)$")
```

A dropped span's events and links go with it. Its child spans are kept, without their parent.

### Delete and mask secrets

```txt title="logs · statements"
delete_key(log.attributes, "user.email")
delete_matching_keys(log.attributes, "(?i).*(password|secret|token).*")
replace_pattern(log.body, "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>")
replace_all_patterns(log.attributes, "value", "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>")
```

```txt title="traces · statements"
delete_key(span.attributes, "http.request.header.authorization")
replace_pattern(span.attributes["url.query"], "token=[^&]*", "token=****")
```

### Redact before you parse

Mask the body **before** you parse it into attributes. Parsing copies the value into an
attribute, so a mask applied to the body afterward leaves the copy unmasked.

```txt title="logs · statements · wrong order"
merge_maps(log.attributes, ParseJSON(log.body), "upsert") where IsMatch(log.body, "^\\{")
replace_pattern(log.body, "\\b\\d{13,16}\\b", "****")
```

```txt title="logs · statements · right order"
replace_pattern(log.body, "\\b\\d{13,16}\\b", "****")
merge_maps(log.attributes, ParseJSON(log.body), "upsert") where IsMatch(log.body, "^\\{")
```

For the body `{"msg":"order placed","card":"4111111111111111"}`:

| | Body stored | `card` attribute stored |
| --- | --- | --- |
| Wrong order | `{"msg":"order placed","card":"****"}` | `4111111111111111` |
| Right order | `{"msg":"order placed","card":"****"}` | `****` |

### Hash a user ID

```txt title="logs · statements"
set(log.attributes["user.id"], SHA256(String(log.attributes["user.id"]))) where log.attributes["user.id"] != nil
```

The same ID always gets the same hash, so you can still count and search by user. `String` makes
a numeric ID a string first: without it, `SHA256` fails on a number and the ID is stored as sent.
A hash of a value that's easy to guess, such as a short number or a known email, can be matched by
hashing candidates; delete those instead.

### Parse a JSON body

```txt title="logs · statements"
merge_maps(log.attributes, ParseJSON(log.body), "upsert") where IsMatch(log.body, "^\\s*\\{")
merge_maps(log.attributes, log.body, "upsert") where IsMap(log.body)
```

The first parses a body that's JSON text; the second copies a body that was sent as a map.
`"upsert"` overwrites attributes of the same name; `"insert"` keeps the ones already there. A JSON
object nested in the body becomes a map attribute.

### Parse key=value

```txt title="logs · statements"
merge_maps(log.attributes, ParseKeyValue(log.body), "upsert") where IsMatch(log.body, "^\\w+=")
```

The body `level=warn user=u_1842 msg="cache miss"` gets the attributes `level` (`warn`), `user`
(`u_1842`) and `msg` (`cache miss`).

### Parse with a regular expression

```txt title="logs · statements"
merge_maps(log.attributes, ExtractPatterns(log.body, "^(?P<method>[A-Z]+) (?P<path>\\S+) (?P<status>\\d{3})"), "upsert")
```

The body `GET /api/cart 404` gets `method`, `path` and `status`. Each named group becomes an
attribute. Regular expressions use [RE2 syntax](https://github.com/google/re2/wiki/Syntax).

### Set severity from a parsed field

After a statement that parses the body:

```txt title="logs · statements"
set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.attributes["level"] == "error"
set(log.severity_number, SEVERITY_NUMBER_WARN) where log.attributes["level"] == "warn"
set(log.severity_number, SEVERITY_NUMBER_INFO) where log.attributes["level"] == "info"
set(log.severity_number, SEVERITY_NUMBER_DEBUG) where log.attributes["level"] == "debug"
set(log.severity_text, ToUpperCase(log.attributes["level"])) where log.attributes["level"] != nil
```

Drop conditions run after these, so a debug-log condition then drops the records parsed as
`debug`.

### Rename a span attribute

```txt title="traces · statements"
set(span.attributes["http.request.method"], span.attributes["http.method"]) where span.attributes["http.method"] != nil
delete_key(span.attributes, "http.method")
```

### Drop a metric by name

```txt title="metrics · drop"
metric.name == "go_gc_duration_seconds"
IsMatch(metric.name, "^process\\.runtime\\.")
```

### Rename a metric

```txt title="metrics · statements"
set(metric.name, "http.server.request.duration") where metric.name == "http_server_duration"
```

Points stored under the old name stay under it. Don't rename a metric onto one you already send
with the same attributes: their points would [merge](#metrics-dont-remove-identifying-labels).

## Metrics: don't remove identifying labels

Rules see metric names and attributes **as sent**, not as PromQL reads them:
`http.server.request.duration` and `http.response.status_code`, not
`http_server_request_duration_seconds` and `http_response_status_code`. A remote-write series'
name is `metric.name`, and its labels are its points' `datapoint.attributes`, `job` and `instance`
included. `job` and `instance` are also on its resource, as `service.name` (and
`service.namespace`, for a `job` of `namespace/name`) and `service.instance.id`;
`ezgh_environment` and `ezgh_application` are only on the resource, as the `ezgh.metadata.*`
attributes.

A point's attributes and resource tell its series apart. **Removing or overwriting an attribute
that tells live series apart isn't supported.** If only a point's `k8s.pod.name` attribute tells
two pods' series apart, deleting it stores both pods' points as one series: queries can't tell
them apart, and the two pods' cumulative counters interleave, so `rate()` and `increase()` are
wrong, not approximate.

- Rules don't stop you: they can't tell an identifying attribute from one that's the same on
  every series. Deleting an attribute that never tells series apart is safe.
- **Hashing** a value (`SHA256`) keeps series apart. Replacing different values with one fixed
  string, such as `****`, doesn't.
- **To reduce cardinality, aggregate before sending**, in your SDK's views or your Collector.
  Rules can't aggregate.
- **[Test](#in-the-console)** a metrics section on sample points: it reports points from different
  series that the rules put in the same series at the same time.

## With sampling, span metrics and the service catalog

- **Rules run before [trace sampling](/observability/sampling).** Sampling rules see the
  transformed spans, and spans a rule drops never reach sampling.
- **Spans a rule drops aren't counted** in [span metrics](/observability/span-metrics), the
  [service map](/observability/service-map) or the [service catalog](/observability/service-catalog)'s
  stats. Spans sampling drops still are.
- **Changing `service.name` or `service.namespace`** changes the service's
  [key](/observability/service-catalog#how-services-are-named): the catalog shows the new service,
  and the old one ages out as its data does.
- **Prometheus remote write** is transformed like OTLP metrics. A point a rule drops is counted
  as written in the `X-Prometheus-Remote-Write-Samples-Written`, `…-Histograms-Written` and
  `…-Exemplars-Written` headers, so senders don't see a partial write.

## Copying rules from a Collector

Rules use the Collector's OTTL with its standard functions (editors such as `set`,
`delete_key` and `replace_pattern`, and converters such as `ParseJSON`, `SHA256` and `IsMatch`),
from OpenTelemetry Collector Contrib v0.162.0.

- **Write paths with their context prefix**, as the `transform` processor's flat configuration
  does: `log.body`, not `body`. A `transform` processor's `log_statements`, `trace_statements` and
  `metric_statements` go in `statements`; a `filter` processor's conditions go in `drop`.
- **Contexts are per statement here.** The Collector infers one context for a whole list of
  statements, the most specific one its paths need. A list with a `span.` statement and a
  `spanevent.` statement runs once for every span event there, so **spans without events never
  see the `span.` statement**, a redaction included. Here each statement runs in its own context,
  so the `span.` statement reaches every span. A list whose statements all share one context
  behaves the same in both.
- **Errors are always ignored**, as with the processors' `error_mode: ignore`.
- The `transform` processor's metric-only functions, such as `aggregate_on_attributes`,
  `convert_sum_to_gauge` and `extract_count_metric`, aren't available.
- `flatten`, `GetXML`, `InsertXML` and `ExtractGrokPatterns`' `pattern_definitions` aren't
  available, and a few arguments are limited so that one rule can't make a value far larger than
  it was: see [function limits](#function-limits).

## In the console

Open **Observability** > **Datasets**, a dataset, then the **Transforms** tab.

- **Add rules** starts a document. Each signal has a card with two editors, **Statements** and
  **Drop**, one rule per line. Lines are numbered from 1, and rules that don't compile are marked
  on their lines when you save.
- The **Enabled** switch turns the rules on or off. Turning them off asks you to confirm; the
  rules stay saved.
- **Save** saves the whole document. If someone else saved the rules since you opened them,
  saving says so and shows theirs, and your edits aren't saved.
- Without `obs.datasets.update`, the tab shows the rules read-only.

**Test** on a card runs that signal's rules, as edited and unsaved, on records you paste: an
OTLP/JSON export request, prefilled with a sample you can edit. It shows how many records went in and
were dropped and how many times a rule failed, then each record before and after, with dropped records struck
through and the statements that failed on a record. For metrics, it warns when the rules merge
series. Nothing is saved or stored, and the test runs whether or not the rules are enabled.

## From the CLI and the API

### CLI

```sh
ezgh obs datasets transforms get production --project web
ezgh obs datasets transforms get production -o json --project web > rules.json
ezgh obs datasets transforms test production -f rules.json --records sample-logs.json --signal logs --project web
ezgh obs datasets transforms set production -f rules.json --project web
```

- `get` shows whether the rules run, the version, and each signal's statements and drop
conditions, numbered from 0 under the paths errors use (`logs.statements.0`). With `-o json` it
prints the document with its version; edit the file and pass it to `test` and `set`.
- `test` runs one signal's rules from a file on the records in `--records`, an OTLP/JSON export
request, without saving anything. It shows how many records were dropped and where, the rules
that failed on a record and, for metrics, series the rules merge. `--output-records` writes the
transformed records to a file.
- `set` replaces the whole document from a file (`-f -` reads standard input). The file can be a
document or the output of `get -o json`. If someone changed the rules since they were read,
nothing changes and `set` says so; `--force` overwrites them anyway.

The dataset is given by environment, ID, `dst_` slug or name. See
[`ezgh obs datasets transforms`](/cli/reference/obs-datasets-transforms).
### API

A dataset's rules are a sub-resource of the dataset, in the project's Observability API:

| Operation | Request |
| --- | --- |
| [GetDatasetTransforms](/obs-api/GetDatasetTransforms/) | `GET /v1/organizations/<organization ID>/projects/<project ID>/obs/datasets/<dataset ID>/transforms` |
| [UpdateDatasetTransforms](/obs-api/UpdateDatasetTransforms/) | `PUT …/obs/datasets/<dataset ID>/transforms` |
| [TestDatasetTransforms](/obs-api/TestDatasetTransforms/) | `POST …/obs/datasets/<dataset ID>/transforms/test` |

- `GET` returns `{ "document": {…}, "version": 3, "updatedAt": "…" }` with an `ETag`. A dataset
that never had rules returns `{ "document": { "enabled": false }, "version": 0 }`, with the
dataset's creation time as `updatedAt`.
- `PUT` takes a document and replaces the whole document; `version` goes up by one. `If-Match` is
optional: send the ETag from `GET` so a change saved in the meantime answers
`412 precondition_failed` instead of being overwritten. It takes an `Idempotency-Key`.
- A document outside the [bounds](#limits), or an enabled one with a rule that doesn't compile, is
`400 invalid_request`, with every problem in `issues` as `{ "path", "message" }`. `path` is
dotted with numeric indices from 0, such as `logs.statements.3`, and `message` is OTTL's error.
A document with `"enabled": false` is checked for shape and bounds only, so rules that stopped
compiling can always be turned off.
- `PUT` on a dataset being deleted is `409 dataset_deleting`.
- The test takes `{ "signal": "logs", "document": {…}, "records": { "resourceLogs": […] } }`, at
most 1 MiB in all. `document` is a whole document, checked as `PUT` checks it, and `records`
is an OTLP/JSON export request of the signal, at most 256 KiB as sent. Only `signal`'s section runs, on every resource
in `records` whatever its environment. The answer is `output` (the transformed request, in the
input's order without what was dropped), `dropped` (locations), `errors` (locations with the
failing rule's `path` and `message`), `collisions` (metrics) and `stats`. A location is
`{ "resource", "scope", "record", "item" }`, numbered from 0 in the input: `record` is a log
record, span or metric, and `item` a span event or data point. `dropped` and `errors` hold at
most their first 1,000 entries; `stats` counts every one.

## Limits

| Limit | Value |
| --- | --- |
| Statements | 50 per signal |
| Drop conditions | 50 per signal |
| A statement or condition | 4 KiB, not blank |
| Document | 64 KiB |
| Test request | 1 MiB, rules and records together; the records at most 256 KiB |
| Functions | See [function limits](#function-limits) |
| Growth | The rules may add at most 1 MiB or twice the size of the records they run on, whichever is larger, to a request |
| Test answer | The first 1,000 dropped locations and the first 1,000 errors; `stats` counts them all |

### Function limits

So that one statement can't make a value far larger than it was, a rule that breaks one of these
doesn't compile: `PUT` and the test answer `400` with an issue at its path.

- **Not available:** `flatten`, `GetXML` and `InsertXML`, whose output can grow far beyond their
  input, and `ExtractGrokPatterns`' `pattern_definitions`. Write the expression in the grok
  pattern itself instead, as `(?P<name>…)`.
- **Replacements:** in `replace_pattern`, `replace_all_patterns`, `replace_match` and
  `replace_all_matches`, the replacement and `replacement_format` are strings written in the rule,
  at most 256 bytes together. The replacement has at most 8 capture references (`$1`, `${name}`)
  and the format one `%s`. `function` is one of `FNV`, `MD5`, `Murmur3Hash`, `Murmur3Hash128`,
  `SHA1`, `SHA256`, `SHA512`, `ToLowerCase`, `ToUpperCase`, `XXH128` and `XXH3`.
- **Lists:** `Concat`, `Format` and `append` take at most 16 values in a list. Any list or map
  written in a rule holds at most 16 fields of the record, such as `log.body`.
- **Per-element text:** `Concat`'s delimiter is a string of at most 256 bytes written in the rule.
  `ToKeyValueString`'s delimiters and `ConvertTextToElementsXML`'s element name are at most 256
  bytes.
- **`Format`:** at most 16 verbs, whose widths and precisions add up to at most 256, and no `*`.
- **Captures:** `ExtractPatterns` and `ExtractGrokPatterns` take a pattern written in the rule,
  with at most 16 named groups (and, for grok, patterns such as `%{WORD:name}`) written in it.
- **One enlarging function per rule:** a statement or condition calls at most one of
  `Base64Encode`, `Concat`, `ConvertTextToElementsXML`, `Decode`, `ExtractGrokPatterns`,
  `ExtractPatterns`, `Format`, `Hex`, `String` and `ToKeyValueString`. To combine them, store the
  first result in an attribute in one statement and use it in the next.

## Access and Trails

| Action | Allows |
| --- | --- |
| `obs.datasets.get` | Viewing a dataset's rules, and testing rules |
| `obs.datasets.update` | Changing a dataset's rules |

`ObsFullAccess` includes both; `ObsReadOnlyAccess` and `ReadOnlyAccess` include
`obs.datasets.get`.

Changing the rules is recorded in [Trails](/trails) as `UpdateDatasetTransforms`, with
`eventSource` `obs.ezghcloud.com`. The event records whether the rules are enabled, each signal's
number of statements and drop conditions, and the new version, never the rules themselves,
because their text can contain your data and the very values you redact. Viewing and testing rules
isn't recorded.

Source: https://docs.ezghcloud.com/observability/transforms/index.mdx