Transform rules

Transform rules change your logs, spans and metric points as they arrive, before they’re stored or billed. They’re written in the OpenTelemetry Transformation Language (OTTL), the language of the OpenTelemetry Collector’s transform and filter processors, and set on a dataset with a section for each signal.

• Drop what isn’t worth storing: debug logs, health-check spans, metrics you don’t use.
• Redact: delete, hash or mask attributes and text in log bodies (emails, tokens, card numbers) before they’re written anywhere.
• Reshape: set, rename, copy and delete attributes; set severity; rename metrics.
• Parse log bodies: JSON, key=value and regular expressions into attributes.

What to expect:

• Nothing changes where you send from. SDKs, Collectors and the EZGH Obs Agent send as they do now.
• Rules aren’t retroactive. They apply to records that arrive after you save them. Records already stored keep what they were stored with, including values a new rule would redact, until their retention ends.
• Only stored data is billed. Dropped records and removed attributes aren’t counted in the ingested or stored bytes.
• Rate limits count bytes received. Your organization’s and dataset’s rates are charged before rules run, so dropping records doesn’t free up rate.
• Off by default. A dataset without rules, or with rules turned off, stores records as they arrive.

Each dataset (one environment of a project) has its own rules, so production can redact while staging keeps everything.

A rule document

{
  "enabled": true,
  "logs": {
    "statements": [
      "replace_pattern(log.body, \"\\\\b\\\\d{13,16}\\\\b\", \"****\")",
      "merge_maps(log.attributes, ParseJSON(log.body), \"upsert\") where IsMatch(log.body, \"^\\\\{\")",
      "set(log.severity_number, SEVERITY_NUMBER_WARN) where log.attributes[\"level\"] == \"warn\"",
      "set(log.attributes[\"user.id\"], SHA256(log.attributes[\"user.id\"])) where log.attributes[\"user.id\"] != nil",
      "delete_key(log.attributes, \"user.email\")"
    ],
    "drop": [
      "log.severity_number < SEVERITY_NUMBER_INFO",
      "IsMatch(log.body, \"GET /healthz\")"
    ]
  },
  "traces": {
    "statements": ["delete_key(span.attributes, \"http.request.header.authorization\")"],
    "drop": ["span.name == \"GET /healthz\""]
  },
  "metrics": {
    "statements": ["set(metric.name, \"http.server.request.duration\") where metric.name == \"http_server_duration\""],
    "drop": ["IsMatch(metric.name, \"^go_gc_.*\")"]
  }
}

For logs, this masks 13- to 16-digit numbers in the body, then parses JSON bodies into attributes, sets the severity of records whose level is warn, hashes the user ID and deletes the user’s email, then drops records below INFO and health-check lines. For traces it deletes the Authorization header attribute and drops health-check spans; for metrics it renames a metric and drops the Go garbage collector’s.

Each statement and condition is a JSON string, so its quotes and backslashes are escaped once more in the document. The console’s editor, and the recipes below, show them as plain OTTL, one per line: the first statement above is replace_pattern(log.body, "\\b\\d{13,16}\\b", "****").

How rules run

Contexts

Every path starts with its context, the kind of element it reads or changes:

Each statement and each drop condition runs in the context its own paths need: a statement with span. paths runs once for every span, one with only resource. paths once for every resource. A statement can read its element’s parents, so this one runs for every log record:

set(log.attributes["region"], resource.attributes["cloud.region"])

Statements, then drop conditions

• Statements run first, in the order you write them, then the drop conditions run on the result. A drop condition can use a field a statement parsed out of the body, or a severity a statement set.
• Order matters: a statement sees what earlier statements did. See Redact before you parse.
• A drop condition removes the element of its context when it holds: a resource (with all its records), a scope, a log record, a span (with its events and links), a span event, a metric (with all its points) or a data point. A scope, resource or metric left with nothing is removed too. A record is dropped when any condition holds.

When a rule fails

• A statement that fails is skipped, and the record goes on through the rest of the rules and is stored. For example, merge_maps(log.attributes, ParseJSON(log.body), "upsert") on a body that’s plain text adds nothing; the record is stored as it was. A where clause that checks the value first (where IsMatch(log.body, "^\\{")) keeps that from happening.
• A drop condition that fails counts as false, so an error never drops a record.
• Some functions do nothing rather than fail. replace_pattern on a body that isn’t a string (a map, for example) leaves it unchanged.
• So a redaction can miss. If a statement that redacts a value fails on a record, or doesn’t match it, the value is stored as sent. Try your rules on real samples with Test before you rely on them, and prefer rules that can’t fail, such as delete_key.
• Rules that can’t run don’t let records through. If a dataset’s saved rules stop compiling (only possible when Observability upgrades OTTL), its records are rejected in partial_success with the reason transform rules can't run; check them with GET …/transforms, while records for other datasets in the same request are stored. This lasts until you fix the rules or turn them off. Testing or saving the rules shows which ones don’t compile. Turning them off always saves, because rules that are off aren’t compiled: the dataset’s records are then stored as sent, without any of the rules.
• Rules can’t grow records without limit. If a dataset’s rules add more than the growth limit to a request (for example by repeating a value into itself), they stop, and that dataset’s records in the request are rejected in partial_success with the reason transform rules made the records too large. They aren’t stored, since their rules didn’t finish. Other datasets in the same request are stored. Test shows the same error.

What rules can’t change

After a resource’s rules run, its ezgh.metadata.environment attribute is set back to what was sent (or removed again, if it wasn’t sent). A rule can’t move records to another dataset, whichever function touched the attribute. Every other attribute can change, including service.name, service.namespace and ezgh.metadata.application.

After the rules, invalid UTF-8 in strings is replaced with U+FFFD, and a value nested more than 1,000 levels deep (from ParseJSON, for example) is stored as its JSON text.

After the rules

Everything after the rules sees the transformed records: record limits, the event-time window, trace sampling, storage, search, PromQL, span metrics and the service catalog.

• A record a rule dropped is accepted, not rejected: it isn’t listed in partial_success.
• A record a rule pushed over a limit is rejected in partial_success with its usual reason, as if it had been sent that way: a map with more than 256 attributes after merge_maps, for example, or an event time a rule moved outside the dataset’s window.

Recipes

Each recipe shows a signal’s statements or drop conditions as plain OTTL, one per line, as the console’s editor shows them.

Drop debug logs

log.severity_number > SEVERITY_NUMBER_UNSPECIFIED and log.severity_number < SEVERITY_NUMBER_INFO

Records sent without a severity have SEVERITY_NUMBER_UNSPECIFIED (0), so the condition log.severity_number < SEVERITY_NUMBER_INFO alone drops them too.

Drop health-check spans

span.name == "GET /healthz"
IsMatch(span.attributes["url.path"], "^/(healthz|readyz|livez)$")

A dropped span’s events and links go with it. Its child spans are kept, without their parent.

Delete and mask secrets

delete_key(log.attributes, "user.email")
delete_matching_keys(log.attributes, "(?i).*(password|secret|token).*")
replace_pattern(log.body, "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>")
replace_all_patterns(log.attributes, "value", "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "<email>")

delete_key(span.attributes, "http.request.header.authorization")
replace_pattern(span.attributes["url.query"], "token=[^&]*", "token=****")

Redact before you parse

Mask the body before you parse it into attributes. Parsing copies the value into an attribute, so a mask applied to the body afterward leaves the copy unmasked.

merge_maps(log.attributes, ParseJSON(log.body), "upsert") where IsMatch(log.body, "^\\{")
replace_pattern(log.body, "\\b\\d{13,16}\\b", "****")

replace_pattern(log.body, "\\b\\d{13,16}\\b", "****")
merge_maps(log.attributes, ParseJSON(log.body), "upsert") where IsMatch(log.body, "^\\{")

For the body {"msg":"order placed","card":"4111111111111111"}:

Hash a user ID

set(log.attributes["user.id"], SHA256(String(log.attributes["user.id"]))) where log.attributes["user.id"] != nil

The same ID always gets the same hash, so you can still count and search by user. String makes a numeric ID a string first: without it, SHA256 fails on a number and the ID is stored as sent. A hash of a value that’s easy to guess, such as a short number or a known email, can be matched by hashing candidates; delete those instead.

Parse a JSON body

merge_maps(log.attributes, ParseJSON(log.body), "upsert") where IsMatch(log.body, "^\\s*\\{")
merge_maps(log.attributes, log.body, "upsert") where IsMap(log.body)

The first parses a body that’s JSON text; the second copies a body that was sent as a map. "upsert" overwrites attributes of the same name; "insert" keeps the ones already there. A JSON object nested in the body becomes a map attribute.

Parse key=value

merge_maps(log.attributes, ParseKeyValue(log.body), "upsert") where IsMatch(log.body, "^\\w+=")

The body level=warn user=u_1842 msg="cache miss" gets the attributes level (warn), user (u_1842) and msg (cache miss).

Parse with a regular expression

merge_maps(log.attributes, ExtractPatterns(log.body, "^(?P<method>[A-Z]+) (?P<path>\\S+) (?P<status>\\d{3})"), "upsert")

The body GET /api/cart 404 gets method, path and status. Each named group becomes an attribute. Regular expressions use RE2 syntax.

Set severity from a parsed field

After a statement that parses the body:

set(log.severity_number, SEVERITY_NUMBER_ERROR) where log.attributes["level"] == "error"
set(log.severity_number, SEVERITY_NUMBER_WARN) where log.attributes["level"] == "warn"
set(log.severity_number, SEVERITY_NUMBER_INFO) where log.attributes["level"] == "info"
set(log.severity_number, SEVERITY_NUMBER_DEBUG) where log.attributes["level"] == "debug"
set(log.severity_text, ToUpperCase(log.attributes["level"])) where log.attributes["level"] != nil

Drop conditions run after these, so a debug-log condition then drops the records parsed as debug.

Rename a span attribute

set(span.attributes["http.request.method"], span.attributes["http.method"]) where span.attributes["http.method"] != nil
delete_key(span.attributes, "http.method")

Drop a metric by name

metric.name == "go_gc_duration_seconds"
IsMatch(metric.name, "^process\\.runtime\\.")

Rename a metric

set(metric.name, "http.server.request.duration") where metric.name == "http_server_duration"

Points stored under the old name stay under it. Don’t rename a metric onto one you already send with the same attributes: their points would merge.

Metrics: don’t remove identifying labels

Rules see metric names and attributes as sent, not as PromQL reads them: http.server.request.duration and http.response.status_code, not http_server_request_duration_seconds and http_response_status_code. A remote-write series’ name is metric.name, and its labels are its points’ datapoint.attributes, job and instance included. job and instance are also on its resource, as service.name (and service.namespace, for a job of namespace/name) and service.instance.id; ezgh_environment and ezgh_application are only on the resource, as the ezgh.metadata.* attributes.

A point’s attributes and resource tell its series apart. Removing or overwriting an attribute that tells live series apart isn’t supported. If only a point’s k8s.pod.name attribute tells two pods’ series apart, deleting it stores both pods’ points as one series: queries can’t tell them apart, and the two pods’ cumulative counters interleave, so rate() and increase() are wrong, not approximate.

• Rules don’t stop you: they can’t tell an identifying attribute from one that’s the same on every series. Deleting an attribute that never tells series apart is safe.
• Hashing a value (SHA256) keeps series apart. Replacing different values with one fixed string, such as ****, doesn’t.
• To reduce cardinality, aggregate before sending, in your SDK’s views or your Collector. Rules can’t aggregate.
• Test a metrics section on sample points: it reports points from different series that the rules put in the same series at the same time.

With sampling, span metrics and the service catalog

• Rules run before trace sampling. Sampling rules see the transformed spans, and spans a rule drops never reach sampling.
• Spans a rule drops aren’t counted in span metrics, the service map or the service catalog’s stats. Spans sampling drops still are.
• Changing service.name or service.namespace changes the service’s key: the catalog shows the new service, and the old one ages out as its data does.
• Prometheus remote write is transformed like OTLP metrics. A point a rule drops is counted as written in the X-Prometheus-Remote-Write-Samples-Written, …-Histograms-Written and …-Exemplars-Written headers, so senders don’t see a partial write.

Copying rules from a Collector

Rules use the Collector’s OTTL with its standard functions (editors such as set, delete_key and replace_pattern, and converters such as ParseJSON, SHA256 and IsMatch), from OpenTelemetry Collector Contrib v0.162.0.

• Write paths with their context prefix, as the transform processor’s flat configuration does: log.body, not body. A transform processor’s log_statements, trace_statements and metric_statements go in statements; a filter processor’s conditions go in drop.
• Contexts are per statement here. The Collector infers one context for a whole list of statements, the most specific one its paths need. A list with a span. statement and a spanevent. statement runs once for every span event there, so spans without events never see the span. statement, a redaction included. Here each statement runs in its own context, so the span. statement reaches every span. A list whose statements all share one context behaves the same in both.
• Errors are always ignored, as with the processors’ error_mode: ignore.
• The transform processor’s metric-only functions, such as aggregate_on_attributes, convert_sum_to_gauge and extract_count_metric, aren’t available.
• flatten, GetXML, InsertXML and ExtractGrokPatterns’ pattern_definitions aren’t available, and a few arguments are limited so that one rule can’t make a value far larger than it was: see function limits.

In the console

Open Observability > Datasets, a dataset, then the Transforms tab.

• Add rules starts a document. Each signal has a card with two editors, Statements and Drop, one rule per line. Lines are numbered from 1, and rules that don’t compile are marked on their lines when you save.
• The Enabled switch turns the rules on or off. Turning them off asks you to confirm; the rules stay saved.
• Save saves the whole document. If someone else saved the rules since you opened them, saving says so and shows theirs, and your edits aren’t saved.
• Without obs.datasets.update, the tab shows the rules read-only.

Test on a card runs that signal’s rules, as edited and unsaved, on records you paste: an OTLP/JSON export request, prefilled with a sample you can edit. It shows how many records went in and were dropped and how many times a rule failed, then each record before and after, with dropped records struck through and the statements that failed on a record. For metrics, it warns when the rules merge series. Nothing is saved or stored, and the test runs whether or not the rules are enabled.

From the CLI and the API

ezgh obs datasets transforms get production --project web
ezgh obs datasets transforms get production -o json --project web > rules.json
ezgh obs datasets transforms test production -f rules.json --records sample-logs.json --signal logs --project web
ezgh obs datasets transforms set production -f rules.json --project web

Limits

Function limits

So that one statement can’t make a value far larger than it was, a rule that breaks one of these doesn’t compile: PUT and the test answer 400 with an issue at its path.

• Not available: flatten, GetXML and InsertXML, whose output can grow far beyond their input, and ExtractGrokPatterns’ pattern_definitions. Write the expression in the grok pattern itself instead, as (?P<name>…).
• Replacements: in replace_pattern, replace_all_patterns, replace_match and replace_all_matches, the replacement and replacement_format are strings written in the rule, at most 256 bytes together. The replacement has at most 8 capture references ($1, ${name}) and the format one %s. function is one of FNV, MD5, Murmur3Hash, Murmur3Hash128, SHA1, SHA256, SHA512, ToLowerCase, ToUpperCase, XXH128 and XXH3.
• Lists: Concat, Format and append take at most 16 values in a list. Any list or map written in a rule holds at most 16 fields of the record, such as log.body.
• Per-element text: Concat’s delimiter is a string of at most 256 bytes written in the rule. ToKeyValueString’s delimiters and ConvertTextToElementsXML’s element name are at most 256 bytes.
• Format: at most 16 verbs, whose widths and precisions add up to at most 256, and no *.
• Captures: ExtractPatterns and ExtractGrokPatterns take a pattern written in the rule, with at most 16 named groups (and, for grok, patterns such as %{WORD:name}) written in it.
• One enlarging function per rule: a statement or condition calls at most one of Base64Encode, Concat, ConvertTextToElementsXML, Decode, ExtractGrokPatterns, ExtractPatterns, Format, Hex, String and ToKeyValueString. To combine them, store the first result in an attribute in one statement and use it in the next.

Access and Trails

ObsFullAccess includes both; ObsReadOnlyAccess and ReadOnlyAccess include obs.datasets.get.

Changing the rules is recorded in Trails as UpdateDatasetTransforms, with eventSource obs.ezghcloud.com. The event records whether the rules are enabled, each signal’s number of statements and drop conditions, and the new version, never the rules themselves, because their text can contain your data and the very values you redact. Viewing and testing rules isn’t recorded.