Dashboards

A dashboard is a saved grid of panels in a project. Each panel charts or lists logs, traces or metrics, and one dashboard can mix all three. Dashboards are resources like datasets: each has an ID, a dsh_ slug, a resource name, tags and its own URL, and its last 50 saved versions are kept.

A dashboard holds queries, not data. Every panel is read as the person viewing the dashboard, with their own access, each time the dashboard loads or refreshes.

Build a dashboard

For clusters that run EZGH Obs Agent, the Kubernetes section’s pages are built in; Duplicate as dashboard copies one into a dashboard of your own to edit.

You can also start from what you’re looking at: on the Logs, Traces and Metrics pages, Add to dashboard turns the current query into a panel on a new or existing dashboard. On a dashboard, a panel’s Explore opens its query in the matching page, with the same time range and environment.

Edits stay unsaved until you select Save. If someone else saved the dashboard since you opened it, saving asks whether to reload their version or overwrite it with yours.

Panels

All of a panel’s queries (up to 4) use one signal. Logs and traces panels use the same filter syntax as the Logs and Traces pages, such as level:error service:checkout or attr.http.route:/cart.

Logs and traces measures are count, rate (per second), countDistinct (approximate), sum, avg, min, max, p50, p90, p95 and p99. Traces add errors and errorRate. A measure of a field reads a numeric attribute, or a span’s duration; values that aren’t numbers are skipped. Group by severityText, serviceName, scopeName or application (and name, statusCode or kind for spans), or by an attribute such as attr.http.route. A panel shows up to 50 groups (10 by default); the rest add up into one other group.

Panels can set a unit, decimals, minimum and maximum, a legend (list, table or hidden), and, for stat and table panels, thresholds that color values as success, warning or danger. A panel can also use its own relative time range, such as the last hour on a 7-day dashboard.

• Value mappings (stat and table panels) show numbers as words: up to 20 entries, each a value, the text to show (up to 64 characters) and optionally a status (success, warning or danger) that colors it. A pod phase of 2 can show as a green Running. A mapped value takes the mapping’s status instead of the thresholds’. In a table, mappings apply to the first value column.
• No data text (every panel) replaces “No data in this range” with your own text, up to 200 characters, such as how to start sending what the panel needs.
• Row links (table and top list panels) make each row open a dashboard; see Row links.

A panel that can’t be read shows why, and the rest of the dashboard still loads: a query over its budget (422) asks you to narrow the time range or filters, as the explorers do, and a panel you don’t have access to names the action you need.

Variables

Variables are pickers at the top of the dashboard, used in queries as $name. A dashboard has up to 10.

application, values and custom variables can allow several values and offer All. In PromQL, several values become a regular expression (a|b) and All becomes .*, so use them with =~: {service_name=~"$service"}. In logs and traces filters, several values match any of them.

Logs and traces panels read one environment at a time; with All environments selected they ask you to pick one. Metrics panels can read all environments at once.

A values variable’s match can use other variables, so one variable can depend on another: a pod variable matched on k8s_pod_cpu_usage{k8s_namespace_name="$namespace"} offers only the pods in the selected namespace. When a variable changes, those that depend on it keep the values that are still offered; one left with none goes back to its default, or to All or the first value when the default isn’t offered. Variables can’t depend on each other in a loop.

PromQL queries can also use $__interval (the panel’s step), $__rate_interval (a range for rate() suited to the step) and $__range (the time range).

Row links

A table or top list panel can open another dashboard from each row, with that row’s values set as the other dashboard’s variables. The link carries the time range and the environment, and this dashboard’s other variables, such as cluster; the row’s values win.

"viz": {
  "type": "table",
  "rowLink": {
    "dashboard": "builtin:kubernetes-pod",
    "vars": { "namespace": "{{k8s_namespace_name}}", "pod": "{{k8s_pod_name}}" }
  }
}

• dashboard is a dashboard’s id (a UUID, not its dsh_ slug), or builtin: and the ID of one of the Kubernetes section’s pages: builtin:kubernetes-cluster, builtin:kubernetes-node, builtin:kubernetes-workload, builtin:kubernetes-pod or builtin:kubernetes-agent, which open that page for the row’s cluster, namespace, node, workload or pod. Rows that don’t name a page aren’t links. A dashboard that doesn’t exist or that the viewer can’t see opens as it does from its own URL.
• vars is required, and sets up to 10 of the target’s variables ({} sets none). Values use the legend’s {{label}} syntax, filled from the row’s labels (metrics) or group values (logs and traces), up to 256 characters. A row missing one of the labels isn’t a link.

Time range and refresh

A dashboard has a default time range, relative (such as the last 6 hours) or absolute, and an auto-refresh interval: off (the default), 30 seconds, or 1, 5 or 15 minutes, or 1 hour. Auto-refresh pauses while the browser tab is hidden. Dragging across a time series chart zooms the whole dashboard to that range.

Share a dashboard

A dashboard’s URL holds what you’re looking at: the time range, the environment and each variable’s values. Copy it from the address bar or with Copy link, and anyone with access to the dashboard opens the same view. Dashboards aren’t shared outside your organization: there are no public links.

Version history

Creating a dashboard saves version 1. Each save that changes its name, description or panels saves the next version; a change of tags alone doesn’t. The last 50 versions are kept, with who saved each one and when, so you can see which queries a dashboard ran at any time.

To restore a version, open the dashboard’s version history and select Restore, or apply the version’s spec from the CLI. Restoring saves it as a new version; history isn’t rewritten.

Manage dashboards as code

A dashboard is a JSON document: the file the CLI applies is the JSON the console shows in View JSON. Keep dashboard files in Git next to the services they describe, and apply them from CI.

{
  "name": "Checkout service",
  "description": "Requests and errors of the checkout service",
  "tags": { "team": "payments" },
  "spec": {
    "schemaVersion": 1,
    "time": { "range": "6h" },
    "refresh": "off",
    "variables": [
      { "name": "environment", "kind": "environment", "default": "production" },
      {
        "name": "service",
        "kind": "values",
        "label": "Service",
        "source": { "signal": "metrics", "label": "service_name", "match": "http_server_request_duration_seconds_count" },
        "multi": true,
        "includeAll": true
      }
    ],
    "panels": [
      {
        "id": "p1",
        "title": "Request rate by service",
        "grid": { "x": 0, "y": 0, "w": 12, "h": 8 },
        "viz": { "type": "timeseries", "style": "lines", "unit": "reqps", "legend": "table" },
        "signal": "metrics",
        "queries": [
          {
            "ref": "A",
            "promql": "sum by (service_name) (rate(http_server_request_duration_seconds_count{service_name=~\"$service\"}[$__rate_interval]))",
            "legend": "{{service_name}}"
          }
        ]
      },
      {
        "id": "p2",
        "title": "Errors by route",
        "grid": { "x": 12, "y": 0, "w": 12, "h": 8 },
        "viz": { "type": "timeseries", "style": "bars", "stack": true },
        "signal": "logs",
        "queries": [
          {
            "ref": "A",
            "filter": "level:error service:$service",
            "aggregate": { "fn": "count" },
            "groupBy": [{ "field": "attr.http.route" }],
            "limit": 10
          }
        ]
      },
      {
        "id": "p3",
        "title": "Recent errors",
        "grid": { "x": 0, "y": 8, "w": 24, "h": 10 },
        "viz": { "type": "logs", "columns": ["time", "level", "service", "body"] },
        "signal": "logs",
        "queries": [{ "ref": "A", "filter": "level:error service:$service" }]
      }
    ]
  }
}

• name (1 to 100 characters) and spec are required; description (up to 1,000 characters) and tags (an object of strings) are optional.
• In spec, the grid has 24 columns: x and w count columns, and y and h count 30-pixel rows. Panel ids are 1 to 16 lowercase letters and digits, unique in the dashboard; query refs are A to D.
• The newer optional fields viz.mappings, viz.noData and viz.rowLink keep schemaVersion 1. Consoles from before they existed ignore them.
• Unknown fields are refused with 400 naming the field. Queries aren’t checked when you save: a query with a mistake fails when its panel runs.

ezgh obs dashboards apply -f checkout.json --project web
ezgh obs dashboards list --project web
ezgh obs dashboards get "Checkout service" -o json > checkout.json
ezgh obs dashboards versions "Checkout service"
ezgh obs dashboards versions get "Checkout service" 3 -o json
ezgh obs dashboards delete "Checkout service"

An update or delete sent with an ETag that’s no longer current is refused with 412 precondition_failed, and nothing changes: someone saved the dashboard after you read it. Read it again, or run apply again to apply your file over the current version.

Access

ObsFullAccess includes all five. ObsReadOnlyAccess and ReadOnlyAccess include list and get.

A dashboard grants nothing. Each panel is read with the viewer’s own obs.logs.search, obs.traces.search or obs.metrics.query on the environment’s dataset. ObsFullAccess, ObsReadOnlyAccess and ReadOnlyAccess all include them, so read-only users see the data. With a custom policy that leaves one out, the panels that need it say which action is missing.

Limits

Creating a dashboard over the quota is 409 quota_exceeded; ezgh quotas get obs.dashboards.perProject --scope-key <project> shows the limit and usage. See Service quotas.

Panels are reads like any other: the search limits and metrics query limits apply to each one. Long time ranges over busy datasets can be over a panel’s budget; give that panel a shorter range of its own.

Trails

Creating, changing and deleting dashboards are recorded in Trails as management events (CreateDashboard, UpdateDashboard, DeleteDashboard) with eventSource obs.ezghcloud.com. The spec is recorded as a hidden value, because its queries can contain your data; the version history keeps it. Viewing a dashboard and reading its panels aren’t recorded.