Observability
Find and inspect traces in the Chalk dashboard with the Traces and AI traces pages, search, filters, and the trace panel.
The Chalk dashboard has two pages for exploring traces. Open them from the dashboard sidebar:
Both pages work the same way. Viewing traces requires the monitoring.read permission.
Try the AI traces page below. It runs on sample traces from a support agent.
Use the Spans, Traces, and Sessions switch above the list to change what one row represents:
A chart above the list shows how many rows fall in each part of the time range. The list loads more rows as you scroll. In Traces view, click Download as CSV to save the list.
Use the time selector at the top of the page. Traces and Sessions view start at the last hour and update live. Drag across the chart to zoom in on part of the range. The browser’s back and forward buttons step through the filters you applied.
Type a search into the search box, or click values in the facet sidebar to add them as filters.
A search is a list of key:value terms. Combine terms with AND, OR, and NOT (or a leading
-), and group them with parentheses. In Spans view, values accept * as a wildcard.
service:recommender AND status:error
ai_kind:llm -operation:healthcheck
min_duration:2s (session:abc123 OR user:alice@example.com)
In Spans view, a term matches any span in the time range:
| Key | Matches |
|---|---|
trace, span | Trace ID or span ID |
operation or name | Span name |
service | Service name |
status, kind | Span status, OpenTelemetry span kind |
ai_kind | AI span kind, such as llm, tool, agent, or chain |
session | Session ID |
user or principal | User ID |
min_duration, max_duration | Span duration |
attr:<key> or attr:<key>=<value> | A span attribute, present or equal to a value |
resource_attr:<key> | A resource attribute |
input or prompt, output or completion | Text in an LLM span’s prompt or completion |
content or message | Text in either the prompt or the completion |
A key not in this list matches a span attribute with that name.
Durations take a unit: us, ms, s, m, or h, as in min_duration:500ms. A number with no
unit is in microseconds.
In Traces and Sessions view, a term matches the root span of each trace. These keys are
available: trace, session, root_span_name, root_span_id, user or principal,
min_duration, max_duration, min_spans, max_spans, attr, resource_attr, attr_value,
and resource_attr_value.
In Spans view, the sidebar lists values for operation name, service name, status, span kind, AI span kind, trace ID, session ID, user, and duration. In Traces view, it lists root span names, attributes, and resources. Click a value to filter on it.
Click a row to open the trace in a side panel. The top of the panel shows the trace ID, a download button, and a button that asks the Chalk assistant to investigate the trace.
The panel has these tabs:
Click a span in any tab to select it. An LLM span opens with Messages and Details tabs. See AI Traces.