A trace records one request as it moves through your code and Chalk: each operation is a span, with its start time, duration, status, and attributes, and spans nest under the operation that started them. Chalk records traces in the OpenTelemetry format and keeps them in your Chalk environment.


​
Kinds of traces

Chalk records these traces:

  • Query traces: each step of an online query, from resolver execution to data retrieval. See Query Traces.
  • Function and sandbox traces: calls to functions, including nested calls between functions, and spans your own code adds in functions, scaling groups, and sandboxes. See Function and Sandbox Traces.
  • AI traces: traces from agents and LLM calls, such as requests through the Model Gateway and agent code instrumented with OpenInference. They add LLM, tool, and session detail. See AI Traces.

​
Where to view traces

WhereWhat it shows
Traces pageEvery trace in the current Chalk environment. See Trace Explorer.
AI traces pageOnly AI traces, with sessions. See AI Traces.
Trace tab on an online query’s pageThat query’s trace. See Query Traces.
Traces tab on a function’s or scaling group’s pageCalls to that function or scaling group.

You can also read traces with SQL from the chalk.telemetry schema. See Chalk Catalog Components.


​
How many traces Chalk keeps

Each kind of trace has its own sampling setting:

  • Online queries use the OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG environment variables, and a single query can request a trace. See Query Traces.
  • Functions, scaling groups, and sandboxes use the tracing setting. See Tracing Policy.
  • The Model Gateway uses its sampling policy. See Model Gateway.

​
Exporting traces

To send traces to your own monitoring service as well, see Export telemetry over OTLP.