Chalk provides traces for online queries so you can identify performance bottlenecks and optimize low-latency queries. A trace breaks down every function call, from feature computation to data retrieval, and shows where time is spent.

Query traces follow OpenTelemetry standards. You view them on the query pages in the Chalk dashboard. By default, Chalk keeps trace data for 7 days and shortens that automatically as storage limits are approached.


​
Enabling traces

Tracing is typically enabled by default in Chalk environments. In order to fetch a trace for an online query, you can either add a flag at query time or set configuration variables for environment-wide behavior.

​
Requirements

To use tracing features, add the following Python dependency:

  • chalkpy[tracing]

This links required peer dependencies for OpenTelemetry tracing:

  • opentelemetry-api
  • opentelemetry-sdk

Tracing is fully supported in the Python client starting with chalkpy 2.95.9.

​
Query-time traces

To fetch a trace for a specific online query, use the --trace flag in the Chalk CLI:

chalk query --in customer.id=123 --out customer.num_recent_transactions --trace

You can also specify a trace flag in your ChalkClient call:

from chalk.client import ChalkClient

ChalkClient().query(
    input={"customer.id": 123},
    output=["customer.avg_recent_transactions_amt"],
    trace=True
)

After you run a traced query, open the Trace tab of the Online Query page. The trace opens as a flame graph of the query’s spans. Click a span to see its attributes in the panel below the graph, or switch to the Waterfall, Span list, Resolvers, Details, Logs, or Metrics view.


​
Environment configuration

Following OpenTelemetry standards, you can also configure tracing via configuration variables. Tracing across your environment can be configured using the configuration variables OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, which you can set in your dashboard under Integrations > Config Variables. The default values of OTEL_TRACES_SAMPLER=parentbased_traceidratio and OTEL_TRACES_SAMPLER_ARG=0.01 will be used, unless an explicit value is set.

The supported sampling strategies are detailed below:

Configuration variablesDescription
OTEL_TRACES_SAMPLER=always_onTrace 100% of online queries; no query-time overrides
OTEL_TRACES_SAMPLER=always_offDisable tracing for all online queries; no query-time overrides
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.005
Sample a percentage of online queries based on OTEL_TRACES_SAMPLER_ARG (e.g., 0.005 for 0.5%); no query-time overrides
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.01
Sample a percentage of online queries based on OTEL_TRACES_SAMPLER_ARG (e.g., 0.01 for 1%) if there is no parent span; accepts query-time overrides
OTEL_TRACES_SAMPLER=parentbased_always_onTrace 100% of online queries if there is no parent span; accepts query-time overrides
OTEL_TRACES_SAMPLER=parentbased_always_offDisable tracing for all online queries if there is no parent span; accepts query-time overrides

When configuring tracing via configuration variables, you can either enforce tracing settings for all queries (“always_on” or “always_off”) or enable query-time overrides (“parentbased_*” samplers). For example, if you set OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.01, then 1% of online queries will be traced by default, but you can still enable tracing for specific queries using the --trace flag or the trace=True argument in ChalkClient. However, if you set OTEL_TRACES_SAMPLER=always_on, then all online queries will be traced, and the query-time flags will be ignored.


​
Troubleshooting

Traces not showing up, a “ClickHouse server is not currently reachable” error, or a trace waterfall that does not render? See the Telemetry FAQ for fixes to common telemetry pipeline issues.