# Query Traces
source: https://docs.chalk.ai/docs/query-traces

## Use query traces to debug and optimize online query performance.

Chalk provides traces for online queries so you can identify performance bottlenecks and optimize low-latency
queries. A trace breaks down every function call, from feature computation to data retrieval, and shows where
time is spent.

Query traces follow OpenTelemetry standards. You view them on the query pages in the Chalk dashboard. By default, Chalk
keeps trace data for 7 days and shortens that automatically as storage limits are approached.

### Enabling traces

Tracing is typically enabled by default in Chalk environments. In order to fetch a trace for an online query,
you can either add a flag at query time or set configuration variables for environment-wide behavior.

### Requirements

To use tracing features, add the following Python dependency:

- chalkpy[tracing]

This links required peer dependencies for OpenTelemetry tracing:

- opentelemetry-api
- opentelemetry-sdk

Tracing is fully supported in the Python client starting with chalkpy 2.95.9.

### Query-time traces

To fetch a trace for a specific online query, use the --trace flag in the Chalk CLI:

```
chalk query --in customer.id=123 --out customer.num_recent_transactions --trace
```

You can also specify a trace flag in your ChalkClient call:

```
from chalk.client import ChalkClient

ChalkClient().query(
    input={"customer.id": 123},
    output=["customer.avg_recent_transactions_amt"],
    trace=True
)
```

After you run a traced query, open the Trace tab of the Online Query page. The trace opens as a flame
graph of the query's spans. Click a span to see its attributes in the panel below the graph, or switch to the
Waterfall, Span list, Resolvers, Details, Logs, or Metrics view.

### Environment configuration

Following OpenTelemetry standards,
you can also configure tracing via configuration variables. Tracing across your environment can be configured using the
configuration variables OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, which you can set in your dashboard under
Integrations > Config Variables. The default values of OTEL_TRACES_SAMPLER=parentbased_traceidratio
and OTEL_TRACES_SAMPLER_ARG=0.01 will be used, unless an explicit value is set.

The supported sampling strategies are detailed below:

| Configuration variables                                                         | Description                                                                                                                                           |
| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OTEL_TRACES_SAMPLER=always_on`                                                 | Trace 100% of online queries; no query-time overrides                                                                                                 |
| `OTEL_TRACES_SAMPLER=always_off`                                                | Disable tracing for all online queries; no query-time overrides                                                                                       |
| `OTEL_TRACES_SAMPLER=traceidratio` ; `OTEL_TRACES_SAMPLER_ARG=0.005`            | Sample a percentage of online queries based on `OTEL_TRACES_SAMPLER_ARG` (e.g., 0.005 for 0.5%); no query-time overrides                              |
| `OTEL_TRACES_SAMPLER=parentbased_traceidratio` ; `OTEL_TRACES_SAMPLER_ARG=0.01` | Sample a percentage of online queries based on `OTEL_TRACES_SAMPLER_ARG` (e.g., 0.01 for 1%) if there is no parent span; accepts query-time overrides |
| `OTEL_TRACES_SAMPLER=parentbased_always_on`                                     | Trace 100% of online queries if there is no parent span; accepts query-time overrides                                                                 |
| `OTEL_TRACES_SAMPLER=parentbased_always_off`                                    | Disable tracing for all online queries if there is no parent span; accepts query-time overrides                                                       |

When configuring tracing via configuration variables, you can either enforce tracing settings for all queries
("always_on" or "always_off") or enable query-time overrides ("parentbased_*" samplers). For example, if you set
OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.01, then 1% of online queries
will be traced by default, but you can still enable tracing for specific queries using the --trace flag or
the trace=True argument in ChalkClient. However, if you set OTEL_TRACES_SAMPLER=always_on, then all
online queries will be traced, and the query-time flags will be ignored.

### Troubleshooting

Traces not showing up, a "ClickHouse server is not currently reachable" error, or a trace waterfall that does not
render? See the Telemetry FAQ for fixes to common telemetry pipeline issues.





