# Tracing
source: https://docs.chalk.ai/docs/tracing

## Use tracing to debug and optimize query performance.

Chalk provides traces for online queries so you can identify performance bottlenecks and optimize low-latency
queries. A trace breaks down every function call, from feature computation to data retrieval, and shows where
time is spent.

Chalk's tracing is built on OpenTelemetry standards. Traces are stored in a ClickHouse database deployed within your Kubernetes cluster and
viewable via the query pages in the Chalk dashboard. Chalk, by default, sets a TTL of 7 days for trace data, but
will adjust this automatically as storage limits are approached.

### Enabling traces

Tracing is typically enabled by default in Chalk environments. In order to fetch a trace for an online query,
you can either add a flag at query time or set configuration variables for environment-wide behavior.

### Requirements

To use tracing features, add the following Python dependency:

- chalkpy[tracing]

This links required peer dependencies for OpenTelemetry tracing:

- opentelemetry-api
- opentelemetry-sdk

Tracing is fully supported in the Python client starting with chalkpy 2.95.9.

### Query-time traces

To fetch a trace for a specific online query, use the --trace flag in the Chalk CLI:

```
chalk query --in customer.id=123 --out customer.num_recent_transactions --trace
```

You can also specify a trace flag in your ChalkClient call:

```
from chalk import ChalkClient

ChalkClient().query(
    input={"customer.id": 123},
    output=["customer.avg_recent_transactions_amt"],
    trace=True
)
```

After you run a traced query, open the Trace tab of the Online Query page. The trace opens as a flame
graph of the query's spans. Click a span to see its attributes in the drawer below the graph, or switch to the
Waterfall, Span list, Resolvers, Details, Logs, or Metrics view.

### Environment configuration

Following OpenTelemetry standards,
you can also configure tracing via configuration variables. Tracing across your environment can be configured using the
configuration variables OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG, which you can set in your dashboard under
Integrations > Config Variables. The default values of OTEL_TRACES_SAMPLER=parentbased_traceidratio
and OTEL_TRACES_SAMPLER_ARG=0.01 will be used, unless an explicit value is set.

The supported sampling strategies are detailed below:

| Configuration variables                                                         | Description                                                                                                                                           |
| ------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OTEL_TRACES_SAMPLER=always_on`                                                 | Trace 100% of online queries; no query-time overrides                                                                                                 |
| `OTEL_TRACES_SAMPLER=always_off`                                                | Disable tracing for all online queries; no query-time overrides                                                                                       |
| `OTEL_TRACES_SAMPLER=traceidratio` ; `OTEL_TRACES_SAMPLER_ARG=0.005`            | Sample a percentage of online queries based on `OTEL_TRACES_SAMPLER_ARG` (e.g., 0.005 for 0.5%); no query-time overrides                              |
| `OTEL_TRACES_SAMPLER=parentbased_traceidratio` ; `OTEL_TRACES_SAMPLER_ARG=0.01` | Sample a percentage of online queries based on `OTEL_TRACES_SAMPLER_ARG` (e.g., 0.01 for 1%) if there is no parent span; accepts query-time overrides |
| `OTEL_TRACES_SAMPLER=parentbased_always_on`                                     | Trace 100% of online queries if there is no parent span; accepts query-time overrides                                                                 |
| `OTEL_TRACES_SAMPLER=parentbased_always_off`                                    | Disable tracing for all online queries if there is no parent span; accepts query-time overrides                                                       |

When configuring tracing via configuration variables, you can either enforce tracing settings for all queries
("always_on" or "always_off") or enable query-time overrides ("parentbased_*" samplers). For example, if you set
OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.01, then 1% of online queries
will be traced by default, but you can still enable tracing for specific queries using the --trace flag or
the trace=True argument in ChalkClient. However, if you set OTEL_TRACES_SAMPLER=always_on, then all
online queries will be traced, and the query-time flags will be ignored.

### Troubleshooting

Traces not showing up, a "ClickHouse server is not currently reachable" error, or a trace waterfall that does not
render? See the Telemetry FAQ for fixes to common telemetry pipeline issues.