Telemetry FAQ

---

Telemetry is the observability pipeline that powers the logs and traces you see in the Chalk dashboard: the structured-logs view, the query-logs view, and the Trace tab on the Online Query page. It runs entirely inside your Kubernetes cluster, and almost all of it is configurable from the Telemetry tab of your cluster on the Cloud Clusters page. This page answers the most common questions about why telemetry might not behave as expected, and how to resolve them yourself.

---

​

How do I export metrics to Prometheus?

Follow these steps to send Chalk metrics to your Prometheus-compatible monitoring service. When you finish, you will be able to query those metrics in your destination. This export sends metrics; for logs, follow Export logs to Datadog.

​

1. Prepare your destination

You need Metrics Export enabled for your Chalk team, permission to edit telemetry settings, and access to your monitoring service’s metrics query interface.

Get the full Prometheus remote-write URL from your monitoring service, including its write path. For example:

https://prometheus.example.com/api/v1/write

Use the ingestion URL supplied by your provider. The destination must have remote-write receiving enabled and be reachable from Chalk.

If the destination requires authentication or a tenant header, store its required HTTP headers as a JSON object in a cloud secret that Chalk can read. For AWS, use Secrets Manager in the account and region used by Chalk and copy the secret’s full ARN. For example, a service that requires a bearer token and tenant ID would use this secret value:

{
  "Authorization": "Bearer YOUR_TOKEN",
  "X-Scope-OrgID": "YOUR_TENANT_ID"
}

Use only the headers required by your destination, with string values. If you need help storing the secret or granting Chalk access, ask your cloud administrator or Chalk support. You can skip the secret when your destination does not require custom headers.

​

2. Enable the export in Chalk

1. Open Cloud Clusters, select your cluster’s Telemetry tab, and open its telemetry configuration.
2. In the guided form, select Vector under Telemetry Runtime. Open Exporters → Export to Prometheus, alongside Export to Datadog. If Exporters is missing, contact Chalk support to check that Metrics Export is enabled for your team.
3. Turn on Export Metrics to Prometheus.
4. Enter your full remote-write URL in Endpoint URL.
5. If you prepared a headers secret, paste its ARN into Headers secret ARN. Otherwise, leave the field empty.
6. Open VRL and add this line to Metrics remap VRL to label the exported metrics. Replace chalk-production with a fixed name for this export. Preserve any remap code already in the field.

.tags.chalk_export = "chalk-production"

7. Save, review the changes, and confirm the update. Wait for the dashboard to confirm that the update succeeded. Leave Telemetry enabled.

The label applies to this Prometheus export. You can keep existing Datadog exports enabled at the same time. New metric samples are sent automatically; historical samples are not backfilled.

​

3. Verify metrics at your destination

Run a few Chalk queries in the environment you configured, then open your monitoring service’s metrics query interface, such as Prometheus or Grafana Explore. Select the data source and tenant that receive your remote writes, and a time range covering the last 15 minutes.

Run this PromQL selector, using the label value you configured:

{chalk_export="chalk-production"}

Allow time for collection and ingestion, then refresh the results. Check that the returned series have the chalk_export label and samples from after you enabled the export.

The export is working when recent metric samples appear at your destination with your label. You can retain the label to identify Chalk metrics in dashboards and alerts.

If no samples appear, confirm the export is enabled and saved, check the full write URL and required headers, and verify that your query uses the receiving data source and tenant. If you still see no data, send Chalk support your Chalk environment, destination URL, label value, and the time you enabled export. Do not include credentials in the support message.

​

How do I export metrics over OTLP?

Follow these steps to send Chalk metrics to an OpenTelemetry Protocol (OTLP) endpoint. When you finish, you will be able to find recent Chalk metric samples in your monitoring service. This export sends metrics over HTTP. For logs, follow Export logs to Datadog.

​

1. Prepare your OTLP destination

You need Metrics Export enabled for your Chalk team, permission to edit telemetry settings, and access to your destination’s metrics query interface or received metrics view.

Get the full OTLP/HTTP metrics ingestion URL from your provider, including the /v1/metrics path. For example:

https://otlp.example.com/v1/metrics

Enter the complete URL; Chalk sends requests to it as supplied. The receiver must accept OTLP/HTTP protobuf metrics and be reachable from Chalk. Use the provider’s HTTP endpoint, even if it also offers a gRPC endpoint.

If your provider requires authentication or a tenant header, store the required HTTP headers as a JSON object in a cloud secret that Chalk can read. For AWS, use Secrets Manager in the account and region used by Chalk and copy the secret’s full ARN. For example:

{
  "Authorization": "Bearer YOUR_TOKEN",
  "X-Scope-OrgID": "YOUR_TENANT_ID"
}

Use only the headers required by your provider, with string values. A provider that uses an api-key header instead would use a secret such as {"api-key":"YOUR_API_KEY"}. If you need help storing the secret or granting Chalk access, ask your cloud administrator or Chalk support. Skip the secret when your destination does not require custom headers.

​

2. Enable OTLP export in Chalk

1. Open Cloud Clusters, select your cluster’s Telemetry tab, and open its telemetry configuration.
2. In the guided form, select Vector under Telemetry Runtime. Open Exporters → Export to OTLP, alongside the Prometheus and Datadog exports. If the panel is missing, contact Chalk support to check availability for your team.
3. Turn on Export Metrics to OTLP.
4. Enter the full OTLP/HTTP metrics URL in Endpoint URL.
5. If you prepared a headers secret, paste its ARN into Headers secret ARN. Otherwise, leave the field empty.
6. Open VRL and add this line to Metrics remap VRL so you can identify the received metrics. Replace chalk-otlp-production with a fixed name for this export. Preserve any remap code already in the field.

.tags.chalk_export = "chalk-otlp-production"

7. Save, review the changes, and confirm the update. Wait for the dashboard to confirm that the update succeeded. Leave Telemetry enabled.

If Legacy header name and Legacy header value secret ARN appear on an existing export, you can keep using those settings. To switch to a JSON headers secret, include the required header in the new secret and clear both legacy fields in the same update. Use one authentication method at a time.

The remap applies only to this OTLP export, after any shared metrics remap. You can keep Prometheus and Datadog exports enabled at the same time. New metric samples are sent automatically; historical samples are not backfilled.

​

3. Verify OTLP metrics at your destination

Run a few Chalk queries in the environment you configured, then open your destination’s metrics query interface or received metrics view. Select the account, tenant, and data source that receive your OTLP data, and a time range covering the last 15 minutes.

Filter for the metric attribute chalk_export with the value chalk-otlp-production, or the value you chose. Your monitoring service may display metric attributes as tags or labels. If it exposes them as Prometheus labels, you can use:

{chalk_export="chalk-otlp-production"}

Allow time for collection and ingestion, then refresh. Check that matching metrics have samples from after you enabled export. If you forward from an OTLP receiver to a separate monitoring service, verify the samples in that final service as well.

The export is working when recent metric samples appear at your destination with your attribute. You can retain the attribute to identify Chalk metrics in dashboards and alerts.

If no samples appear, confirm the export is enabled and saved, check the full metrics URL and required headers, and verify that you are searching the receiving account and tenant. Check any attribute mapping or filtering in your destination. If data still does not arrive, send Chalk support your Chalk environment, destination URL, attribute value, and the time you enabled export. Do not include credentials in the support message.

​

How do I export logs or traces over OTLP?

Follow Export telemetry over OTLP to configure independent log and trace endpoints, JSON headers secrets, and optional VRL. The guide includes a resolver log marker and a traced online query to verify receipt in your destination. Logs and traces default to disabled; you can enable either without Datadog and keep existing metrics exports running.

​

How do I export logs to Datadog?

Follow Export logs to Datadog to connect your Datadog account, turn on log export, and verify delivery with a test message in Datadog Log Explorer.

---

​

How telemetry works

Logs and traces flow through four stages, all running in your cluster in the chalk-telemetry namespace:

Collector (OpenTelemetry or Vector) → Aggregator → ClickHouse → Dashboard

• The collector receives logs and traces emitted by the engine and resolvers. These pods are named chalk-collector-*.
• The aggregator batches and forwards them. This pod is named chalk-telemetry-aggregator-*.
• ClickHouse is the database that stores the logs and traces. This is the clickhouse-0 pod. Traces have a default TTL of 7 days, which Chalk shortens automatically as storage fills.
• The dashboard reads from ClickHouse to render the logs views and the Trace tab.

Because the whole pipeline is just a set of pods, most telemetry problems are infrastructure problems (a pod that can’t schedule, or one that’s out of memory) rather than bugs in your features or resolvers. The two places you’ll work from are:

• Cloud Clusters > (your cluster) > Telemetry: click your telemetry deployment to view and edit it (CPU/memory for ClickHouse, the collector, and the aggregator; ClickHouse storage size; collector runtime; and more). The guided form and a raw JSON editor are both available here.
• Infrastructure > Kubernetes: inspect pod health. On the All Pods tab, set the namespace filter to chalk-telemetry to see only the telemetry pods and their status, restarts, and age. Click a pod to view its logs.

---

​

Why are my logs or traces not showing up in the dashboard?

Logs and traces share the same pipeline, so the first checks are the same:

1. Did the telemetry settings change recently? A recent edit to the ClickHouse or aggregator resources in the cluster’s telemetry deployment is the most common trigger.
2. Are the telemetry pods healthy? Go to Infrastructure > Kubernetes, filter the namespace to chalk-telemetry, and confirm clickhouse-0, the chalk-collector-* pods, and the chalk-telemetry-aggregator-* pod are all Running. If any are Pending, OOMKilled, or in CrashLoopBackOff, see the next question.
3. For traces specifically, also confirm tracing is actually enabled and sampled. See below.

Note the 7-day trace TTL: traces older than roughly a week are expected to age out, and the window can be shorter if ClickHouse storage filled up. A missing old trace is usually expected behavior, not a failure.

---

​

What does “ClickHouse server is not currently reachable” mean?

You may see:

“Clickhouse server is not currently reachable. It is likely that the instance is under maintenance or the pod has been moved to a new node. Please try again in a few minutes…”

This is the most frequent telemetry error. It appears when the logs / traces views try to query ClickHouse and can’t reach it.

It only affects your ability to view telemetry in the dashboard. Online and offline query serving and feature computation are completely unaffected, and your application keeps working normally.

Causes, in order of likelihood:

1. Transient unavailability. ClickHouse was briefly down for maintenance, or its pod was just rescheduled onto a new node. Wait a few minutes and refresh. If it clears, no action is needed.
2. The ClickHouse (or aggregator) pod can’t schedule. Check the clickhouse-0 pod in the chalk-telemetry namespace (Infrastructure > Kubernetes, namespace filter chalk-telemetry). If it’s Pending with FailedScheduling events, its resource request is too large for the node it targets. The classic mistake is setting a memory request equal to the node’s total memory (for example, a 32Gi request on a 32Gi m6a.2xlarge). Daemonsets and system overhead consume part of every node, so a request that equals node capacity can never be placed and the pod stays Pending forever. Pinning a fixed instance type and a high request makes this especially easy to trip.
3. OOM / CrashLoopBackOff. The aggregator can run out of memory under heavy query volume, and ClickHouse can restart under memory pressure. Repeated OOMKilled status means memory needs to go up, but only to a value that still fits the node.

Self-serve fix: open the telemetry deployment from Cloud Clusters > (your cluster) > Telemetry and adjust the memory in the ClickHouse and/or Aggregator sections.

• If a pod can’t schedule: lower the request so it fits the node (leave headroom below the instance’s total memory, for example request 30Gi on a 32Gi machine rather than 32Gi), or raise the instance type. Never set the request equal to the machine size.
• If a pod is OOMing: raise the memory request for the resource that ran out of memory (the aggregator is often the right knob), staying under node capacity.

Then Save & Apply, confirm clickhouse-0 reaches Running, and reload the dashboard. If it still can’t reach ClickHouse after about 10 minutes, contact Chalk support.

---

​

I added --trace but no trace appears

Work through these in order:

• Check the flag syntax. It is --trace, with no space. Writing chalk query … -- trace (a space after --) is parsed as “end of flags” plus a stray positional argument, so tracing is silently not enabled and you get no error.
• Check sampling. Tracing across your environment is governed by OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG (Settings > Integrations > Config Variables). The defaults are parentbased_traceidratio at 0.01 (1%). If the sampler is set to always_off, query-time flags are ignored and nothing is traced. With a parentbased_* sampler you can still force a single query with --trace (CLI) or trace=True (ChalkClient). See Tracing for the full table of samplers.
• Check the client. Tracing requires the chalkpy[tracing] extra and chalkpy >= 2.95.9.
• View it in the right place. A traced query’s waterfall appears in the Trace tab of the Online Query page.

---

​

How do I use traces to find a slow query?

Once tracing works, traces are the primary tool for locating latency. Enable tracing on a representative query, open the Trace tab, and look for the longest spans. The waterfall shows where time is spent and what called what. For example, it distinguishes fast Redis reads from slow warehouse or Postgres round trips.

From there:

• Target the offending data source (add indices, increase its CPU/memory, or add read replicas).
• Accelerate any unaccelerated resolvers that dominate the trace. See Static Resolver Optimization.
• Consider querying through the gRPC client (ChalkGRPCClient), which is generally faster than HTTP.

For the broader set of tools for debugging queries (Query Plan Visualizer, Resolver Replay, logging), see Debugging Queries.

---

​

When should I contact Chalk support?

Reach out to Chalk support if:

• The “not reachable” error persists beyond about 10 minutes after confirming the telemetry pods are healthy.
• Pods keep OOMing or restarting even after correctly sizing memory within node limits.
• ClickHouse won’t schedule and it’s unclear which combination of request and instance type will fit.
• You’re considering Serve over HTTP or other advanced fields. Confirm with the Chalk team first.

---

​

Additional resources

• Tracing: how to enable tracing, sampling configuration, and viewing the Trace tab.
• Observability Overview: the full observability section, including metrics and alerting.
• Log Export: export logs to external monitoring systems and the log field schema.
• Debugging Queries: Query Plan Visualizer, Resolver Replay, and logging.
• Kubernetes Resources Overview: how Chalk uses Kubernetes resources.