Observability
Send Chalk logs, traces, and metrics to OTLP/HTTP destinations and verify receipt.
Send logs, traces, and metrics to your monitoring service with independent OTLP/HTTP exports. You can enable any combination of signals and keep Prometheus and Datadog exports enabled. Log and trace exports are disabled by default.
You need Metrics Export enabled for your Chalk team, permission to edit telemetry settings, and access to the receiving service’s log or trace UI. Contact Chalk support if Exporters is missing from your telemetry configuration. These exports require Vector under Telemetry Runtime.
Get each signal’s complete HTTP ingestion URL from your provider. Common paths are /v1/logs,
/v1/traces, and /v1/metrics, for example https://otlp.example.com/v1/logs. Enter the provider’s
full path, including any required query parameters. Chalk uses that URL directly and does not
append a signal path. The destination must accept OTLP protobuf over HTTP and be reachable from
Chalk. A gRPC endpoint cannot receive these exports.
If authentication or tenant headers are required, store them in a cloud secret Chalk can read. For AWS, use Secrets Manager in the account and region used by Chalk and copy the secret’s full ARN. The value must be a JSON object whose header values are strings:
{
"Authorization": "Bearer YOUR_TOKEN",
"X-Scope-OrgID": "YOUR_TENANT_ID"
}Use only the headers your provider requires. For example, a provider that requires api-key uses
{"api-key":"YOUR_API_KEY"}. Duplicate header names, including case variants, invalid header
names, and values containing line breaks are rejected. Leave the ARN empty if no custom headers
are needed. Do not put credentials in the URL.
Each signal has its own headers secret setting. You may reference the same cloud secret when the headers are identical, or use different secrets for different destinations. Header values are kept in Kubernetes Secrets and referenced by the export configuration. Ask your cloud administrator or Chalk support for help granting secret access; you do not need Kubernetes access to configure or verify an export.
| Signal | Enable switch | URL field | Headers field | Optional VRL |
|---|---|---|---|---|
| Logs | Export Logs to OTLP | Logs endpoint URL | Logs headers secret ARN | Logs VRL → Logs remap VRL |
| Traces | Export Traces to OTLP | Traces endpoint URL | Traces headers secret ARN | Traces VRL → Traces remap VRL |
| Metrics | Export Metrics to OTLP | Endpoint URL | Headers secret ARN | VRL → Metrics remap VRL |
Existing metrics exports retain their configuration and authentication. If Legacy header name and Legacy header value secret ARN are populated, you can keep them. To migrate metrics to a JSON headers secret, clear both legacy fields and configure Headers secret ARN in the same update. Log and trace exports use JSON headers secrets only.
Add this import to an existing Python resolver file and the log call inside a resolver you can query. Replace the marker with a unique value for your test:
from chalk.clogging import chalk_logger
# Inside your resolver:
chalk_logger.info("chalk-otlp-log-check-001")Deploy the resolver to the environment whose telemetry you configured, then run a query that executes it. Find the exact message in that environment’s Chalk logs. If it is absent, check that the query used the updated deployment and executed the resolver rather than returning a cached result.
Open the destination’s log search UI, select the receiving account and tenant, and search for
chalk-otlp-log-check-001 over a time range covering the test. Allow time for ingestion and refresh.
Open the result and confirm the message and timestamp match the Chalk log. If an intermediate
collector forwards your data, verify it in the final log UI too. Remove the test log call after
confirming delivery.
Run a traced online query using the supported query-time tracing controls. For example, replace the input and output features below with features in your environment:
chalk query --in customer.id=123 --out customer.num_recent_transactions --traceOpen the resulting Online Query page in Chalk. Confirm that its Trace tab contains spans, and copy Trace ID from the query details. Record the query ID and execution time as well.
Open your destination’s trace search UI, choose the receiving account and tenant, and look up that trace ID in the test’s time range. Confirm that the received spans have the same trace ID, execution time, and resolver or operation names. Inspect parent and child spans to confirm the relationships. If you forward through an intermediate collector, verify the trace in the final trace UI.
Export does not enable trace collection or override sampling. With a parentbased_* sampler,
--trace can request a trace for one query. An always_off sampler suppresses tracing even with
that flag. See environment configuration for the full
sampling behavior and Python client requirements. A missing trace in Chalk should be investigated
before checking export delivery.
Follow the existing OTLP metrics setup and verification guide to add an identifying metric attribute and find recent samples at your destination. Metrics have independent controls, so you can leave them disabled while exporting logs or traces.
Under Exporters → Shared VRL, the log, trace, or metric remap applies to all customer export destinations for that signal. The corresponding OTLP remap runs next, followed by conversion to OTLP and export. OTLP remaps do not affect the events sent to Prometheus or Datadog.
Remaps operate on native Vector events before OTLP envelopes are constructed. Do not use OTLP
wire paths such as resourceLogs in these remaps. Leave VRL empty unless you have verified the
fields available in your event. A remap that removes identifiers, changes types, or drops events
can prevent export or make received data harder to find. Repeat the receipt checks after changes.
Log export carries native log messages, including structured bodies, timestamps, severity, resource attributes, and available trace and span IDs. Trace export carries span identifiers, parent relationships, timestamps, attributes, events, links, and status. The destination may rename fields or apply its own indexing and retention rules.
Exports contain new events that reach the customer aggregator. They do not replay historical logs or traces. Upstream sampling, log limits, truncation, routing, and shared or exporter-specific VRL can affect what is delivered. Export cannot recover data already dropped or truncated.
Each OTLP signal uses an independent in-memory buffer of up to 500,000 events and drops new events when its buffer is full. Batches flush at 2,000 events, approximately 512,000 bytes, or one second. Each signal permits one HTTP request at a time, with a 15-second timeout and up to two retries for retryable failures. Restarts can lose buffered events, exhausted retries can lose data, and retries can produce duplicates. Delivery is best effort, without an exactly-once guarantee.
Customer exports do not wait for destination acknowledgements before accepting events. Bounded buffers isolate destination backpressure from other exports and Chalk ingestion. Destination healthchecks are disabled, so a successful settings update does not prove receipt. Unlike the customer Prometheus metrics sink, OTLP uses no Prometheus normalization, worker settings, memory budget, or circuit breaker. Its transport defaults match OTLP metrics for all three signals.
If an event appears in Chalk but not at the destination:
After updating credentials or the endpoint, save the configuration and send a new test event. Disabling an export stops that signal’s delivery and removes its mounted credentials after the configuration update; other signals keep their settings. Removing the export has the same effect.
If delivery still fails, send Chalk support your environment, destination URL without credentials, signal, test marker or trace ID, query ID, and test time. Do not send header values or secret contents.