# Export telemetry over OTLP
source: https://docs.chalk.ai/docs/otlp-telemetry-export

## Send Chalk logs, traces, and metrics to OTLP/HTTP destinations and verify receipt.

Send logs, traces, and metrics to your monitoring service with independent OTLP/HTTP exports.
You can enable any combination of signals and keep Prometheus and Datadog exports enabled.
Log and trace exports are disabled by default.

### Prepare your destination

You need Metrics Export enabled for your Chalk team, permission to edit telemetry settings,
and access to the receiving service's log or trace UI. Contact Chalk support if Exporters
is missing from your telemetry configuration. These exports require Vector under
Telemetry Runtime.

Get each signal's complete HTTP ingestion URL from your provider. Common paths are /v1/logs,
/v1/traces, and /v1/metrics, for example https://otlp.example.com/v1/logs. Enter the provider's
full path, including any required query parameters. Chalk uses that URL directly and does not
append a signal path. The destination must accept OTLP protobuf over HTTP and be reachable from
Chalk. A gRPC endpoint cannot receive these exports.

If authentication or tenant headers are required, store them in a cloud secret Chalk can read.
For AWS, use Secrets Manager in the account and region used by Chalk and copy the secret's full ARN.
The value must be a JSON object whose header values are strings:

```
{
  "Authorization": "Bearer YOUR_TOKEN",
  "X-Scope-OrgID": "YOUR_TENANT_ID"
}
```

Use only the headers your provider requires. For example, a provider that requires api-key uses
{"api-key":"YOUR_API_KEY"}. Duplicate header names, including case variants, invalid header
names, and values containing line breaks are rejected. Leave the ARN empty if no custom headers
are needed. Do not put credentials in the URL.

Each signal has its own headers secret setting. You may reference the same cloud secret when the
headers are identical, or use different secrets for different destinations. Header values are kept
in Kubernetes Secrets and referenced by the export configuration. Ask your cloud administrator
or Chalk support for help granting secret access; you do not need Kubernetes access to configure
or verify an export.

### Enable the signals you need

- Open Cloud Clusters, select your cluster's Telemetry tab,
and open its telemetry configuration.
- Select Vector under Telemetry Runtime, then open Exporters → Export to OTLP.
- Configure each signal using the controls below. Preserve any exports you already use.

| Signal  | Enable switch              | URL field               | Headers field                 | Optional VRL                      |
| ------- | -------------------------- | ----------------------- | ----------------------------- | --------------------------------- |
| Logs    | **Export Logs to OTLP**    | **Logs endpoint URL**   | **Logs headers secret ARN**   | **Logs VRL → Logs remap VRL**     |
| Traces  | **Export Traces to OTLP**  | **Traces endpoint URL** | **Traces headers secret ARN** | **Traces VRL → Traces remap VRL** |
| Metrics | **Export Metrics to OTLP** | **Endpoint URL**        | **Headers secret ARN**        | **VRL → Metrics remap VRL**       |

- Leave optional VRL empty for initial setup. Save, review the changes, and confirm the update.
Wait for the dashboard to confirm success and leave Telemetry enabled.
- Verify each enabled signal using the steps below. Enabling logs does not enable traces or metrics.

Existing metrics exports retain their configuration and authentication. If Legacy header name
and Legacy header value secret ARN are populated, you can keep them. To migrate metrics to a
JSON headers secret, clear both legacy fields and configure Headers secret ARN in the same
update. Log and trace exports use JSON headers secrets only.

### Verify logs

Add this import to an existing Python resolver file and the log call inside a resolver you can query.
Replace the marker with a unique value for your test:

```
from chalk.clogging import chalk_logger

# Inside your resolver:
chalk_logger.info("chalk-otlp-log-check-001")
```

Deploy the resolver to the environment whose telemetry you configured,
then run a query that executes it. Find the exact message in that environment's Chalk logs.
If it is absent, check that the query used the updated deployment and executed the resolver rather
than returning a cached result.

Open the destination's log search UI, select the receiving account and tenant, and search for
chalk-otlp-log-check-001 over a time range covering the test. Allow time for ingestion and refresh.
Open the result and confirm the message and timestamp match the Chalk log. If an intermediate
collector forwards your data, verify it in the final log UI too. Remove the test log call after
confirming delivery.

### Verify traces

Run a traced online query using the supported query-time tracing controls.
For example, replace the input and output features below with features in your environment:

```
chalk query --in customer.id=123 --out customer.num_recent_transactions --trace
```

Open the resulting Online Query page in Chalk. Confirm that its Trace tab contains spans,
and copy Trace ID from the query details. Record the query ID and execution time as well.

Open your destination's trace search UI, choose the receiving account and tenant, and look up that
trace ID in the test's time range. Confirm that the received spans have the same trace ID, execution
time, and resolver or operation names. Inspect parent and child spans to confirm the relationships.
If you forward through an intermediate collector, verify the trace in the final trace UI.

Export does not enable trace collection or override sampling. With a parentbased_* sampler,
--trace can request a trace for one query. An always_off sampler suppresses tracing even with
that flag. See environment configuration for the full
sampling behavior and Python client requirements. A missing trace in Chalk should be investigated
before checking export delivery.

### Verify metrics

Follow the existing OTLP metrics setup and verification guide
to add an identifying metric attribute and find recent samples at your destination. Metrics have
independent controls, so you can leave them disabled while exporting logs or traces.

### Customize events with VRL

Under Exporters → Shared VRL, the log, trace, or metric remap applies to all customer export
destinations for that signal. The corresponding OTLP remap runs next, followed by conversion to
OTLP and export. OTLP remaps do not affect the events sent to Prometheus or Datadog.

Remaps operate on native Vector events before OTLP envelopes are constructed. Do not use OTLP
wire paths such as resourceLogs in these remaps. Leave VRL empty unless you have verified the
fields available in your event. A remap that removes identifiers, changes types, or drops events
can prevent export or make received data harder to find. Repeat the receipt checks after changes.

### Supported data and delivery behavior

Log export carries native log messages, including structured bodies, timestamps, severity,
resource attributes, and available trace and span IDs. Trace export carries span identifiers,
parent relationships, timestamps, attributes, events, links, and status. The destination may
rename fields or apply its own indexing and retention rules.

Exports contain new events that reach the customer aggregator. They do not replay historical
logs or traces. Upstream sampling, log limits, truncation, routing, and shared or exporter-specific
VRL can affect what is delivered. Export cannot recover data already dropped or truncated.

Each OTLP signal uses an independent in-memory buffer of up to 500,000 events and drops new events
when its buffer is full. Batches flush at 2,000 events, approximately 512,000 bytes, or one second.
Each signal permits one HTTP request at a time, with a 15-second timeout and up to two retries for
retryable failures. Restarts can lose buffered events, exhausted retries can lose data, and retries
can produce duplicates. Delivery is best effort, without an exactly-once guarantee.

Customer exports do not wait for destination acknowledgements before accepting events. Bounded
buffers isolate destination backpressure from other exports and Chalk ingestion. Destination
healthchecks are disabled, so a successful settings update does not prove receipt. Unlike the
customer Prometheus metrics sink, OTLP uses no Prometheus normalization, worker settings, memory
budget, or circuit breaker. Its transport defaults match OTLP metrics for all three signals.

### Troubleshooting

If an event appears in Chalk but not at the destination:

- Confirm the signal's switch is enabled and the saved update succeeded.
- Check the complete HTTP URL, signal path, required headers, and Chalk's permission to read the secret.
- Select the correct destination account, tenant, time range, and data source. Clear unrelated filters.
- Check destination sampling, log exclusion, indexing rules, and field mappings.
- Check shared and OTLP-specific VRL for filters or changes to identifiers.
- Check whether the destination is rejecting requests, throttling ingestion, or exceeding its quota.

After updating credentials or the endpoint, save the configuration and send a new test event.
Disabling an export stops that signal's delivery and removes its mounted credentials after the
configuration update; other signals keep their settings. Removing the export has the same effect.

If delivery still fails, send Chalk support your environment, destination URL without credentials,
signal, test marker or trace ID, query ID, and test time. Do not send header values or secret contents.