Chalk traces calls to functions and collects the spans that code in functions, scaling groups, and sandboxes sends with OpenTelemetry. Use these traces to see where time is spent in a call, follow calls from one function to another, and find the step that failed.

To change how many traces Chalk keeps, see Tracing Policy.


​
Function calls

Chalk traces every function call by default. Calls a function makes to other functions appear in the same trace, so one trace shows the outer call, each nested call, and each function’s own execution:

import chalkcompute

@chalkcompute.function()
def fetch_profile(user_id: str) -> dict[str, str]:
    return {"user_id": user_id, "segment": "active"}


@chalkcompute.function()
def score_user(user_id: str) -> float:
    with chalkcompute.span("score_user.fetch_profile"):
        profile = fetch_profile.remote(user_id)

    return 0.9 if profile["segment"] == "active" else 0.1


if __name__ == "__main__":
    with chalkcompute.span("demo.score_user"):
        score = score_user.remote("user_123")

A function with tracing=False records no spans of its own.


​
Add your own spans

Use chalkcompute.span() to label a step in your code, such as a model call, a retrieval step, or preprocessing. The span becomes the parent of the calls made inside it:

import chalkcompute

with chalkcompute.span("checkout.score_transaction"):
    score = score_transaction.remote("txn_123")

To put a whole function in a span, decorate it with @chalkcompute.traced. Pass a name, or leave it out to use the function’s name:

import chalkcompute

@chalkcompute.traced("batch.score_transactions")
def run_batch(transaction_ids: list[str]) -> list[float]:
    return [score_transaction.remote(txn_id) for txn_id in transaction_ids]

chalkcompute.span() also works as a decorator, including on async functions.

Wrapping local code in a span starts a trace, so calls it makes to functions are recorded under it.

​
Pass the trace to an HTTP service

To continue a trace in a service you call over HTTP, such as a scaling group, send the trace context in the request headers. chalkcompute.trace_headers() returns the W3C trace context headers for the current trace:

import chalkcompute
import requests

with chalkcompute.span("checkout.call_pricing"):
    response = requests.post(
        "https://pricing.example.com/quote",
        json={"cart_id": "cart_123"},
        headers=chalkcompute.trace_headers(),
    )

The service continues the trace only if it reads these headers with an OpenTelemetry SDK.


​
Scaling groups and sandboxes

Chalk sets the standard OpenTelemetry environment variables in each scaling group replica and sandbox, including OTEL_EXPORTER_OTLP_ENDPOINT. An application instrumented with an OpenTelemetry SDK reads them and sends its spans to Chalk with no further setup. An application that is not instrumented sends no spans.


​
View the traces

  • The Traces tab on a function’s or scaling group’s page lists recent calls. Switch between List and Call graph to see the calls as rows or as a graph of which function called which.
  • The Traces page lists every trace in the current Chalk environment. In Spans view, search for attr:chalk.remote_function.name=<function name> to find one function’s calls. See Trace Explorer.

​
Troubleshooting

If a function’s spans do not appear:

  • Redeploy the function after changing its tracing setting.
  • Check that the function and every function it calls do not set tracing=False.
  • Open the Traces tab of the function that was called, and set a time range that includes the call.