Observability
Trace function calls, add your own spans with chalkcompute, and view traces from functions, scaling groups, and sandboxes.
Chalk traces calls to functions and collects the spans that code in functions, scaling groups, and sandboxes sends with OpenTelemetry. Use these traces to see where time is spent in a call, follow calls from one function to another, and find the step that failed.
To change how many traces Chalk keeps, see Tracing Policy.
Chalk traces every function call by default. Calls a function makes to other functions appear in the same trace, so one trace shows the outer call, each nested call, and each function’s own execution:
import chalkcompute
@chalkcompute.function()
def fetch_profile(user_id: str) -> dict[str, str]:
return {"user_id": user_id, "segment": "active"}
@chalkcompute.function()
def score_user(user_id: str) -> float:
with chalkcompute.span("score_user.fetch_profile"):
profile = fetch_profile.remote(user_id)
return 0.9 if profile["segment"] == "active" else 0.1
if __name__ == "__main__":
with chalkcompute.span("demo.score_user"):
score = score_user.remote("user_123")
A function with tracing=False records no spans of its own.
Use chalkcompute.span() to label a step in your code, such as a model call, a retrieval step, or
preprocessing. The span becomes the parent of the calls made inside it:
import chalkcompute
with chalkcompute.span("checkout.score_transaction"):
score = score_transaction.remote("txn_123")
To put a whole function in a span, decorate it with @chalkcompute.traced. Pass a name, or leave it
out to use the function’s name:
import chalkcompute
@chalkcompute.traced("batch.score_transactions")
def run_batch(transaction_ids: list[str]) -> list[float]:
return [score_transaction.remote(txn_id) for txn_id in transaction_ids]
chalkcompute.span() also works as a decorator, including on async functions.
Wrapping local code in a span starts a trace, so calls it makes to functions are recorded under it.
To continue a trace in a service you call over HTTP, such as a scaling group, send the trace
context in the request headers. chalkcompute.trace_headers() returns the W3C trace context
headers for the current trace:
import chalkcompute
import requests
with chalkcompute.span("checkout.call_pricing"):
response = requests.post(
"https://pricing.example.com/quote",
json={"cart_id": "cart_123"},
headers=chalkcompute.trace_headers(),
)
The service continues the trace only if it reads these headers with an OpenTelemetry SDK.
Chalk sets the standard OpenTelemetry environment variables in each scaling group replica and
sandbox, including OTEL_EXPORTER_OTLP_ENDPOINT. An application instrumented with an OpenTelemetry
SDK reads them and sends its spans to Chalk with no further setup. An application that is not
instrumented sends no spans.
attr:chalk.remote_function.name=<function name> to find one function’s calls. See
Trace Explorer.If a function’s spans do not appear:
tracing setting.tracing=False.