Observability
View traces from agents and LLM calls, follow agent sessions, and read model calls, tool calls, and token usage in the Chalk dashboard.
AI traces are traces that record agent and LLM activity: model calls, tool calls, agent turns, and the conversations they belong to. The AI traces page in the Chalk dashboard lists them, adds a Sessions view that groups an agent’s traces into conversations, and shows model, token, and cost details for each LLM call.
For searching, filtering, and the trace panel, which work the same way on the Traces and AI traces pages, see Trace Explorer.
The AI traces page shows a trace when its root span is one of these:
openinference.span.kind attribute, which OpenInference
instrumentation libraries set on agent, chain, LLM, and tool spans.In Spans view, the page lists every span that has the openinference.span.kind attribute.
All other traces appear only on the Traces page.
LLM calls made through the Model Gateway are recorded without any code. To
trace calls your agent makes directly to a model provider, instrument them with an
OpenInference instrumentation library, such as
openinference-instrumentation-openai, and wrap each agent turn in a chalkcompute.span() marked as
an AGENT span. In a function, scaling group, or sandbox, chalkcompute sends the spans to Chalk
using the OpenTelemetry settings Chalk provides, so no exporter setup is needed.
Set these attributes on the turn’s span:
openinference.span.kind: AGENT, which marks the span as an agent turn.session.id: an ID shared by every turn of the conversation, so the turns group into one
session.input.value and output.value: the turn’s input and reply.This example is complete and runs as written. It traces one turn of a support agent that answers with the OpenAI API.
OPENAI_API_KEY.agent.py and run_agent.py below in the same folder.python run_agent.py.agent.py answers one question and records the turn:
import sys
import chalkcompute
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument()
client = OpenAI()
def answer(question: str, conversation_id: str) -> str:
with chalkcompute.span("support_agent.turn") as span:
span.set_attribute("openinference.span.kind", "AGENT")
span.set_attribute("session.id", conversation_id)
span.set_attribute("input.value", question)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a support agent. Answer in one sentence."},
{"role": "user", "content": question},
],
)
reply = response.choices[0].message.content
span.set_attribute("output.value", reply)
return reply
if __name__ == "__main__":
print(answer(question=sys.argv[1], conversation_id=sys.argv[2]))run_agent.py runs the agent in a sandbox. It passes the
OPENAI_API_KEY secret to the sandbox, and sets tracing=True so Chalk keeps every trace, since a
sandbox keeps about 1% by default. See Tracing Policy.
import chalkcompute
if __name__ == "__main__":
image = (
chalkcompute.Image.debian_slim("3.12")
.pip_install(["chalkcompute", "openai", "openinference-instrumentation-openai"])
.add_local_file("./agent.py", "/app/agent.py")
.workdir("/app")
)
sandbox = chalkcompute.Sandbox(
image=image,
name="support-agent",
secrets=[chalkcompute.Secret.from_chalk_env("OPENAI_API_KEY")],
tracing=True,
).run()
result = sandbox.exec("python", "/app/agent.py", "Where is my order?", "conversation-123")
print(result.stdout_text)
sandbox.terminate()support_agent.turn trace. The OpenAI call is nested
under it, with the model, prompt, completion, and token counts. To find the conversation, open
the Sessions view and search for session:conversation-123.Open a trace and select the Thread tab to read it as a conversation: user and agent messages, system and developer instructions, and each tool call with its input and output. Click Show full instruction to expand a long instruction.
Click an LLM span to open it. The Messages tab shows the prompt and completion messages. The Details tab shows the model, token counts, estimated cost, timing, generation settings such as temperature, and token usage for the conversation so far. The token bar splits tokens into input, cache read, cache write, and output.
Tool, agent, and turn spans show their input, output, tool name, call ID, and model.
A session groups the traces of one agent conversation, one trace per turn. Chalk reads a trace’s session ID from the first of these span attributes that is set:
chalk.session_idsession.idgen_ai.conversation.idlangfuse.session.idag.session.idsession_idIf none of them is set on a span, Chalk uses the session.id resource attribute. A trace with no
session ID appears in Traces view but not in Sessions view.
Chalk reads the user a session belongs to from chalk.auth.principal.id, user.id, or
enduser.id, in that order, and then from the user.id resource attribute. A Chalk user ID shows
as that user’s email address.
Select Sessions above the list. Each row shows the session’s start time, ID, user, first operation, last activity, duration, time spent in LLM calls, and counts of turns, spans, LLM calls, tool calls, and tokens, with an estimated cost. A column is empty when the traces did not report that value.
Click a session to open it in a side panel with these tabs:
Click Open session page to open the session on its own page, or Show turns in the traces list to filter the list to the session’s traces. Click Debug agent session to ask the Chalk assistant to investigate the session.