# AI Traces
source: https://docs.chalk.ai/docs/traces/ai-traces

## View traces from agents and LLM calls, follow agent sessions, and read model calls, tool calls, and token usage in the Chalk dashboard.

AI traces are traces that record agent and LLM activity: model calls, tool calls, agent turns, and
the conversations they belong to. The AI traces page in the Chalk dashboard lists them, adds a
Sessions view that groups an agent's traces into conversations, and shows model, token, and
cost details for each LLM call.

For searching, filtering, and the trace panel, which work the same way on the Traces and
AI traces pages, see Trace Explorer.

### Which traces appear

The AI traces page shows a trace when its root span is one of these:

- A request through the Model Gateway. The Model Gateway
records a trace for each request it keeps.
- A span with the OpenInference openinference.span.kind attribute, which OpenInference
instrumentation libraries set on agent, chain, LLM, and tool spans.

In Spans view, the page lists every span that has the openinference.span.kind attribute.

All other traces appear only on the Traces page.

### Send AI traces from your own code

LLM calls made through the Model Gateway are recorded without any code. To
trace calls your agent makes directly to a model provider, instrument them with an
OpenInference instrumentation library, such as
openinference-instrumentation-openai, and wrap each agent turn in a chalkcompute.span() marked as
an AGENT span. In a function, scaling group, or sandbox, chalkcompute sends the spans to Chalk
using the OpenTelemetry settings Chalk provides, so no exporter setup is needed.

Set these attributes on the turn's span:

- openinference.span.kind: AGENT, which marks the span as an agent turn.
- session.id: an ID shared by every turn of the conversation, so the turns group into one
session.
- input.value and output.value: the turn's input and reply.

### Example: trace a support agent in a sandbox

This example is complete and runs as written. It traces one turn of a support agent that answers
with the OpenAI API.

- Add your OpenAI API key as a Chalk secret named OPENAI_API_KEY.
- Save agent.py and run_agent.py below in the same folder.
- From that folder, run python run_agent.py.

agent.py answers one question and records the turn:

```
import sys

import chalkcompute
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor

OpenAIInstrumentor().instrument()
client = OpenAI()


def answer(question: str, conversation_id: str) -> str:
    with chalkcompute.span("support_agent.turn") as span:
        span.set_attribute("openinference.span.kind", "AGENT")
        span.set_attribute("session.id", conversation_id)
        span.set_attribute("input.value", question)

        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "You are a support agent. Answer in one sentence."},
                {"role": "user", "content": question},
            ],
        )
        reply = response.choices[0].message.content

        span.set_attribute("output.value", reply)
        return reply


if __name__ == "__main__":
    print(answer(question=sys.argv[1], conversation_id=sys.argv[2]))
```

run_agent.py runs the agent in a sandbox. It passes the
OPENAI_API_KEY secret to the sandbox, and sets tracing=True so Chalk keeps every trace, since a
sandbox keeps about 1% by default. See Tracing Policy.

```
import chalkcompute

if __name__ == "__main__":
    image = (
        chalkcompute.Image.debian_slim("3.12")
        .pip_install(["chalkcompute", "openai", "openinference-instrumentation-openai"])
        .add_local_file("./agent.py", "/app/agent.py")
        .workdir("/app")
    )

    sandbox = chalkcompute.Sandbox(
        image=image,
        name="support-agent",
        secrets=[chalkcompute.Secret.from_chalk_env("OPENAI_API_KEY")],
        tracing=True,
    ).run()

    result = sandbox.exec("python", "/app/agent.py", "Where is my order?", "conversation-123")
    print(result.stdout_text)

    sandbox.terminate()
```

- Open the AI traces page and find the support_agent.turn trace. The OpenAI call is nested
under it, with the model, prompt, completion, and token counts. To find the conversation, open
the Sessions view and search for session:conversation-123.

### Read a trace as a conversation

Open a trace and select the Thread tab to read it as a conversation: user and agent messages,
system and developer instructions, and each tool call with its input and output. Click Show full
instruction to expand a long instruction.

### LLM span details

Click an LLM span to open it. The Messages tab shows the prompt and completion messages. The
Details tab shows the model, token counts, estimated cost, timing, generation settings such as
temperature, and token usage for the conversation so far. The token bar splits tokens into input,
cache read, cache write, and output.

Tool, agent, and turn spans show their input, output, tool name, call ID, and model.

### Sessions

A session groups the traces of one agent conversation, one trace per turn. Chalk reads a trace's
session ID from the first of these span attributes that is set:

- chalk.session_id
- session.id
- gen_ai.conversation.id
- langfuse.session.id
- ag.session.id
- session_id

If none of them is set on a span, Chalk uses the session.id resource attribute. A trace with no
session ID appears in Traces view but not in Sessions view.

Chalk reads the user a session belongs to from chalk.auth.principal.id, user.id, or
enduser.id, in that order, and then from the user.id resource attribute. A Chalk user ID shows
as that user's email address.

### Sessions view

Select Sessions above the list. Each row shows the session's start time, ID, user, first
operation, last activity, duration, time spent in LLM calls, and counts of turns, spans, LLM calls,
tool calls, and tokens, with an estimated cost. A column is empty when the traces did not report
that value.

### Open a session

Click a session to open it in a side panel with these tabs:

- Overview: totals for the session, including turns, spans, wall-clock time, LLM time, tool
calls, tokens, cached prompt tokens, estimated cost, errored turns, and the models used.
- Timeline: the session's spans on a timeline.
- Turns: one row per turn, with its model, time to first token, prompt and completion tokens,
cache reads, tools called, finish reason, and idle time before the turn.
- Conversation: the full conversation across all turns.
- Span List: every span in the session.

Click Open session page to open the session on its own page, or Show turns in the traces
list to filter the list to the session's traces. Click Debug agent session to ask the Chalk
assistant to investigate the session.





