Reading and Comparing Evaluation Results

The examples on this page continue the Quick start and the built-in scorers example on the Scorers page. To run them on their own, start a file with the code below, which creates that evaluation, and add the examples after it in order. As in the Quick start, definitions stay at the top level and the code that uploads data or creates and runs evaluations goes under if __name__ == "__main__":.

import chalkcompute

@chalkcompute.function
def triage(message: str) -> str:
    import json, re
    text = message.lower()
    if "refund" in text or "money back" in text:
        intent = "refund"
    elif "cancel" in text or "close my account" in text:
        intent = "cancel"
    elif "invoice" in text or "billed" in text or "card" in text:
        intent = "billing"
    elif "where is" in text or "waiting" in text or "arrived" in text:
        intent = "status"
    else:
        return "unclassified"
    amount = re.search(r"\d+(?:\.\d+)?", message)
    return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})

scorers = [
    chalkcompute.scorers.json_valid(inputs=("output",), name="valid-json"),
    chalkcompute.scorers.json_match(inputs=("output", "expected"), name="full-match"),
    chalkcompute.scorers.json_match(
        inputs=("output", "expected"), paths=["intent"], name="intent-match"
    ),
    chalkcompute.scorers.contains(inputs=("output", "expected_intent"), name="mentions-intent"),
]

if __name__ == "__main__":
    dataset = chalkcompute.DatasetClient().upload("support-triage", [
        {"message": "I was charged twice for order 1182 ($49.99), please refund",
         "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
        {"message": "Where is my order? It was due Tuesday.",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
        {"message": "Please close my account and refund the last $12 charge",
         "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
        {"message": "Following up on my last email",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
    ])

    evaluation = chalkcompute.Evaluation.create(
        "support-triage",
        dataset=dataset,
        task=triage,
        scorers=scorers,
    )

---

​

Reading results

A run writes one row per dataset row into its result dataset. Read it with DatasetClient.read, which returns a PyArrow table:

import chalkcompute

if __name__ == "__main__":
    run = evaluation.run().wait()
    table = chalkcompute.DatasetClient().read(run.result_dataset)

result_dataset is the DatasetRevisionRef for that run. See Reading a dataset for read’s other arguments.

The result carries the dataset’s own columns, plus:

Column	Holds
output	the task’s return value for that row
output_error	the failure detail, when the task raised
<scorer>_value	that scorer’s score
<scorer>_error	the failure detail, when that scorer raised
<scorer>_metadata	whatever the scorer recorded alongside its score

Check the task and scorer error columns for every row. A run-level SUCCEEDED status means a result dataset was produced; it does not mean that every task and scorer call succeeded.

A SQL expression scorer contributes only <name>_value, since nothing is deployed to record an error or metadata against it.

The result also carries __chalk__-prefixed columns that the runtime uses to tie a row to its telemetry. These are internal and not part of the reference above.

table is an ordinary PyArrow table, so summarize or filter it in place:

import pyarrow.compute as pc

if __name__ == "__main__":
    print(pc.mean(table["intent-match_value"]).as_py())   # 0.5, on the Quick start rows
    print(table.filter(pc.equal(table["intent-match_value"], 0.0)))

table.to_pandas() works too, if you would rather go on in pandas.

Index a score column with brackets rather than an attribute. A scorer’s name becomes its column prefix, and a built-in names itself in kebab case, so intent-match_value is not a valid Python identifier.

That is the per-row detail. For a run’s average score, its per-scorer averages, and its cost and duration, use the dashboard, which computes those aggregates for you.

---

​

The Evaluations dashboard

The Chalk dashboard shows the same evaluations, runs and scores, and computes the aggregates for you. Click Evaluations in an environment’s sidebar to open the Evaluations page, which lists the environment’s evaluation suites. Each suite holds evaluation definitions and their runs. The list can be filtered by Created by and sorted by Created.

Create evaluation, at the top of the page, opens a dialog that creates and runs an evaluation without writing code. See Creating an evaluation in the dashboard. Its menu has Use SDK, which shows example code that uploads a dataset, creates an evaluation, and runs it. Create suite adds a suite.

In an environment with no evaluations, Evaluations shows a Get started with evaluations checklist in place of the suites. Its steps are to create a dataset, define task and scorer functions, and create an evaluation. All three are checked off together, once the environment has its first evaluation.

​

Suites and runs

Open a suite on the Evaluations page. Its Overview tab shows a chart of the suite’s runs, the runs table, and the suite’s evaluation definitions. A Details panel lists the suite’s name, description, creator, datasets and definitions.

The runs table gives each run’s status, average score with the spread of its row scores, duration, LLM calls, tool calls, tokens, and estimated cost; Columns chooses which appear. Filter the runs by Status or Evaluation, and click Refresh to reload them. The table loads 100 runs at a time, and Load next 100 loads more. The chart and the table’s sorting cover only the runs loaded so far.

The chart plots the runs whose Plot box is checked in the runs table. Performance compares their scores, and Token usage, Cost analysis, Execution time and Tool use compare the rest. Series picks the fields to plot, such as the average score and each scorer’s score. Aggregate plots each field’s average, and Percentiles plots its 5th, 25th, 50th, 75th and 95th percentiles across the run’s rows.

Right-click a run in the runs table for two actions:

• Set as baseline marks the run with a Baseline badge. An environment has at most one baseline run, so setting a new one clears the previous one.
• Archive removes the run from the list.

The Average score of a run or evaluation leaves out scorers whose values fall outside 0 to 1, such as a SQL expression scorer that returns a cost in dollars. The dashboard shows those scorers as plain numbers.

​

Evaluation pages

Click an evaluation to open its page. Run evaluation starts a new run, and the tabs show:

• Runs: the evaluation’s runs.
• Scorers: each scorer’s average score and the distribution of its scores in the evaluation’s latest run.
• Task: the task function’s source, or the task’s SQL expression.
• Definition: the chalkcompute.Evaluation.create(...) call that recreates the evaluation on the same dataset revision. With Function names, the default, the call refers to the task and function scorers by name, so it uses their current versions. Pinned versions switches it to the exact versions the evaluation used.

Edit scorers adds, changes or removes scorers. Because an evaluation’s scorers are fixed when it is created, saving creates a copy of the evaluation with the same dataset revision, task and suite. It can also rescore the latest succeeded run into the copy, which calls only the new and changed scorers. A scorer in the editor is an LLM judge, a Python function, or a Chalk SQL expression.

Compare opens the evaluation in a table where you add other evaluations side by side. Each column is one evaluation, averaged over its succeeded runs. Alongside the scores, the table compares cost, tokens, duration, and LLM calls, and reads every value against the first column, where cheaper and faster count as improvements. A column whose dataset revision differs from the first column’s is labeled “different dataset”.

​

Run pages

Click a run to open it. A Details panel lists the run’s ID, status, duration, timestamps, creator, result dataset and result dataset revision, followed by a summary of its average score, LLM and tool calls, tokens, and estimated cost. The run page keeps the evaluation’s Scorers, Task and Definition tabs, and adds:

• Results: the run’s result rows, as described in Reading results. Columns holding JSON objects, such as a structured task output or a scorer’s metadata, are split into one column per field.
• Summary: the run’s average score, row count, wall time, total cost, tokens, LLM and tool calls, and median row time, then a timeline of when each row ran and a table of rows.
• Run trace: a waterfall of the run’s calls, with one bar for each row’s task call and that row’s scorer calls beneath it.
• Sandboxes: any sandboxes the task started for the run, each linked to the result row it served.

Click a row on Results or Summary, or a bar on Run trace, to open that row in a drawer. The drawer opens on the Task run tab, which shows the row’s task duration, cost, tokens, and LLM and tool calls, followed by its scores, input, and output. The drawer’s other tabs, such as Flame graph and Waterfall, show the row’s trace.

Wall time runs from when the run was created to when it completed, so it includes any wait before the run started. Duration, in the Details panel and the runs table, covers only the time the run’s work took.

A run that is still in progress shows its rows as they arrive.

The Rerun icon beside the run’s title opens the evaluation’s saved definition in a read-only dialog. Rerun evaluation starts another run of the same evaluation and opens its results.

​

Showing images

When a task takes or returns images, an evaluation can tell the dashboard which columns hold them. The row’s Task run tab then shows them side by side above its scores. Declare them under the chalk.media key of the evaluation’s metadata:

{
  "chalk.media": [
    {"role": "input", "label": "Photo", "column": "input_image"},
    {"role": "output", "label": "Edited", "column": "output", "path": "result_jpeg_b64", "mime": "image/jpeg"},
    {"role": "reference", "label": "Ideal edit", "column": "reference_image"}
  ]
}

• column is any result column: a dataset column, output, or a scorer’s column.
• path reaches into a JSON value with dots and [index], for example a field of the task’s JSON output.
• role is input, output, reference or other, and sets the order: input first.
• A value can be a data:image/...;base64, URI, an http(s) URL, raw base64 (the format is detected; mime covers formats that can’t be), bytes, or an object with a url, or with data and mime. PNG, JPEG, WebP, GIF, AVIF and BMP are shown; SVG is not.

Click an image to enlarge it. Dataset columns shown this way aren’t repeated as text under Input.

​

Creating an evaluation in the dashboard

Create evaluation on the Evaluations page creates an evaluation and starts its first run in one step. Name it, then pick a dataset and revision under Input dataset. To make a dataset in the dialog, use Quick create dataset to enter a few rows by hand, or Upload dataset to upload a file. Either one selects the new revision.

Choose a task:

• Function: a function you have already deployed, called once per row. Its parameters must match dataset columns by name.
• LLM prompt: a prompt template filled from each row’s columns, written as {{column}}, and sent to a model through the Model Gateway. Templates such as Answer a question and Classify text fill in a starting prompt. The task is deployed as a function when the evaluation is created.
• SQL expression: a Chalk SQL expression over the row’s columns. Nothing is deployed or called.

Then pick scorers. Built-in scorers offers the deterministic built-ins, embedding similarity, and four LLM judges: Factuality, Closed QA, Answer relevance, and Custom rubric, which grades against labels you define. Create new scorers adds scorer functions and Chalk SQL expressions. The dialog lists the parameters a scorer can take, the dataset’s columns plus output and trace, and the values it can return.

Create and run creates the evaluation in the environment’s Default suite and opens the run’s results. When the task or scorers need deploying, Generated evaluation module shows the Python module the dialog deploys.