Reading and Comparing Evaluation Results

The examples on this page continue the Quick start and the built-in scorers example on the Scorers page. To run them on their own, start a file with the code below, which creates that evaluation, and add the examples after it in order. As in the Quick start, definitions stay at the top level and the code that uploads data or creates and runs evaluations goes under if __name__ == "__main__":.

import chalkcompute

@chalkcompute.function
def triage(message: str) -> str:
    import json, re
    text = message.lower()
    if "refund" in text or "money back" in text:
        intent = "refund"
    elif "cancel" in text or "close my account" in text:
        intent = "cancel"
    elif "invoice" in text or "billed" in text or "card" in text:
        intent = "billing"
    elif "where is" in text or "waiting" in text or "arrived" in text:
        intent = "status"
    else:
        return "unclassified"
    amount = re.search(r"\d+(?:\.\d+)?", message)
    return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})

scorers = [
    chalkcompute.scorers.json_valid(inputs=("output",), name="valid-json"),
    chalkcompute.scorers.json_match(inputs=("output", "expected"), name="full-match"),
    chalkcompute.scorers.json_match(
        inputs=("output", "expected"), paths=["intent"], name="intent-match"
    ),
    chalkcompute.scorers.contains(inputs=("output", "expected_intent"), name="mentions-intent"),
]

if __name__ == "__main__":
    dataset = chalkcompute.DatasetClient().upload("support-triage", [
        {"message": "I was charged twice for order 1182, please refund",
         "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
        {"message": "Where is my order? It was due Tuesday.",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
        {"message": "Please close my account and refund the last charge",
         "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
        {"message": "Following up on my last email",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
    ])

    evaluation = chalkcompute.Evaluation.create(
        "support-triage",
        dataset=dataset,
        task=triage,
        scorers=scorers,
    )

---

​

Reading results

A run writes one row per dataset row into its result dataset. Read it with DatasetClient.read, which returns a PyArrow table:

import chalkcompute

if __name__ == "__main__":
    run = evaluation.run().wait()
    table = chalkcompute.DatasetClient().read(run.result_dataset)

result_dataset is the DatasetRevisionRef for that run. See Reading a dataset for read’s other arguments.

The result carries the dataset’s own columns, plus:

Column	Holds
output	the task’s return value for that row
output_error	the failure detail, when the task raised
<scorer>_value	that scorer’s score
<scorer>_error	the failure detail, when that scorer raised
<scorer>_metadata	whatever the scorer recorded alongside its score

A SQL expression scorer contributes only <name>_value, since nothing is deployed to record an error or metadata against it.

The result also carries __chalk__-prefixed columns that the runtime uses to tie a row to its telemetry. These are internal and not part of the reference above.

table is an ordinary PyArrow table, so summarize or filter it in place:

import pyarrow.compute as pc

if __name__ == "__main__":
    print(pc.mean(table["intent-match_value"]).as_py())   # 0.5, on the Quick start rows
    print(table.filter(pc.equal(table["intent-match_value"], 0.0)))

table.to_pandas() works too, if you would rather go on in pandas.

Index a score column with brackets rather than an attribute. A scorer’s name becomes its column prefix, and a built-in names itself in kebab case, so intent-match_value is not a valid Python identifier.

That is the per-row detail. For a run’s average score, its per-scorer averages, and its cost and duration, use the dashboard, which computes those aggregates for you.

---

​

Comparing runs in the dashboard

Evaluations, runs and scores also appear in the Chalk dashboard, on an environment’s Evaluations page. Three things live there and nowhere else.

Compare evaluations side by side. Each column is one evaluation, averaged over its succeeded runs. Alongside the scores, the table compares cost, tokens, duration, and LLM calls, and reads every value against the first column, where cheaper and faster count as improvements.

Choose a baseline run. At most one run per environment can be the baseline. Picking the run that represents what you ship turns the evaluation into a release gate, since every later run is then read as an improvement or a regression against that fixed point.

Archive a run. Archiving removes a run from the list. A rescored run lists and reads like any other.

The dashboard also shows the summaries behind those comparisons: an average per run, an average per scorer, and how the scores are distributed across a run’s rows.