Creating and Running Evaluations

The examples on this page continue the Quick start. To run them on their own, start a file with the code below, which deploys the Quick start’s task, uploads its dataset, and finds or creates a suite, and add the examples after it in order. As in the Quick start, definitions stay at the top level and the code that uploads data or creates and runs evaluations goes under if __name__ == "__main__":.

import chalkcompute

@chalkcompute.function
def triage(message: str) -> str:
    import json, re
    text = message.lower()
    if "refund" in text or "money back" in text:
        intent = "refund"
    elif "cancel" in text or "close my account" in text:
        intent = "cancel"
    elif "invoice" in text or "billed" in text or "card" in text:
        intent = "billing"
    elif "where is" in text or "waiting" in text or "arrived" in text:
        intent = "status"
    else:
        return "unclassified"
    amount = re.search(r"\d+(?:\.\d+)?", message)
    return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})

if __name__ == "__main__":
    dataset = chalkcompute.DatasetClient().upload("support-triage", [
        {"message": "I was charged twice for order 1182, please refund",
         "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
        {"message": "Where is my order? It was due Tuesday.",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
        {"message": "Please close my account and refund the last charge",
         "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
        {"message": "Following up on my last email",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
    ])

    # Suite names are unique in an environment, so reuse "Release" if it exists.
    suite = next(
        (s for s in chalkcompute.EvaluationSuite.list_all() if s.name == "Release"), None
    ) or chalkcompute.EvaluationSuite.create("Release")

---

​

Creating an evaluation

Pass the dataset, the task, and the scorers to Evaluation.create, along with any metadata you want stored on the definition:

import chalkcompute

valid_json = chalkcompute.scorers.json_valid(name="valid-json")

if __name__ == "__main__":
    evaluation = chalkcompute.Evaluation.create(
        "support-triage",
        dataset=dataset,
        task=triage,
        scorers=[valid_json],
        metadata={"owner": "support-eng"},
        suite_id=suite.id,
    )

name is required and must be non-empty, and at least one scorer is required. metadata must be a JSON-serializable mapping with string keys; Chalk stores it on the definition and returns it on every read. suite_id files the evaluation under a suite.

The returned Evaluation exposes the pinned identifiers as dataset_id, dataset_revision_id, task_function_version_id, and scorers, a tuple of EvaluationScorer carrying an evaluation-local id and then whichever of function_version_id, builtin_id, or sql_name and sql_expression matches how that scorer was defined.

Attach to an existing definition with chalkcompute.Evaluation.from_id("..."), and read the latest server state with evaluation.refresh().

---

​

Running an evaluation

evaluation.run() is an asynchronous process that queues a run and returns a handle immediately:

if __name__ == "__main__":
    run = evaluation.run(metadata={"git_sha": "abc123"})
    print(run.status)          # queued, not yet finished
    final = run.wait()
    print(final.status, final.result_dataset)

wait() polls until the run reaches a terminal state and returns the final handle:

Parameter	Type	Default	Description
timeout	float | None	600.0	Seconds to wait; None waits indefinitely. Exceeding it raises TimeoutError.
poll_interval	float	2.0	Seconds between status polls.
raise_on_failure	bool	True	Raise EvaluationRunFailedError for a failed or canceled run. Pass False to inspect the terminal handle yourself.

EvaluationRunStatus covers PENDING, RUNNING, FINALIZING, SUCCEEDED, FAILED, CANCELED, and UNSPECIFIED. run.is_terminal is true for the three that end a run: SUCCEEDED, FAILED, and CANCELED.

A successful run exposes run.result_dataset, a DatasetRevisionRef for the dataset revision holding the task output and each scorer’s score and metadata per row. It stays None until the run produces one. run.error_message carries the failure detail, and a raised EvaluationRunFailedError exposes the terminal handle as error.run. A run also carries created_at, started_at, completed_at, created_by, the ID of whoever started it, and result_dataset_id and result_dataset_revision_id, the two IDs inside result_dataset.

Read a definition’s history with evaluation.runs(limit=...), newest first, or attach to one run with chalkcompute.EvaluationRun.from_id("...") and run.refresh().

run() also takes execution_mode, which sets how Chalk carries a run to completion:

• "workflow": a workflow on your environment’s workflow orchestrator finishes the run, waits for its telemetry to arrive, and then computes its score and metric summaries. The run completes whether or not anything reads it.
• "on_read": the run advances when something reads it, such as wait(), refresh(), or the dashboard.

Without execution_mode, Chalk uses "workflow" when your environment has a workflow orchestrator and "on_read" when it does not. run.execution_mode holds the mode a run was given from the moment it is queued. A run that asks for "workflow" in an environment without an orchestrator is rejected. chalkcompute.EvaluationRunExecutionMode names the same two values.

---

​

Identifying the run from a task or scorer

Each call a run makes to its task or to a function scorer carries two headers: x-chalk-evaluation-id, the evaluation, and x-chalk-evaluation-run-id, the run the call serves. Read them inside the function with chalkcompute.get_call_context():

import chalkcompute

@chalkcompute.function
def which_run(message: str) -> str:
    context = chalkcompute.get_call_context()
    return f'{context.get("x-chalk-evaluation-id")} {context.get("x-chalk-evaluation-run-id")}'

get_call_context() returns an empty mapping when the function is called outside Chalk, so both lookups return None there. During a rescore, x-chalk-evaluation-run-id is the rescored run.

---

​

Rescoring a run

The task is usually the slow, expensive half of an evaluation, and it stays the same while you iterate on how its outputs are graded. run.rescore() scores a finished run again and calls only what changed since that run. It reads the run’s rows back out of its result dataset, copies the task outputs and scores that are unchanged, and calls the rest:

import chalkcompute

intent_match = chalkcompute.scorers.json_match(
    inputs=("output", "expected"), paths=["intent"], name="intent-match"
)

if __name__ == "__main__":
    run = chalkcompute.EvaluationRun.from_id("...")

    # Score the same outputs with new scorers.
    rescored = run.rescore(scorers=[valid_json, intent_match]).wait()

An evaluation’s scorers are fixed when it is created, so rescore(scorers=...) creates a new evaluation. It copies the run’s dataset revision, task and suite, replaces the scorers, and records the rescored run under it. That evaluation is named "<evaluation name> (rescored)" unless you pass name=. Because the new evaluation is a normal one, later calls to evaluation.run() on it do run the task.

To rescore into an evaluation that already exists, pass it instead:

if __name__ == "__main__":
    strict = chalkcompute.Evaluation.from_id("...")
    rescored = run.rescore(evaluation=strict)   # or strict.rescore(run)

That evaluation must use the run’s dataset revision, since the rescored run reads its rows out of the run’s results. The server rejects any other dataset revision. The evaluation’s task and scorers can differ from the run’s, and the rescore compares them with the run’s to decide what to call:

• If the evaluation uses the run’s task, each row’s output is copied from the run. Otherwise the evaluation’s task is called on the same rows. A run started by a post-training called the policy model in place of the task, so rescoring one always calls the task.
• A scorer is copied when the run had the same scorer: the same function version, or a SQL scorer with the same expression. A copied function scorer is still called on the rows where its call failed in the run.
• When the task is called again, a scorer that takes output or trace, and every SQL scorer, is called again too.
• Every other scorer is called on every row.

run.rescore() with no arguments rescores into the run’s own evaluation, so it calls nothing except each function scorer on the rows where that scorer failed.

Only a run that succeeded has outputs to rescore. The rescored run is an ordinary run of its evaluation: wait(), result_dataset and the dashboard all treat it like any other. Each row’s new calls go to a session of their own, and a scorer that takes trace reads the spans of the task that produced the row’s output: the original task’s when it was copied, the new call’s otherwise.