Evaluations

---

​

Overview

An evaluation is defined by three things: a completed dataset revision, one task, and one or more scorers. The task is usually a deployed @chalkcompute.function version. In the dashboard, Quick evaluation can also make it an LLM prompt, which it deploys as a function for you, or a Chalk SQL expression. A scorer is a deployed function version or a Chalk SQL expression. Creating an evaluation saves that definition in the active environment. Running it queues a separate execution that calls the task once per dataset row, passes each output to every scorer, and writes the scores to a result dataset.

The definition is immutable, so two runs of the same evaluation stay comparable: the data, the task, and the scorers are fixed at creation time. To evaluate a new version of the task, create a new evaluation.

┌──────────────────┐   define                 execute
│ dataset revision │──┐       ┌──────────────┐       ┌─────────────────┐
│ task version     │──┼──────▸│  Evaluation  │──────▸│  EvaluationRun  │
│ scorer versions  │──┘       │  immutable   │       │   repeatable    │
└──────────────────┘          └──────────────┘       └────────┬────────┘
   pinned inputs                                              │ produces
                                                              ▾
                                                   result dataset revision

Function tasks and scorers are ordinary Chalk Functions, so everything on that page applies to them: resource configuration, secrets, retries, rate limits, and tracing.

Evaluations are offline. They score a pinned dataset revision, not live traffic.

Before your first run, ask Chalk support to enable Evaluations and the async Function queue in your environment. A new environment also needs an active deployment. The Quick start walks through the remaining setup and shows how to check the result.