# Evaluations
source: https://docs.chalk.ai/docs/compute/evaluations

## Evaluate any LLM workload against historical production data and compare outcomes based on criteria you define.

### Overview

An evaluation is defined by three things: a completed dataset revision, one
task, and one or more scorers. The task is usually a deployed
@chalkcompute.function version. In the dashboard,
Create evaluation can also make it
an LLM prompt, which it deploys as a function for you, or a Chalk SQL expression.
A scorer is a deployed function version or a
Chalk SQL expression.
Creating an evaluation saves that definition in the active environment.
Running it queues a separate execution that calls the task once per dataset row,
passes each output to every scorer, and writes the scores to a result dataset.

The definition is immutable, so two runs of the same evaluation stay comparable:
the data, the task, and the scorers are fixed at creation time. To evaluate a new
version of the task, create a new evaluation.

Evaluations are grouped into evaluation suites. Each suite holds evaluation
definitions and their runs, such as a release gate or one team's work. An
evaluation created without a suite goes in the environment's Default suite.
In the dashboard, the Evaluations page lists the suites. See
Evaluation Suites.

```
┌──────────────────┐   define                 execute
│ dataset revision │──┐       ┌──────────────┐       ┌─────────────────┐
│ task version     │──┼──────▸│  Evaluation  │──────▸│  EvaluationRun  │
│ scorer versions  │──┘       │  immutable   │       │   repeatable    │
└──────────────────┘          └──────────────┘       └────────┬────────┘
   pinned inputs                                              │ produces
                                                              ▾
                                                   result dataset revision
```

Function tasks and scorers are ordinary Chalk Functions, so
everything on that page applies to them: resource configuration, secrets,
retries, rate limits, and tracing.

Evaluations are offline. They score a pinned dataset revision, not live traffic.

Before your first run, ask Chalk support to enable Evaluations and the async
Function queue in your environment. A new environment also needs an active
deployment. The Quick start walks through the
remaining setup and shows how to check the result.