Evaluation Scorers

A scorer reads one row and returns a number between 0 and 1. chalkcompute.scorers ships twelve of them, and you can write your own. Most evaluations mix several.

The examples on this page continue the Quick start. To run them on their own, start a file with the code below, which deploys the Quick start’s task and uploads its dataset, and add an example after it. As in the Quick start, definitions stay at the top level and the code that uploads data or runs an evaluation goes under if __name__ == "__main__":.

import chalkcompute

@chalkcompute.function
def triage(message: str) -> str:
    import json, re
    text = message.lower()
    if "refund" in text or "money back" in text:
        intent = "refund"
    elif "cancel" in text or "close my account" in text:
        intent = "cancel"
    elif "invoice" in text or "billed" in text or "card" in text:
        intent = "billing"
    elif "where is" in text or "waiting" in text or "arrived" in text:
        intent = "status"
    else:
        return "unclassified"
    amount = re.search(r"\d+(?:\.\d+)?", message)
    return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})

if __name__ == "__main__":
    dataset = chalkcompute.DatasetClient().upload("support-triage", [
        {"message": "I was charged twice for order 1182, please refund",
         "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
        {"message": "Where is my order? It was due Tuesday.",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
        {"message": "Please close my account and refund the last charge",
         "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
        {"message": "Following up on my last email",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
    ])

---

​

Choosing a scorer

The built-in scorers differ in how they arrive at a score:

• Deterministic comparisons grade one column against another by a fixed rule, covering text, numbers, JSON, and regular expressions. They are cheap and repeatable, and they are the place to start.
• Model-based scorers judge meaning rather than form. embedding_similarity catches an answer that is right but worded differently, and llm_judge grades qualities you can describe but not compute, such as tone or whether an answer actually helped. jev grades with a fixed set of labels and sends the rows it is unsure of to a second model.
• SQL expressions score from the dataset’s own columns through scorers.sql, the one built-in that deploys nothing.

Writing your own function is the fourth option, and the only one outside chalkcompute.scorers. Reach for it for business rules, and anything that needs real computation.

​

Built-in scorers

Every built-in is a function you call. Calling it deploys the scorer and returns a handle to pass to scorers=, so call it even when you change none of its options. Call it at the top level of the file, as with your own functions. scorers.sql is the exception, deploying nothing.

llm_judge, jev and sql are configured differently from the rest, and have their own sections below. The options for the other nine follow the table.

import chalkcompute

scorers = [
    chalkcompute.scorers.json_valid(inputs=("output",), name="valid-json"),
    chalkcompute.scorers.json_match(inputs=("output", "expected"), name="full-match"),
    chalkcompute.scorers.json_match(
        inputs=("output", "expected"), paths=["intent"], name="intent-match"
    ),
    chalkcompute.scorers.contains(inputs=("output", "expected_intent"), name="mentions-intent"),
]

if __name__ == "__main__":
    evaluation = chalkcompute.Evaluation.create(
        "support-triage",
        dataset=dataset,
        task=triage,
        scorers=scorers,
    )

On the four rows from the Quick start, those four score valid-json 0.75, intent-match 0.5, mentions-intent 0.5 and full-match 0.25. full-match is lowest because it compares the whole document, so the wrong amount in the first row counts against it even though the intent is right.

Scorer	What it measures
exact_match	Two columns hold the same text
contains	Expected substrings appear in a column
regex_match	A column matches a regular expression
levenshtein	Edit distance, normalized into a score
string_similarity	A named similarity measure, such as Jaro-Winkler
json_valid	A column parses as JSON
json_match	Two columns agree as JSON documents
numeric_close	Two columns agree as numbers
embedding_similarity	Two columns mean the same thing
llm_judge	A quality you describe in a pydantic model
jev	A label a model picks, regraded by a second model when the first is unsure
sql	A Chalk SQL expression you write yourself

Each of the nine keyword-configured scorers takes name, which labels the scorer, and inputs, which names the columns it reads. inputs defaults to the columns that scorer needs: ("output", "expected") for the seven that compare two columns, and ("output",) for regex_match and json_valid, which read one. Any remaining keywords go to @chalkcompute.function, so a built-in takes the same image, resource, and retry options as any other Chalk Function. Each one records metadata explaining its score.

The options worth knowing per scorer:

• exact_match strips each side by default. normalize_whitespace collapses runs of whitespace instead, and case_insensitive folds case.
• contains reads one substring, or a JSON array of them, from the reference column. mode scores them together: "all" demands every one, "any" one of them, and "fraction" gives the share that appeared.
• regex_match takes its pattern positionally, and full_match requires the pattern to cover the whole column. Named groups are recorded as metadata, so a pattern can also pull a value out of the row.
• string_similarity takes its measure positionally: "jaro_winkler", "jaccard", "token_set", "token_sort", "partial", or "sequence".
• numeric_close with a tolerance scores 1.0 or 0.0 on whether the two are within it, taken as a share of the larger value when relative. Without one, the score falls off with the gap between them.
• json_match compares whole documents unless you pass paths, so object key order does not matter but list order does. With paths, written dotted as in "user.id", the score is the share of those paths whose values agree.
• embedding_similarity embeds both columns with model, defaulting to text-embedding-3-small, and scores them by cosine similarity. It catches an answer that is right but worded differently, which the string scorers mark wrong. base_url and api_key name the endpoint and the Secret holding its key.

​

LLM judges

chalkcompute.scorers.llm_judge deploys a scorer described by a pydantic model (v1 or v2) instead of a function body. The model’s score field becomes the score, and every other field is recorded as row metadata. The judge requests the reply with structured outputs and validates it against the model, so a malformed grade fails the row.

With no api_key and no base_url, the judge calls the Model Gateway as your environment, so it needs no provider key. model names a chat model the gateway serves; run chalk router model list to list them.

This example grades a browser agent instead of the Quick start’s task, so it goes in a file of its own:

import chalkcompute
from pydantic import BaseModel, Field

class TraceQuality(BaseModel):
    score: float = Field(ge=0, le=1, description="Overall quality.")
    directness: int = Field(ge=0, le=2, description="Shortest reasonable path to the goal.")
    task_correctness: int = Field(ge=0, le=2, description="Was the task actually completed?")
    reason: str

trace_quality = chalkcompute.scorers.llm_judge(
    TraceQuality,
    model="openai/gpt-5",
    instructions="You are grading a browser agent's login attempt.",
)

The scorer deploys under the model’s class name in kebab case, so TraceQuality becomes trace-quality. Pass name= to choose another. The remaining keywords:

• inputs names the dataset columns the judge reads, defaulting to ("output",).
• prompt_fn replaces the default prompt, and parse_fn replaces structured outputs for an endpoint that does not support them.
• completion_kwargs, for example {"temperature": 0, "max_tokens": 400}, are sent with every request.
• base_url points at another OpenAI-compatible endpoint, and api_key takes a Secret holding its key, which the judge reads in the pod. With either one set, the judge calls that endpoint instead of the Model Gateway, and api_key on its own calls OpenAI.
• image supplies a base image, and any other keyword goes to @chalkcompute.function.

​

Escalating judges

chalkcompute.scorers.jev is a judge that grades each row with one of a fixed set of labels, and passes the rows it is unsure of to a second model. model picks a label for the row and gives a probability for every label. When its probability on the label it picked is below threshold, which defaults to 0.9, fallback_model grades the row again and its grade stands.

import chalkcompute

intent_judge = chalkcompute.scorers.jev(
    labels={"correct": 1.0, "partial": 0.5, "wrong": 0.0},
    rubric=(
        "correct: the intent in the output is what the customer asked for. "
        "partial: the intent is close, or the output names the right intent but is not valid JSON. "
        "wrong: anything else."
    ),
    model="openai/gpt-5-mini",
    fallback_model="openai/gpt-5",
    inputs=("message", "output"),
    name="intent-judge",
)

if __name__ == "__main__":
    evaluation = chalkcompute.Evaluation.create(
        "support-triage",
        dataset=dataset,
        task=triage,
        scorers=[intent_judge],
    )

labels maps each label to its score, between 0 and 1, and needs at least two. rubric says what each label means. Labels match regardless of case, and a reply that picks a label outside labels scores 0 and goes to fallback_model.

Each row’s metadata records the label in choice, the model’s rationale, the normalized probabilities, confidence in the chosen label, whether the row was escalated, the judge_model whose grade stands, and primary_confidence, the first model’s confidence. The probabilities are what the model reports about its own confidence. They are not calibrated, so the share of rows a given threshold escalates depends on the model.

name defaults to jev and inputs to ("output",). instructions, base_url, api_key, completion_kwargs and image work as they do for llm_judge, including calling the Model Gateway when neither api_key nor base_url is set.

​

SQL expression scorers

scorers.sql scores a row with a Chalk SQL expression rather than a deployed function. Nothing is deployed, so there is no image to build and no cold start.

import chalkcompute

answered = chalkcompute.scorers.sql(
    "answered",
    '''CASE WHEN "output" IS NOT NULL AND "output" <> '' THEN 1.0 ELSE 0.0 END''',
)

The expression is a select item over the evaluation’s own columns: every dataset column by name, plus output, the task’s result for that row. It must yield a number. name labels the result column and the row in the compare table.

Quote column names, since output and other bare identifiers can collide with SQL keywords.

​

Writing your own scorer

Write a scorer function the same way as a task function. Its parameters bind to dataset columns by name, plus one argument the runner supplies: output, the task’s return value for that row.

import chalkcompute

@chalkcompute.function
def amount_is_plausible(message: str, output: str) -> float:
    import json
    try:
        amount = json.loads(output)["amount"]
    except Exception:
        return 0.0
    return 1.0 if amount == 0.0 or str(amount).rstrip("0").rstrip(".") in message else 0.0

Here message comes from the dataset and output comes from the task. A scorer can read any column in the dataset, not only the one holding the expected answer, which is what lets this one check the extracted amount against the message it came from.

​

Scorer results

A scorer must return a floating point number between 0 and 1 inclusive, one EvaluationScorerResult, or a list[EvaluationScorerResult]. A list lets one scorer emit several scores from shared computation, such as a single model call you grade on three axes. An empty list emits no scores for that row.

import chalkcompute

@chalkcompute.function
def per_field(output: str, expected: str) -> list[chalkcompute.EvaluationScorerResult]:
    import json
    try:
        got, want = json.loads(output), json.loads(expected)
    except ValueError:
        return []
    return [
        chalkcompute.EvaluationScorerResult(
            score=1.0 if got.get(field) == value else 0.0,
            metadata={"field": field},
        )
        for field, value in want.items()
    ]

EvaluationScorerResult takes a score and optional metadata:

• score must be a finite number between 0 and 1, inclusive. A boolean raises TypeError, and a score outside the range raises ValueError. This check runs when EvaluationScorerResult is constructed, so returning the dataclass is what enforces the range. A scorer returning a plain number, and a SQL expression scorer, are not checked.
• metadata must be JSON-serializable, with no NaN or infinity. Chalk stores it per row alongside the score.

The return annotation declares the Arrow schema for the function, and EvaluationScorerResult carries its own Arrow contract. Return the dataclass rather than a plain dictionary, which has no such contract and fails to serialize.