Evaluations Quick Start

This example uploads a small dataset, deploys a task function, and runs an evaluation that scores the task’s output with one built-in scorer.

---

​

Before you start

1. Select an environment with Evaluations, an authenticated Function queue, and dataset storage configured. Managed self-signup serverless environments enable Evaluations and Function queue authentication by default.

2. Install the Chalk CLI, run chalk login, and select the intended environment with chalk environment <environment-id>. Check it with chalk user. A Chalk Sandbox with managed Chalk identity enabled works instead.

3. You do not need to deploy feature definitions or resolvers. The task and scorer functions in this example are provisioned independently.

4. In the Python environment that will run the example, install:

   pip install chalkcompute pyarrow
   

You can run this example from your own machine. A Chalk Sandbox is optional: it hosts the Python client while Chalk runs the task and scorer Functions remotely.

---

import chalkcompute
import pyarrow.compute as pc

@chalkcompute.function
def triage(message: str) -> str:
    import json, re
    text = message.lower()
    if "refund" in text or "money back" in text:
        intent = "refund"
    elif "cancel" in text or "close my account" in text:
        intent = "cancel"
    elif "invoice" in text or "billed" in text or "card" in text:
        intent = "billing"
    elif "where is" in text or "waiting" in text or "arrived" in text:
        intent = "status"
    else:
        return "unclassified"
    amount = re.search(r"\d+(?:\.\d+)?", message)
    return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})

valid_json = chalkcompute.scorers.json_valid(name="valid-json")

if __name__ == "__main__":
    dataset = chalkcompute.DatasetClient().upload("support-triage", [
        {"message": "I was charged twice for order 1182 ($49.99), please refund",
         "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
        {"message": "Where is my order? It was due Tuesday.",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
        {"message": "Please close my account and refund the last $12 charge",
         "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
        {"message": "Following up on my last email",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
    ])

    evaluation = chalkcompute.Evaluation.create(
        "support-triage",
        dataset=dataset,
        task=triage,
        scorers=[valid_json],
    )

    run = evaluation.run(metadata={"git_sha": "abc123"}).wait()
    print(run.status)

    table = chalkcompute.DatasetClient().read(run.result_dataset)
    print(pc.mean(table["valid-json_value"]).as_py())

Save it as a file, such as quick_start.py, and run it with python quick_start.py:

EvaluationRunStatus.SUCCEEDED
0.75

Check that the result dataset has four rows, then inspect output_error and valid-json_error for row failures. A run can finish with SUCCEEDED while some rows have errors. Save the evaluation and run IDs if you need to replay the pinned definition or ask support to investigate. To replay it, call run() on the existing evaluation rather than uploading the same inputs again; see Running an evaluation.

If a run fails with CHALK_FUNCTION_QUEUE_ENDPOINT is empty or unset, check that the environment has an authenticated Function queue configured.

Chalk imports this file inside every function it deploys, here triage and the valid-json scorer, and the import runs the file’s top-level code. That gives the file two parts:

• Functions and scorers are defined at the top level, so each deployed function can find its own definition when it imports the file.
• The code that uploads the dataset and creates and runs the evaluation goes under if __name__ == "__main__":, so it runs only when you run the file.

Rows inline are one of the forms upload accepts, so there is no file to prepare first. A row needs whatever columns the task and scorers read: message for the task, and expected and expected_intent for the scorers on the Scorers page.

triage reads the first number it finds, so the order number in the first row becomes a dollar amount. It checks for “refund” before “cancel”, so the third row is classified as a refund. It returns text that is not JSON when nothing matches, as in the fourth. That last row is the one valid-json marks down, giving the 0.75 above.

To create and run an evaluation from the Chalk dashboard without writing code, click Create evaluation on the Evaluations page. See Creating an evaluation in the dashboard.

To find out what the 0.75 is missing and improve it, continue with the tutorial.

The other Evaluations pages continue with that example, through scorers and on to reading the results. Where a feature is clearer with something else, such as LLM judges, those sections use an example of their own.