Evaluations Quick Start

This example uploads a small dataset, deploys a task function, and runs an evaluation that scores the task’s output with one built-in scorer.

---

import chalkcompute
import pyarrow.compute as pc

@chalkcompute.function
def triage(message: str) -> str:
    import json, re
    text = message.lower()
    if "refund" in text or "money back" in text:
        intent = "refund"
    elif "cancel" in text or "close my account" in text:
        intent = "cancel"
    elif "invoice" in text or "billed" in text or "card" in text:
        intent = "billing"
    elif "where is" in text or "waiting" in text or "arrived" in text:
        intent = "status"
    else:
        return "unclassified"
    amount = re.search(r"\d+(?:\.\d+)?", message)
    return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})

valid_json = chalkcompute.scorers.json_valid(name="valid-json")

if __name__ == "__main__":
    dataset = chalkcompute.DatasetClient().upload("support-triage", [
        {"message": "I was charged twice for order 1182, please refund",
         "expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
        {"message": "Where is my order? It was due Tuesday.",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
        {"message": "Please close my account and refund the last charge",
         "expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
        {"message": "Following up on my last email",
         "expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
    ])

    evaluation = chalkcompute.Evaluation.create(
        "support-triage",
        dataset=dataset,
        task=triage,
        scorers=[valid_json],
    )

    run = evaluation.run(metadata={"git_sha": "abc123"}).wait()
    print(run.status)

    table = chalkcompute.DatasetClient().read(run.result_dataset)
    print(pc.mean(table["valid-json_value"]).as_py())

Save it as a file, such as quick_start.py, and run it with python quick_start.py:

EvaluationRunStatus.SUCCEEDED
0.75

Chalk imports this file inside every function it deploys, here triage and the valid-json scorer, and the import runs the file’s top-level code. That gives the file two parts:

• Functions and scorers are defined at the top level, so each deployed function can find its own definition when it imports the file.
• The code that uploads the dataset and creates and runs the evaluation goes under if __name__ == "__main__":, so it runs only when you run the file.

Rows inline are one of the forms upload accepts, so there is no file to prepare first. A row needs whatever columns the task and scorers read: message for the task, and expected and expected_intent for the scorers on the Scorers page.

triage reads the first number it finds, so the order number in the first row becomes a dollar amount. It checks for “refund” before “cancel”, so the third row is classified as a refund. It returns text that is not JSON when nothing matches, as in the fourth. That last row is the one valid-json marks down, giving the 0.75 above.

The other Evaluations pages continue with that example, through scorers and on to reading the results. Where a feature is clearer with something else, such as LLM judges, those sections use an example of their own.