Using EvaluationClient

The class methods on the other Evaluations pages, such as Evaluation.create and evaluation.run(), each construct a client for a single call. Hold an EvaluationClient when you make several calls, or when you want it to borrow credentials from a ConnectClient or a ChalkPy client you already have:

import chalkcompute

client = chalkcompute.EvaluationClient(chalk_client=my_chalk_client)

suite = client.create_suite("Release")
evaluation = client.create("Nightly", dataset=dataset, task=answer, scorers=[exact_match])
run = client.run(evaluation.id, metadata={"git_sha": "abc123"})

for candidate in client.list(suite_id=suite.id, limit=50):
    print(candidate.name)

for previous in client.list_runs(evaluation_id=evaluation.id, limit=10):
    print(previous.id, previous.status)

The client carries create_suite and list_suites for suites, create, get, and list for definitions, and run, get_run, list_runs, and rescore for runs. client.rescore(run_id, evaluation_id=...) is the call behind run.rescore(), and evaluation_id defaults to the run’s own evaluation. list and list_runs are generators that page in batches of up to 100 and stop at limit. Pass only one of chalk_client or connect_client to the constructor.

Every object a client returns keeps a reference to that client, which is what makes evaluation.run(), run.refresh(), and suite.evaluations() work. A handle you construct yourself has no client and raises EvaluationError on those methods, so fetch it through the client instead.

---

​

Errors

A failed request raises EvaluationError, which carries status_code, the HTTP equivalent of the underlying code. A missing evaluation or run raises EvaluationNotFoundError, a subclass of it:

import chalkcompute

try:
    evaluation = chalkcompute.Evaluation.from_id("does-not-exist")
except chalkcompute.EvaluationNotFoundError as e:
    print(e.status_code)  # 404

A run that fails or is canceled raises EvaluationRunFailedError out of wait(), with the terminal handle attached so you can read its status and error message. In this example, evaluation is an evaluation created as on the Creating and Running page:

try:
    run = evaluation.run().wait()
except chalkcompute.EvaluationRunFailedError as e:
    print(e.run.status, e.run.error_message)

wait() raises TimeoutError when it runs out of time, as does a Trace read that never settles. Local validation raises ValueError or TypeError for an empty name, an empty scorer list, metadata that is not JSON-serializable, a score outside [0, 1], or a bare callable passed as a task or scorer. That validation runs before any request goes out, so a malformed call fails locally.