Evaluation Suites

A suite is a named collection of evaluations and their runs, such as a release gate, a regression set, or one team’s work.

Suite names are unique in an environment, and EvaluationSuite.create raises EvaluationError for a name that is already taken. To reuse a suite across runs of the same script, look it up by name and create it only when it is missing:

import chalkcompute

suites = chalkcompute.EvaluationSuite.list_all()
suite = next((s for s in suites if s.name == "Release"), None) or chalkcompute.EvaluationSuite.create(
    "Release"
)

for existing in suites:
    print(existing.id, existing.name)

for evaluation in suite.evaluations(limit=20):
    print(evaluation.name)

for run in suite.runs(limit=20):
    print(run.id, run.status)

suite.evaluations() and suite.runs() both yield newest first and page transparently.