Evaluation Datasets

Upload a dataset with DatasetClient.upload(name, data). It registers a dataset revision and returns a DatasetRevisionRef you pass straight to Evaluation.create. Your data is ordinary tabular data, so this needs neither ChalkPy nor feature definitions.

Pass any of the following as data:

• a CSV or Parquet path, or a sequence of them, whose Arrow schemas match exactly,
• a PyArrow table or record batch,
• a mapping of column name to values, or a sequence of row mappings,
• any dataframe exposing to_arrow or the dataframe interchange protocol.

CSV files and in-memory values are converted to temporary Parquet files locally before upload. Uploading under a name that already exists creates a new revision of that dataset.

import chalkcompute

client = chalkcompute.DatasetClient()

# From files
revision = client.upload("support-triage", ["jan.parquet", "feb.parquet"])

# From rows, with progress
revision = client.upload(
    "support-triage",
    [{"message": "where is my order?", "expected": '{"intent": "status", "amount": 0.0}'}],
    on_progress=lambda done, total: print(f"{done}/{total}"),
)

upload also accepts part_size_bytes to set the multipart chunk size and timeout to bound each request, in seconds.

Six column names are reserved by the evaluation runtime: output, trace, __chalk_evaluation_output, __chalk_evaluation_row_id, __chalk__session_id__ and __chalk__task_session_id__. A dataset carrying any of them is rejected by Evaluation.create with dataset column "<name>" is reserved by the evaluation runtime.

​

Reading a dataset

DatasetClient.read returns a revision as a PyArrow table. Pass it the DatasetRevisionRef that upload returned, or a revision id:

import chalkcompute

table = chalkcompute.DatasetClient().read(revision)

timeout bounds the request in seconds and defaults to 300. A revision with no output raises DatasetError, which is what you get for an evaluation run that is still going or that failed.

This is how you read a run’s scores as well; see Reading results for the columns a run writes.

​

Referring to a dataset

Pass Evaluation.create(dataset=...) the reference upload returned, or build a DatasetRevisionRef for a dataset that is already there:

import chalkcompute

# What upload returns: the revision it just registered
revision = chalkcompute.DatasetClient().upload("support-triage", "support_messages.csv")

# An exact revision, by id
chalkcompute.DatasetRevisionRef(dataset_id="...", revision_id="...")

# The latest revision of a dataset, resolved when the evaluation is created
chalkcompute.DatasetRevisionRef(dataset_name="support-triage")

A name resolves once, when you create the evaluation, and the resolved revision is then pinned. Later uploads under the same name leave existing evaluations alone. DatasetRevisionRef rejects a mix of the two forms: pass either dataset_name, or dataset_id together with revision_id.

create also takes any object exposing dataset_id and revision_id, which pins that revision, or one exposing dataset_name, whose latest revision it resolves. A dataset handle from another Chalk client therefore passes straight through.