Evaluations
Evaluate a support ticket router. Read the rows behind its score, add a scorer that checks the routing, and fix the router.
Suppose you are building a support ticket router. Each customer message comes in, and your code decides what kind of request it is (a refund, a cancellation, a billing question, or an order status check) so that it goes to the team that handles it. Before you ship a change to the router, you want to know two things: whether it sends messages to the right team, and whether the new version does that better than the old one. An evaluation answers both.
The Quick start built a first version of that
router, triage, and evaluated it on four messages, with a score of 0.75. This
tutorial picks up from there. You will look at the rows behind the score, add a
scorer that checks whether each message was routed correctly, fix the router,
and compare the two versions. Along the way you will see that a score only covers
what you grade for.
You need the setup from the Quick start’s Before you start section. The tutorial reuses the Quick start’s task and dataset, so there is nothing else to prepare.
Save this file as improve_score.py. It is the Quick start with one addition:
show prints each row’s scores next to the task’s output, and then the average
of each score.
import chalkcompute
import pyarrow.compute as pc
@chalkcompute.function
def triage(message: str) -> str:
import json, re
text = message.lower()
if "refund" in text or "money back" in text:
intent = "refund"
elif "cancel" in text or "close my account" in text:
intent = "cancel"
elif "invoice" in text or "billed" in text or "card" in text:
intent = "billing"
elif "where is" in text or "waiting" in text or "arrived" in text:
intent = "status"
else:
return "unclassified"
amount = re.search(r"\d+(?:\.\d+)?", message)
return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})
valid_json = chalkcompute.scorers.json_valid(name="valid-json")
def show(run, *scorers):
table = chalkcompute.DatasetClient().read(run.result_dataset)
for row in table.to_pylist():
print([row[f"{name}_value"] for name in scorers], row["output"])
print("average", [pc.mean(table[f"{name}_value"]).as_py() for name in scorers])
if __name__ == "__main__":
dataset = chalkcompute.DatasetClient().upload("support-triage", [
{"message": "I was charged twice for order 1182, please refund",
"expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
{"message": "Where is my order? It was due Tuesday.",
"expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
{"message": "Please close my account and refund the last charge",
"expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
{"message": "Following up on my last email",
"expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
])
evaluation = chalkcompute.Evaluation.create(
"support-triage",
dataset=dataset,
task=triage,
scorers=[valid_json],
)
run = evaluation.run().wait()
show(run, "valid-json")Run it with python improve_score.py:
[1.0] {"intent": "refund", "amount": 1182.0}
[1.0] {"intent": "status", "amount": 0.0}
[1.0] {"intent": "refund", "amount": 0.0}
[0.0] unclassified
average [0.75]Three of the four rows pass, and 0.75 looks like a reasonable start. Now compare each output with its message:
triage read the order
number 1182 as the dollar amount.triage
checks for “refund” before “cancel”, so it called it a refund.Both answers are wrong, and both scored 1. valid-json checks whether the output
is well-formed JSON, and both outputs are. The scorer is doing exactly what it
was written to do. It has no way to know the answers are wrong, because nothing
in it compares the output with what the row expected.
For the router, that means 0.75 says nothing about whether messages reach the right team. The cancellation in the third row would go to the refunds team.
What the router has to get right is the intent, since the intent decides which
team gets the message. The dataset has an expected column holding the right
answer for each row. Add a
scorer that compares the intent field of the output with the intent field of
expected. Put it below valid_json, at the top level of the file, so that it
is defined when Chalk deploys it:
intent_match = chalkcompute.scorers.json_match(
inputs=("output", "expected"), paths=["intent"], name="intent-match"
)An evaluation’s scorers are fixed when it is created, along with its dataset and
task, so you cannot add one to the evaluation from Step 1. You can
rescore its run instead. Add these
lines at the end of the if __name__ == "__main__": block:
rescored = run.rescore(scorers=[valid_json, intent_match]).wait()
show(rescored, "valid-json", "intent-match")rescore creates a new evaluation named support-triage (rescored) with the
same dataset revision and task and the scorers you pass. It copies each row’s
output from the original run and calls only the scorer that is new, so triage
does not run again.
Run the file again. After the Step 1 output, it prints:
[1.0, 1.0] {"intent": "refund", "amount": 1182.0}
[1.0, 1.0] {"intent": "status", "amount": 0.0}
[1.0, 0.0] {"intent": "refund", "amount": 0.0}
[0.0, 0.0] unclassified
average [0.75, 0.5]The new score is lower, and nothing about the task changed. intent-match marks
down the cancellation that was called a refund, and the row that was not JSON at
all. The task was this good all along; the evaluation now measures more of it.
Two of the four messages would reach the right team, and that is the number to
improve.
intent-match points at two problems in triage: it checks for “refund” before
“cancel”, and it returns plain text when no intent matches. Add a second version
of the task below the first, with both fixed:
@chalkcompute.function
def triage_v2(message: str) -> str:
import json, re
text = message.lower()
if "cancel" in text or "close my account" in text:
intent = "cancel"
elif "refund" in text or "money back" in text:
intent = "refund"
elif "invoice" in text or "billed" in text or "card" in text:
intent = "billing"
elif "where is" in text or "waiting" in text or "arrived" in text:
intent = "status"
else:
intent = "other"
amount = re.search(r"\d+(?:\.\d+)?", message)
return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})An evaluation also pins the version of its task, so evaluating triage_v2
takes a new evaluation. Keeping it as a separate function leaves the original in
place, so both versions stay available to compare. Add these lines at the end of
the if __name__ == "__main__": block:
fixed = chalkcompute.Evaluation.create(
"support-triage-v2",
dataset=dataset,
task=triage_v2,
scorers=[valid_json, intent_match],
)
show(fixed.run().wait(), "valid-json", "intent-match")Run the file again. After the output of the first two steps, it prints:
[1.0, 1.0] {"intent": "refund", "amount": 1182.0}
[1.0, 1.0] {"intent": "status", "amount": 0.0}
[1.0, 1.0] {"intent": "cancel", "amount": 0.0}
[1.0, 0.0] {"intent": "other", "amount": 0.0}
average [1.0, 0.75]The cancellation is now classified correctly, and every output is JSON. Three of the four messages reach the right team, up from two. Both versions ran on the same dataset revision with the same scorers, so the scores compare directly, and the new version is better.
Two of the rows are still wrong.
The fourth row, “Following up on my last email,” still fails intent-match. The
dataset labels it a status question, and triage_v2 answers other. The message
could be about anything, so this may be a case where the label is the thing to
fix. A failing row can point at the dataset as well as the task.
The first row still reads the order number 1182 as the amount, and no score shows it, because neither scorer checks the amount. This is the same gap as in Step 1: an evaluation finds the mistakes you wrote a scorer for, and only those.
An evaluation created without a suite goes in the environment’s Default suite, so all three are there in the Chalk dashboard. Open Evaluations, then Default, to see their runs side by side in the Runs table and the chart above it:
Average score is the mean of a run’s scorers, so it is 0.625 for the
rescored run and 0.875 for support-triage-v2. To line up the two versions of
the task on the same scorers, open support-triage-v2 and use Compare to
add support-triage (rescored) beside it. See
The Evaluations dashboard.
import chalkcompute
import pyarrow.compute as pc
@chalkcompute.function
def triage(message: str) -> str:
import json, re
text = message.lower()
if "refund" in text or "money back" in text:
intent = "refund"
elif "cancel" in text or "close my account" in text:
intent = "cancel"
elif "invoice" in text or "billed" in text or "card" in text:
intent = "billing"
elif "where is" in text or "waiting" in text or "arrived" in text:
intent = "status"
else:
return "unclassified"
amount = re.search(r"\d+(?:\.\d+)?", message)
return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})
@chalkcompute.function
def triage_v2(message: str) -> str:
import json, re
text = message.lower()
if "cancel" in text or "close my account" in text:
intent = "cancel"
elif "refund" in text or "money back" in text:
intent = "refund"
elif "invoice" in text or "billed" in text or "card" in text:
intent = "billing"
elif "where is" in text or "waiting" in text or "arrived" in text:
intent = "status"
else:
intent = "other"
amount = re.search(r"\d+(?:\.\d+)?", message)
return json.dumps({"intent": intent, "amount": float(amount.group()) if amount else 0.0})
valid_json = chalkcompute.scorers.json_valid(name="valid-json")
intent_match = chalkcompute.scorers.json_match(
inputs=("output", "expected"), paths=["intent"], name="intent-match"
)
def show(run, *scorers):
table = chalkcompute.DatasetClient().read(run.result_dataset)
for row in table.to_pylist():
print([row[f"{name}_value"] for name in scorers], row["output"])
print("average", [pc.mean(table[f"{name}_value"]).as_py() for name in scorers])
if __name__ == "__main__":
dataset = chalkcompute.DatasetClient().upload("support-triage", [
{"message": "I was charged twice for order 1182, please refund",
"expected": '{"intent": "refund", "amount": 49.99}', "expected_intent": "refund"},
{"message": "Where is my order? It was due Tuesday.",
"expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
{"message": "Please close my account and refund the last charge",
"expected": '{"intent": "cancel", "amount": 12.0}', "expected_intent": "cancel"},
{"message": "Following up on my last email",
"expected": '{"intent": "status", "amount": 0.0}', "expected_intent": "status"},
])
evaluation = chalkcompute.Evaluation.create(
"support-triage",
dataset=dataset,
task=triage,
scorers=[valid_json],
)
run = evaluation.run().wait()
show(run, "valid-json")
rescored = run.rescore(scorers=[valid_json, intent_match]).wait()
show(rescored, "valid-json", "intent-match")
fixed = chalkcompute.Evaluation.create(
"support-triage-v2",
dataset=dataset,
task=triage_v2,
scorers=[valid_json, intent_match],
)
show(fixed.run().wait(), "valid-json", "intent-match")