What even is an eval

Written for a team shipping an AI feature, not for a lab benchmarking a model.

An eval runs one saved input through your AI feature, then marks the answer pass or fail against a written rule.

Take an AI feature that drafts refund replies. You save one real customer email, and write down the rule its reply must satisfy.

The feature writes a reply. The rule checks it. The reply passes or fails. That is one eval, and whatever applies the rule is the grader.

the saved email YOUR AI FEATURE the reply THE RULE does it state the 30 day window? the rule PASS FAIL
The card holds the saved email. The rule at the end marks the reply pass or fail.

One test proves almost nothing, so an eval set holds twenty to fifty cases pulled from real failures.

one case proves little 34 CASES, ONE SLIP EACH red corner = a complaint someone filed
Each slip is one saved input with its own rule. A red corner means a customer filed it.

Models are erratic; one lucky answer says little about the next. So a team reads the replies that went wrong.

They sort them into kinds of mistake, and each kind becomes a case. All the cases together are the eval set, thirty-four of them here.

Each kind of grader reads more of the answer than the last, so each one costs more time.

Three kinds of grader can apply that rule. Code can check that the refund window appears.

A second model can score a rubric, the written list of what good answers contain. A person catches the tone both of them miss.

CODE reads one phrase marks: 30 days, yes MODEL reads a paragraph marks: tone 4 of 5 PERSON reads the page sounds like a form letter milliseconds seconds minutes
The red window is how much of the answer each grader reads. The line below shows time to mark.

You rerun the whole set after every prompt change, because a fix often breaks a case that used to pass.

Someone edits the prompt to fix case four. The whole set runs again in minutes. Watch cases nine and twenty-two: Run 2 marks them wrong. Run 1 had ticked both. The set catches that before a customer does.

RUN 1 before the prompt edit 28 of 34 correct RUN 2 after the prompt edit 31 of 34 correct 9 and 22 passed in Run 1 to customers HOLD
Each small cell is one case: a dark tick passed, a red cross failed. The rings mark the new failures.

Writing the grading rules is product work, because the rules are where your team pins down what good means.

Engineers can run the set; only the team can say what counts. So the product manager drafts the rubric and the team argues it out early.

The score goes on the same sheet. Run 2 fixed five cases and broke two (nine and twenty-two), so the release waits.

written by the product manager A GOOD REFUND REPLY states the refund window names the order number gives one clear next step sounds like a person Run 2: 31 of 34. tone held. hold 9 and 22. code grader model grader a person
The four checkbox lines are the rubric. The red handwriting is Run 2's score and decision.

An eval turns a hunch about quality into a mark anyone on the team can check.