TALECRAFTERS
GLOSSARY · SYSTEMS

EVAL

A fixed set of test cases with known good answers, run against a workflow to catch it getting worse.

ALSO CALLEDevaluation setregression test

Creative systems degrade quietly. A model version changes, a prompt is edited, a connector returns a slightly different shape, and the output gets marginally worse in a way nobody notices until a client does.

An eval is the boring fix: twenty representative inputs, an agreed view of what good looks like, run automatically on every change. It does not need to be sophisticated to be the difference between catching a regression on Tuesday and hearing about it in a review.

For creative work the scoring is usually a person comparing outputs side by side, which is fine. The value is in the fixed inputs and the regular cadence, not in automating the judgement.

QUESTIONS PEOPLE ASK

How do you test a creative workflow?

A fixed set of representative inputs, an agreed view of good, and a side-by-side comparison on every change. The rigour is in the fixed inputs, not in automating taste.

Why do creative systems degrade without anyone noticing?

Because the inputs vary constantly, so a small drop in quality is indistinguishable from a hard brief. A fixed eval set removes that excuse.

THEORY IS FREE

Knowing the word is the cheap part.

Running it on a deadline, at volume, without burning the budget is the expensive part. That is the bit we do.

BRIEF US