TALECRAFTERS
GLOSSARY · SYSTEMS

EVAL

A fixed set of test cases with known good answers, run against a workflow to catch it getting worse.

ALSO CALLEDevaluation setregression test

Creative systems degrade quietly. A model version changes, a prompt is edited, a connector returns a slightly different shape, and the output gets marginally worse in a way nobody notices until a client does.

An eval is the boring fix: twenty representative inputs, an agreed view of what good looks like, run automatically on every change. It does not need to be sophisticated to be the difference between catching a regression on Tuesday and hearing about it in a review.

For creative work the scoring is usually a person comparing outputs side by side, which is fine. The value is in the fixed inputs and the regular cadence.

QUESTIONS PEOPLE ASK

How do you test a creative workflow?

A fixed set of representative inputs, an agreed view of good, and a side-by-side comparison on every change. The rigour is in the fixed inputs.

Why do creative systems degrade without anyone noticing?

Because the inputs vary constantly, so a small drop in quality is indistinguishable from a hard brief. A fixed eval set removes that excuse.

THEORY IS FREE

Knowing the word is the cheap part.

Running it on a deadline, at volume, without burning the budget is the expensive part. That is the bit we do.

BRIEF US →