Most automation projects fail quietly. The demo works, the pilot works, and three months later nobody trusts the output. The fix is boring and it comes first: a set of real cases with known right answers.

We call it the eval set. We build it before we write a prompt, and every change to an agent has to beat it before it ships.

What goes in the set

Two hundred to two thousand real examples from the client's own history: invoices, tickets, contracts, forms. For each one, the answer a careful person would give. We include the ugly ones on purpose: the scanned fax, the reply-all chain, the customer who writes in capitals.

How we build it in a week

We sit with the people who do the work today and label together. They know which cases are hard and why. By Friday we have a spreadsheet, a scoring rule and a baseline: how well the current process does.

Why it pays for itself

The eval set settles arguments. When a stakeholder asks whether the new model is better, we run it. When an edge case breaks in production, it goes into the set and never breaks again. And when the client takes the system over, they inherit the one thing that tells them it still works.