Golden datasets: how to know your AI is right before you trust it

Every AI demo works. That is what a demo is: a walk across prepared ground, on questions chosen because they land. I’ve given plenty of them, and I can tell you the demo proves exactly one thing — that the system can be impressive. It proves nothing about how the system behaves on the ten thousand questions nobody rehearsed.

The question a buyer cannot answer from a demo, and the only question that matters before trusting a system with real work, is this: how often is it wrong, and how would we know?

Most organizations launch without an answer. MIT’s NANDA initiative found that about 95% of enterprise generative AI pilots produce zero measurable return, and in my experience a large share of those pilots share a birth defect: nobody agreed, before launch, on what “working” would mean or how it would be measured. A pilot without measurable acceptance can’t graduate. It runs on enthusiasm until the enthusiasm runs out, and then it joins the 95%.

The fix has an unglamorous name and requires no machine learning expertise: a golden dataset.

Golden datasets in plain English

A golden dataset is a set of questions you already know the answers to, each one pinned to the source-of-truth document that proves the answer, run against the system before launch and again after every model update. That’s the whole idea. If the system gets 143 of 150 right this month and 131 right next month, you know something changed, you know by how much, and you can point at the specific questions that broke.

Here’s what one looks like in practice, from a composite client — an accounting firm assembled from patterns across real engagements, not a disguised real firm — preparing an internal knowledge assistant over its tax templates, research memos, and policies. The golden set is 150 question-and-answer pairs: 75 on tax research, 40 on policy and procedure, 35 on client-document lookup. Every pair was curated by the senior tax and advisory teams, reviewed by two people independently, and pinned to a specific document that is the agreed source of truth.

That curation is real money. It’s a couple of weeks of attention from people who bill, and it is the least automatable step in the whole project, because the entire value of the dataset is that humans you trust have vouched for every answer. It is also the cheapest insurance available, because the alternative way to learn your error rate is from a client.

The traps are the product

Of the 150 pairs, 130 are straightforward: a question, a right answer, a source. A competent system should sail through them, and passing them tells you very little. The remaining 20 are traps, built deliberately from the three ways retrieval systems fail in the wild.

Superseded-version traps, eight of them. The 2023 mileage-rate memo is still sitting in SharePoint, right next to the 2026 one, and nothing in either document announces which is current. Ask for the rate. The correct answer cites the current memo; a system without version awareness confidently quotes 2023, with a citation, which is the most dangerous failure available — wrong, with receipts. This isn’t a special shame of one messy firm. Gartner found that 63% of organizations either lack the data management practices AI needs or aren’t sure they have them, and version sprawl is the most common form I see.

Contradiction traps, six. Two memos on S-corp reasonable compensation genuinely disagree, because they were written three years apart by different partners. The passing answer surfaces the conflict and cites both. Silently picking one is a fail even if it picks the better memo, because a system that hides disagreements from you on questions you tested will hide them on the questions you didn’t.

Permission traps, six. Questions where the correct answer is a refusal, because the account asking should not be able to see the source material. The eval runs these as a low-permission user probing for restricted content. Anything other than a refusal is a leak, and a leak in an eval is a gift: it’s the one you found before an employee did.

A system that passes the easy 130 is a demo. The traps are the product.

Thresholds are governance, not engineering

The numbers attached to the golden set get decided before launch, by people accountable for the risk, and written down. The composite plan sets four: permission leakage of zero, as a hard gate — one leaked answer and the system does not ship, no meeting required; groundedness of at least 95%, meaning answers trace to a retrieved source rather than the model’s imagination; version-correct citation of at least 98% on the trap set; and refusal correctness of at least 90%, because a system that refuses everything is safe the way a bricked laptop is secure.

The reason to fix these numbers early is that launch day has momentum. Money has been spent, a date has been announced, and a threshold negotiated on launch day loses to the calendar every time. When Gartner analyzed why 30% of generative AI projects get abandoned after proof of concept, inadequate risk controls made the list, and this is what a risk control concretely looks like: a number, a pass/fail meaning, and a named owner. In the composite plan the owner is a manager, not a committee, and the results land in the monthly report leadership actually reads.

Vendors change models under you

A golden dataset is not a launch artifact. It’s a monitoring instrument, because the system you accepted is not the system you’ll be running in six months. Vendors update models on their schedule, not yours, and an update that improves the average can still break your particulars.

The composite firm learned this cheaply: a model update in May shifted the tone of its AI-drafted engagement letters enough that partners noticed, and the fix was a house style guide pinning the drafting voice. That incident is why the evaluation plan runs the full 150-question suite on every model update, plus a monthly scheduled run regardless. You cannot stop a vendor from changing the model underneath you. You can know within a day whether the change moved your numbers, instead of finding out over a quarter of quiet complaints.

The suite is vendor-neutral, and that’s leverage

Nothing in a golden dataset belongs to a vendor. The questions are yours, the source documents are yours, the thresholds are yours. When the composite firm ran its vendor evaluation, tools were scored against weighted criteria with a contractual no-training commitment as a pass/fail gate before scoring even began; Claude scored 82, ChatGPT 78, Copilot 64. Whichever tool wins, the golden set survives the decision.

That portability changes your negotiating position permanently. If the winning vendor’s pricing turns hostile at renewal, or a model change degrades your trap-question performance and stays degraded, you can put a competitor through the same 150 questions in a week and negotiate from evidence instead of inertia. A buyer with a portable acceptance suite is a buyer who can leave, and vendors price the difference.

The whole apparatus fits in a short document: dataset design, trap construction, thresholds, cadence. I’ve published the composite version, the sample Evaluation Plan — golden datasets, traps, and thresholds for a composite client — with the rest at /examples. Before you trust an AI system with work that matters, make it earn 150 answers you can check. Especially the 20 it was designed to fail.