All guides

Free report

For product and engineering leaders at U.S. software companies shipping or changing an AI feature

One prompt fix can quietly break answers you never re-checked.

Stop testing AI by vibes: the eval plan template

How to test an AI feature before it ships and keep it working after: the golden set, the graders, the release gate, and a spreadsheet to run it.

This answers

  • How do we test an AI feature before it ships?
  • What goes in a golden set, and what blocks a release?
  • How do model upgrades stay safe?

Report · 15 pages, PDF · Deployed in client work

What is inside

  1. 1Every change gets a number, not a feelingExecutive summary
  2. 2Build the golden set from your own productThe golden set
  3. 3Grade every case, and compare every runGraders and runs
  4. 4Release gates, and keeping the set aliveRelease gates
  5. 5A filled eval plan: Harborline, and the spreadsheetWorked example
  6. 6Bring this page to the Product ReviewYour results

You also get

  • The golden-set method: five kinds of case, and who writes the expected answer
  • Four graders, with a per-rule check for the AI judge
  • Side-by-side runs and a release gate with a stated tolerance
  • The model upgrade steps, and keeping the set alive after launch
  • A filled plan for a fictional product, and the spreadsheet that runs it

Get the full report

The golden-set method, four graders with a per-rule check for the AI judge, side-by-side runs, a release gate with a stated tolerance, the model-upgrade steps, a filled plan for a fictional product, and the spreadsheet that runs it. The report opens on this page as soon as you send the form, and we email you a link to it, with the PDF and the spreadsheet to download.

We use these details to send you the report and to follow up once by email about a free 30-minute Product Review. You can ask us to stop at any time. See our Privacy Policy.