Wrong answers nobody counts
The demo looked right on ten questions. Nobody has measured how often it is wrong on the questions your customers really ask.
AI Scorecard for EdTech · 2 weeks · $5,000
AI audit for the AI you already built. Know if it is ready for every customer.
You built an AI feature into your learning platform and the demo works. We test it on 50 of your own learner and staff cases, try to break it, grade it in seven areas, from data separation between institutions to cost per learner, and rank what to fix first, with a fixed quote. Two weeks, and the fee counts toward the fixes.
50 of your cases
Picked with your reviewer, not a vendor
Two weeks
In staging, with no production data
Fee credited
The $5,000 counts toward the fixes
Reviews of AI already built, at
Northstar Course Guide · AI Scorecard
Summary
- Answer qualityB
- Grounding and citationsC
- Data separationA
- Misuse and prompt injectionD
- Accessibility of AI screensC
- Cost per learnerB
- Response timeA
Biggest gap: text inside a course description can change what the assistant says.
Illustrative example on a fictional product. Not client work.
Why a scorecard
The demo works. Production is a different test.
Most AI features fail in production for reasons a demo never shows. Read why the test set comes before the first release.
One customer's data in another's answer
Retrieval, caching and shared prompts can leak across accounts. It rarely shows up until a customer finds it.
Instructions hidden in your own data
Text inside a record, a document or a message can tell the model what to do. Most features are never tested for it.
A cost per customer no one has worked out
Token use on a demo says little about the bill at full volume, or whether the feature should be included or sold.
What we test
Seven areas. Each graded on evidence.
Every grade links back to the cases behind it, so your team can see exactly why it passed or failed.
Answer quality
Your 50 cases, graded by your reviewer against the answer your team expects.
Grounding and citations
Does each answer come from, and point to, the record or document it used?
Data separation
Questions from one test account crafted to reach another account's data.
Misuse and prompt injection
Instructions hidden in user messages and stored content, plus off-topic and unsafe requests.
Accessibility of AI screens
Keyboard use, screen-reader labels, contrast and focus on the AI parts of the interface.
Cost per customer
Token use on the 50 cases, projected to your volume and your pricing.
Response time
Time to the first word and to the full answer, as it runs today.
Try it
Open a sample scorecard.
Pick an area to see what was tested, the result and the fix it led to, on Northstar Course Guide, a fictional course-recommendation assistant on a learner portal.
Pick an area
Sample scorecard
Northstar Course Guide
- Misuse and prompt injection
- Grade D
- What we tested
- Instructions hidden in learner messages and course descriptions
- Result
- Followed injected instructions in 3 of 12 attempts
- Fix on the ranked list
- Strip and fence instructions in catalog text before it reaches the model
Illustrative example on a fictional product. Not client work.
Test it yourselves, or score it
A number for every risk. Before your customers find it.
Your team knows the feature best. The Scorecard adds what is hard to do from the inside: a fixed test set, attacks tried on purpose, and an outside grade.
Testing it yourselves
It looked fine on our questions.
- The test setA few questions someone thought of
- What gets checkedWhether the answers look right
- AttacksRarely tried
- What you getA feeling that it is nearly ready
- After releaseStart again next time
- What it costsYour team's time, with no end date
Nobody is sure what a customer will hit first.
The Datics AI Scorecard
Graded, ranked and quoted.
- The test set50 of your own cases, picked with your reviewer
- What gets checkedSeven areas, from data separation to cost per customer
- AttacksPrompt injection and cross-account requests, on purpose
- What you getGrades on evidence and a ranked fix list
- After releaseThe same 50 cases, ready to run on every release
- What it costs$5,000 and two weeks, counted toward the fixes
- Your own cases
- Attacks tried
- A fixed quote
Security questionnaire answers
Draft answers to the AI questions your customers send.
Written from what we tested, with the details your team confirms in brackets. They describe how the feature works; they are not compliance claims.
Is our institution's data used to train AI models?
Draft: No. The assistant calls [model provider] through [endpoint and region] with training on customer data turned off. Conversations are stored in our environment for [retention period].
Can we switch the AI features off for our institution?
Draft: Yes. Admins can turn the assistant off for their institution in [settings location]; it stays off for all their users.
How do you stop the AI giving our learners wrong information?
Draft: It answers from our published catalog and policy pages and cites them. Questions it cannot answer from those pages go to staff. We test it on a fixed set of cases before every release.
How it runs
Two weeks. Then you decide what to fix.
Four steps, from a free call to a ranked fix list. You can stop after the readout and keep everything.
30 minutes
No chargeProduct Review
You show us the AI feature and what your customers ask it. We agree what the Scorecard would test.
You keep
A straight answer on fit
Week 1
$5,000Cases and access
We pick 50 cases with your reviewer and get access to staging, with two test accounts. No production data.
You keep
Your 50 test cases
Week 2
In stagingTest and grade
Every case run and graded, attacks tried, cost and speed measured, each result linked to its evidence.
You keep
The scorecard
Readout
Fee creditedFixes and quote
We walk your team through the fix list and a fixed quote. Fix it with us, or with your own team.
You keep
A ranked fix list and a quote
In practice
When the AI was already built, a review came first.
Three products where we tested or reviewed AI that already existed before any new build, and what it changed.
Financial services
Peret
A review of an inherited AI product before any new spend, then the launch work it called for.
How it ran
- The existing work verified first, at the founder's request
- What works, what is broken and what to fix before launch, in writing
- The budget went into launch blockers, not a rebuild
Construction
OCCI
An assessment of its Microsoft Copilot agent on live job cost data, with every answer checked against SQL.
How it ran
- Run inside OCCI's own Microsoft tenant, on its real setup
- Its limits traced to one fix in the database
- The next step agreed with the people who own the reports
Automotive and dealerships
Coast Technology
A dealer sales assistant audited against competing tools, with a staged rollout recommended before it scaled.
How it ran
- Competing dealer chat tools reviewed side by side
- A few live dealerships first, before every dealer
- The audit set the next work: lead scoring and memory across visits
The offer · AI Scorecard
Two weeks to know if the AI you built is ready for every customer, and exactly what to fix first.
$5,000
AI Scorecard · 2 weeks
What you keep
- The scorecard, with every case and its result
- The ranked fix list, with impact and effort
- The 50 test cases, ready to run on every release
- Draft answers to your customers' security questions about AI
- A fixed quote for the fixes; the $5,000 counts toward it
Everything is yours after the readout, including the test cases. Your own team can make the fixes; you do not need us for them.
It fits if you have
- An AI feature, agent or assistant already built, live or in staging
- Customers or prospects asking how safe it is
- A team that wants an outside grade before a wider release
It is not for
- AI you have not built yet: start with the Build Sprint or the Planning Sprint
- A compliance audit or certification
Not built yet? Start with the Build Sprint if you know the outcome you want, or the Planning Sprint if you are still deciding. See every EdTech offer and module.
Questions
AI audits, answered.
Building an agent or an assistant next? See AI agent development and RAG development.
Does it check how student and learner data is handled?
Yes, inside the feature: whether one institution's data can reach another, what is sent to the model and what is kept. The draft questionnaire answers describe it for your team to confirm. It is an engineering evaluation, not a FERPA or COPPA compliance audit.
Does it help us decide how to price the AI feature?
Yes. With the cost per learner measured on your own cases, we recommend whether to include the feature in every plan or sell it as a paid AI tier, and quote the fixes.
What is an AI Scorecard?
A two-week evaluation of one AI feature you have already built. We test it in staging on 50 of your own cases, grade it in seven areas, from answer quality and data separation to cost per customer, and give you a ranked fix list and a fixed quote. It is $5,000, and the fee counts toward the fixes.
What does an AI audit of our product include?
The AI Scorecard audits one AI feature, agent or assistant you have built, the way your customers use it: answer quality, grounding and citations, data separation between customers, misuse and prompt injection, accessibility of the AI screens, cost per customer and response time. Each grade links to the cases behind it, and the fixes come ranked by impact and effort.
How much does an AI audit cost?
The AI Scorecard is $5,000 for two weeks. If you then fix or extend the feature with us, the $5,000 counts toward that fixed quote. The first call, a 30-minute Product Review, has no charge.
Does the feature need to be live?
No. It can be live with customers, in beta or still a prototype. We test it in staging as it runs today, so the grades describe what your customers would actually get.
Do you need our production data?
No. We work in staging with two test customer accounts, and the 50 cases come from real questions your reviewer picks and removes personal details from.
Which models and platforms can you test?
Any. Features built on OpenAI, Anthropic, AWS Bedrock, Azure AI, Google Vertex AI or self-hosted models, built by your team, by us or by another vendor. The Scorecard tests the feature your customers use, not one model.
What if we have not built the AI yet?
Then the Scorecard is not the start. If you know the outcome you want, the Build Sprint ($10,000, three weeks) builds it in your staging. If you are still deciding what to build, the Planning Sprint ($5,000, two weeks) ranks it and quotes the first wave.
Can our own team make the fixes?
Yes. The fix list and the 50 cases are yours, ranked so your team can start on the top items the week the scorecard lands. If you fix it with us, it is at a fixed quote and the fee counts toward it.
What do we need to provide?
Staging access to the feature, two test customer accounts, a reviewer who knows what a good answer looks like, and a few hours of their time to pick and grade the cases.
Is it a compliance audit or a certification?
No. It is an engineering evaluation of how the feature behaves. The draft answers to security questionnaires describe how it works, for your team to confirm; they are not compliance claims.
Built an AI feature? Find out what it gets wrong first.
Show us the feature and what your customers ask it. In 30 minutes we agree what the Scorecard would test.