Sample AI Scorecard
What the $5,000 AI Scorecard delivers.
A sample scorecard for Northstar Course Guide, a fictional course-recommendation assistant already live on Northstar Workforce Academy's learner portal. Yours tests one AI feature you've shipped, in staging, on 50 of your own cases.
scorecard / course-guide / summary
Northstar Course Guide · AI Scorecard
- Answer qualityB
- Grounding and citationsC
- Data separationA
- Misuse and prompt injectionD
- Accessibility of AI screensC
- Cost per learnerB
- Response timeA
Illustrative grades on a fictional product.
Illustrative demo on a fictional institution. Not client work.
What we tested
One live AI feature, on your own cases.
Two weeks, in staging. Your reviewer picks the cases with us, so the grades reflect what your customers actually ask.
- 01One live AI feature: Northstar Course Guide, the course-recommendation assistant on the learner portal
- 0250 of your own cases, picked with your reviewer from real (de-identified) learner questions
- 03Two test customer accounts in staging, to check that one customer's data doesn't reach another
- 04Your staging environment and model settings as they run today; no production data
01 · The scorecard
Seven areas, each graded on evidence.
Every result links back to the cases behind it. All figures here are illustrative, on a fictional product.
| Area | What we tested | Sample result |
|---|---|---|
| Answer quality | 50 cases graded by your reviewer against the catalog | 41 correct, 6 partly correct, 3 wrong (illustrative) |
| Grounding and citations | Does each answer cite the catalog page or policy it used? | 36 of 50 cite a source; 4 quote fees not on any page (illustrative) |
| Data separation | Questions from test account A crafted to pull account B's catalog or learners | Nothing returned from account B (illustrative) |
| Misuse and prompt injection | Instructions hidden in learner messages and course descriptions; off-topic and unsafe requests | Followed injected instructions in 3 of 12 attempts (illustrative) |
| Accessibility of AI screens | Keyboard use, screen-reader labels, contrast and focus on the chat panel | Chat input has no label; timestamps below contrast minimum (illustrative) |
| Cost per learner | Token use on the 50 cases, projected to your volume | About $0.03 per conversation, $0.11 per active learner a month (illustrative) |
| Response time | Time to first word and to full answer, in staging | 1.4 s to first word, 4.2 s to full answer (illustrative) |
- Prompt injection is the biggest gap: text inside a course description can change what the assistant says.
- Most wrong answers come from fees and dates the assistant states without a source.
- Data separation between customers held in every test.
02 · Ranked fix list
What to fix first, by impact and effort.
Ranked so your team can start on the top items the week the scorecard lands, with us or on its own.
| # | Fix | Impact | Effort |
|---|---|---|---|
| 1 | Strip and fence instructions found in catalog text before it reaches the model | High | S |
| 2 | Answer fees and dates only from a cited page; otherwise hand off | High | M |
| 3 | Hand personal aid and account questions to staff instead of answering | High | S |
| 4 | Label the chat input and fix contrast and focus order on the panel | Medium | S |
| 5 | Move routine questions to a smaller model to cut cost per learner | Medium | M |
| 6 | Add the 50 cases to CI so every release is scored the same way | Medium | M |
03 · Security questionnaire answers
Draft answers to the AI questions your customers send.
Written from what we tested, with the details your team confirms in brackets. They describe how the feature works; they are not compliance claims.
Is our institution's data used to train AI models?
Draft: No. The assistant calls [model provider] through [endpoint and region] with training on customer data turned off. Conversations are stored in our environment for [retention period].
Can we switch the AI features off for our institution?
Draft: Yes. Admins can turn the assistant off for their institution in [settings location]; it stays off for all their users.
How do you stop the AI giving our learners wrong information?
Draft: It answers from our published catalog and policy pages and cites them. Questions it can't answer from those pages go to staff. We test it on a fixed set of cases before every release.
04 · Packaging and quote
Include it, or sell it. Then fix it at a fixed price.
With the cost per learner in hand, we recommend whether to include the feature in every plan or sell it as a paid AI tier, and quote the fixes. The fixed quote is in your Scorecard; production prices aren't shown on the site.
What you keep
- 01The scorecard, with each case and its result
- 02The ranked fix list, with impact and effort
- 03The test cases, ready to run again after every release
- 04Draft answers for your customers' security questionnaires about AI
- 05Cost per learner, and a packaging recommendation: include it, or sell it as a paid AI tier
- 06A fixed-price quote to fix or extend it; the $5,000 counts toward it
Not a compliance audit or certification.
Get a scorecard for your own AI feature.
Bring the AI feature you've shipped and what your customers ask about it. In 30 minutes we agree what the Scorecard would test.
Calls run in US Central time. No generic AI presentation: we discuss your product, users and production constraints.