What should an AI audit of a SaaS feature test?

01What an AI audit of a product feature is, and what it is not

Most product teams reach the same point. The assistant, the agent or the search feature works in the demo, a few customers have tried it, and now sales wants it switched on for everyone. The questions that come next are rarely about the model. They are about whether one customer could see another customer's data, what the feature does with a hostile instruction, and what it will cost when every account uses it.

An AI audit answers those questions with evidence. It is an engineering test of one feature as it runs today, built from cases your own team picks, with each result linked to the case behind it.

It is not a compliance audit. Management system standards such as ISO/IEC 42001 describe how an organization governs AI across the company, and risk frameworks such as the NIST AI Risk Management Framework describe how to map, measure and manage AI risk. Both are useful, and an audit of a feature gives them real evidence to point to. But a certificate does not tell you whether your support assistant leaks one tenant's tickets into another tenant's answers. Only a test on your product does.

If you have not built the feature yet, an audit is the wrong first step. Start with an AI readiness assessment instead, which looks at what the product needs before any AI is added.

02The seven areas an AI audit should test

We grade every feature in the same seven areas, so results can be compared from one release to the next. The first four decide whether the feature is safe to release. The last three decide whether it is good enough to sell.

Two of these deserve more attention than they usually get.

Data separation fails quietly. In a multi-tenant product, the model never sees the boundary between customers unless retrieval, tools and logs enforce it. We cover how that boundary is built in tenant isolation. An audit tests it from the outside, with two test accounts and questions written on purpose to cross it.

Prompt injection is the first entry in the OWASP Top 10 for LLM Applications, and for good reason. Any feature that reads stored content, such as tickets, documents or emails, can be handed instructions by whoever wrote that content. The test is simple: plant an instruction in the data and see whether the feature obeys it.

03How to run an AI audit in two weeks

A useful audit is short and fixed in scope. Ours runs in two weeks on one feature, and the order matters more than the tooling.

1. 01

   Agree the scope in a short call.

   Name the one feature, who uses it and what customers ask it. One feature per audit keeps the grades meaningful.

2. 02

   Pick 50 real cases with your reviewer.

   Take them from real questions, with personal details removed. Include the common ones, the hard ones and the ones that went wrong before. A reviewer who knows what a good answer looks like writes down the expected answer for each.

3. 03

   Get staging access with two test accounts.

   No production data is needed. Two accounts are the minimum for testing data separation.

4. 04

   Run every case and try to break the feature.

   Each case is run and graded, injection and cross-account attempts are made on purpose, and cost and response time are measured on the same runs.

5. 05

   Grade each area on evidence.

   Every grade links to the cases behind it, so your team can see why something passed or failed instead of trusting a score.

6. 06

   Rank the fixes by impact and effort.

   The output is an ordered list your team can start on the week it lands, with the 50 cases kept as a test set for every future release.

The test set is the part that keeps paying. A feature that is checked once drifts as prompts, models and data change. The same 50 cases run on every release turn a one-time audit into a regression test. We go deeper on building that test set in AI evaluation.

04What we see in production

The most common finding is not a dramatic failure. It is a feature that is right most of the time on easy questions and quietly wrong on the questions that matter most to customers. Teams test with the questions they thought of, and customers ask different ones. Real cases, picked with someone who handles customer questions every day, change the picture within a day.

On one inherited AI product, the founder asked us to verify the existing work before spending more. The review showed what worked, what was broken and what had to be fixed before launch, in writing. The budget then went into the launch blockers instead of a rebuild.

On a copilot answering questions over live financial data, we checked every answer against a direct database query. The limits traced back to one fix in the database, not to the model, and the next step was agreed with the people who own the reports.

On a sales assistant used across many dealerships, the audit compared it with competing tools and recommended a staged rollout to a few live locations first. The same audit set the next work: lead scoring and memory across visits.

Cost is the other surprise. Teams often know their monthly model bill but not their cost per customer, which is the number that decides whether the feature belongs in every plan or in a paid tier. We explain how to measure it in AI token cost.

05What to do with the results

An audit is only useful if it changes what ships. Three decisions usually follow.

First, fix the release blockers. Anything in data separation or prompt injection that fails is fixed before a wider release, whatever the other grades say.

Second, decide how to sell the feature. With cost per customer in hand, you can choose to include it in every plan or to sell it as a paid AI tier, instead of guessing.

Third, answer the security questionnaires. Enterprise customers now ask how an AI feature handles their data, which model it calls and whether it trains on their content. Draft answers written from what was actually tested are far stronger than answers written from memory. They describe how the feature works; they are not compliance claims, and your team confirms the details.

Your own team can make every fix. The ranked list and the test cases belong to you. If you want help, our AI Scorecard is this audit as a fixed two-week engagement, with a fixed quote for the fixes.

Frequently asked questions