How do you build trust and safety into an AI product?

01AI trust and safety is a data problem first

When people talk about AI trust and safety, they usually mean filters on the output: blocked words, refusal messages, moderation. Those matter, but they are the last line. Most trust failures in education products start earlier. An answer comes from a source nobody can name. A fact is out of date because the record was never refreshed. Content is generated for a child without any limit on what it can contain. Student data reaches a model that never needed it.

NIST's AI Risk Management Framework is a voluntary framework for managing the risks AI poses to people, organizations and society. It is built around putting trustworthiness into how AI systems are designed, developed, used and evaluated. That framing is useful because it moves trust from a feature to a property of the whole system, including the data pipeline.

For education products, we focus on three questions. Where did this come from? Who is it for? Who approved it? If the system can answer all three for any output, most other trust work becomes easier.

02Provenance is the foundation

Provenance means knowing where every piece of information came from, when, and how it was changed. The W3C PROV family of documents defines a data model for exactly this, so provenance can be recorded and exchanged between systems. You do not need to adopt the full standard to use the idea. You do need every record in your knowledge base or graph to carry its source, its version and the steps that transformed it.

With provenance in place, an answer can show its sources. A wrong answer can be traced to the record that caused it. A source that turns out to be unreliable can be removed along with everything derived from it. Without provenance, none of that is possible, and every fix becomes a guess.

03Decide what enters the knowledge graph

A knowledge graph or retrieval index is only as trustworthy as its intake rules. These are the checks we put on intake:

1. 01

   Known source. Every record comes from a named source, public or licensed, with a reference back to it.

2. 02

   Normalized format. PDF forms, CSV exports, web portals and API responses are mapped into one schema before they enter the graph, so the same fact always looks the same.

3. 03

   Version and lineage. Each record keeps its version and the chain of steps from raw source to graph node.

4. 04

   Change detection. Sources are re-checked on a schedule, and changes update the graph instead of piling up beside old values.

5. 05

   Minimum personal data. If a capability does not need student data, it does not get student data. See AI for K-12 software without touching student data.

6. 06

   Retrieval relevance tested. Someone checks that the right records come back for real questions, not only that records exist.

04Guardrails belong in the backend

Restrictions that live only in the interface can be bypassed by anyone who calls the API directly, or by a prompt that talks around them. For an open-ended generator, such as a tool that builds websites or games, content rules must sit in the backend generation flow, where every request passes through them.

Guardrails also need to be specific to who is using the product. A tool for young children should generate differently by age group, use images from sources suitable for children, and avoid anything that requires reading skills the user does not have yet. Approval should sit with an adult: nothing goes online without a parent or supervising adult saying yes. Time limits and a lock that keeps a child inside the tool are part of safety too, not extras.

05Children's data raises the bar

In the United States, the FTC's Children's Online Privacy Protection Rule applies to operators of sites and online services directed to children under 13, and to operators that knowingly collect personal information from children. If your product may be used by young children, consent, age checks and deletion of a child's data have to be designed in from the start. They touch sign up, family setup, publishing and account management, and they are far harder to retrofit than to build. This is an area for your own legal counsel, and engineering should make whatever they decide easy to enforce.

06Test the guardrails, not only the features

A QA plan for an AI education product should spend as much effort on what the system refuses as on what it does. Try to generate content outside the age band. Try to publish without approval. Try to exceed the time limit. Try to reach another family's or another institution's data. Each of these is a test case with an expected result, run before every release.

07What we see in production

For a higher education technology leader's advisory firm building an AI-enabled B2B SaaS platform, a data engineer from our team works inside the client's own team, alongside its product manager and backend developer. In the first phase, the work was to collect public records from city-level sources in mixed formats, including web portals, CSV exports, APIs and PDF links. Those records were then normalized into JSON or SQL, with change detection added for periodic updates.

In the second phase, the work is to define the graph schema, normalize those sources into it, track versions and lineage, and work with the client's machine learning engineers on retrieval relevance. The client renewed into a second, larger contract. It is entirely data work, and it is where the trustworthiness of the client's AI platform is decided.

For a founder-led AI company, we built a child-safe website and game builder for kids, families and schools on the foundation of the company's consumer builder. Content rules sit in the backend generation flow, and generation is adjusted by age group, such as simple pop-up games for ages 4 to 6 and guessing games for ages 7 to 9. Pictures come from open, kid-friendly image libraries.

Nothing goes online without a supervising adult's approval, and daily time limits and a PIN lock keep parents in control. An age and consent check runs at sign up, and a child's data can be deleted on request. The product went live, and a full QA cycle passed through authentication, family setup, access controls, generation, editing and publishing.

Those content rules follow a written guardrails document and block unsuitable destinations and content categories in the backend, so no screen in the kids builder can get around them. Every change, including the production deployment pipeline, went through pull request review before it reached development or production.

For the same company's governed AI content platform, every claim in an answer carries a trust tier, a confidence score and a source reference. We wrote a provenance-first filtering approach so the record behind each claim stays attached as it moves through the system.

For K-12 platforms in particular, the same principle shapes how to add AI without touching student data.

Frequently asked questions