How do you run an AI POC with real data?

01Decide what the pilot must prove

A pilot with real data is expensive in attention. Real users give their time, real data needs approvals, and a bad result sticks in people's memory. So the first step is to write down the one question the pilot must answer, and the evidence that would answer it. This is part of our larger guide, What is forward deployed engineering?.

Good pilot questions are narrow and observable. Can the assistant answer a runner's questions about an uploaded route and find similar routes nearby? Can the extraction step fill these twelve fields from our own documents as well as a person does? Weak questions are broad: will users like AI, or is the model good enough.

We also separate a demo from a pilot. A demo shows a capability on data you control. A pilot shows the capability on the client's data, with the client's users, under the client's constraints. Both are useful, but they answer different questions, and confusing them leads to pilots that are really demos with more risk.

When an early-stage AI startup for runners asked us to help convince a global sportswear brand to build an AI experience on its running app data, we kept the first demo deliberately plain. It skipped a polished interface and showed the capability itself: upload a GPX route file and get a summary of the route in a chat. Then ask for similar routes nearby and receive at least three matches based on distance and elevation. That answered the first question, whether the idea worked at all, in days rather than weeks.

02List the data before you estimate

The next step is a written data request. After that demo, the brand asked for a proof of concept on its own data. Before estimating anything, we listed exactly what the pilot would need. It needed course details for one target city, GPX files as a fallback, performance data from many runners on the same courses, and access to the running app's partner interface. Personal training advice depends on many runners' results on the same course, so without that data the second use case could not be proven, however good the model.

Writing the request first does three things. It shows the data owner precisely what they are being asked to share. It exposes gaps early, before a team is staffed. And it makes the estimate honest, because the hard part of most pilots is getting, cleaning and connecting the data, not calling a model.

Sometimes the data does not exist at all. Years ago, our team worked with a hardware innovation platform on an engine that turned a plain request, such as a wish to build a wireless video doorbell, into a circuit board design, firmware and an enclosure. No public training data existed for requests about electronic components. Public text corpora gave poor results, so the team generated domain training examples, filtered them, and compared several language tools before choosing one. When the real data is missing, say so in the plan and budget for creating it.

03Get through the security review early

Real data comes with a review. In the running project, the brand's apps team required a privacy and security vendor assessment before sharing any data, and the startup did not own that data. We filled in the vendor questionnaire. We also recommended that the startup remain the vendor to the brand, with Datics as its technology partner, so the brand would not hesitate to share data with a party it already knew.

For established B2B products, the equivalent is the customer's security review, data processing terms and sometimes a request to keep everything in their own cloud account. Start it in the first week, not after the build. If the data you need cannot be shared in time, consider whether synthetic or de-identified data can answer the pilot question first. Our article on AI for K-12 software without touching student data shows that approach in a sensitive domain.

04Keep a separate staging path

Users who test AI on real documents will find failures you never saw, and they will often find them while you are changing something else. A pilot needs a path where fixes can be made and checked without touching what other users rely on.

Feature flags are the standard tool here. Pete Hodgson's article on feature toggles, published on Martin Fowler's site, separates release toggles, which hide unfinished work, from permission toggles, which give a feature to a chosen group of users. It also describes canary releases, where a small cohort gets a feature first and its metrics are compared with everyone else's. Google's SRE workbook describes canarying the same way: a partial, time-limited deployment, evaluated against a control group, with a clear path to roll back.

05Run the AI POC in a fixed order

This is the order we follow on most pilots:

1. 01

   Write the pilot question and the evidence that would answer it.

2. 02

   Build a plain demo on data you control to prove the idea works at all.

3. 03

   Write the data request, listing every source, owner and field the pilot needs.

4. 04

   Start the security and privacy review in parallel with the data request.

5. 05

   Set up a staging environment and a release flag before the first real user logs in.

6. 06

   Agree the testing windows with the client, so deployments do not land during a user session.

7. 07

   Collect each failure with its input, fix it in staging, and add it to the test set.

8. 08

   Decide at the end

   move to production, run a second pilot, or stop and keep what you learned.

06What we see in production

On the running assistant, built for an early-stage startup in sports and fitness, the demo was delivered and praised by the founder, and the data request turned out to be the real gate. The proof of concept depended on data owned by a third party and on that party's vendor review. We see this often: the pilot plan is written around the model, while the schedule is decided by data access.

On the hardware design engine, built with a platform for product creators, the team split the problem into separate modules, each with its own technique, so each could be tested on its own. Language AI only read the request and pulled out components and parameters. Electrical checks, part placement and routing were rule-based and search-based, so every output was validated before manufacturing files were produced. Splitting a pilot this way lets you show real progress on the parts that work while the hardest part is still in research.

Data was a gate there too. No public data covered requests about electronic components, and general corpora and pretrained word vectors extracted components poorly. We generated our own domain examples, filtered out the noisy ones and tagged every entity and value before choosing the extraction model.

Across pilots, the pattern is consistent. The prompt is rarely what decides success. The data request, the review, the staging path and the release timing do. For the cost side of moving a pilot to production, see how much production AI costs in SaaS.

Frequently asked questions