AI data integration: connect AI without a new platform

01Why the usual answer starts with a new platform

Much of what ranks for this question comes from cloud and data platform vendors, and their answer makes sense from where they sit: bring your data onto their platform, then connect AI there. For a vertical SaaS company whose customer data lives in a live operational database, that answer turns one AI feature into a data migration project.

Your customer data already carries a schema, tenant boundaries, role permissions, retention rules and contractual promises about where it is stored. Copy it somewhere new and all of that has to be rebuilt and proven again before the first AI answer ships.

The vendor-neutral approach starts from what you already run: the database, the APIs your application calls, the permission model your customers trust, and the cloud account your security reviews already cover. The model does not need your whole database. It needs the right slice of context, for the right user, at the moment of the request.

02Which access path fits our data: API, read replica or direct query?

There are three practical ways for an AI feature to read customer data. Most products use more than one, chosen per question type.

Your existing API. The AI calls the same endpoints your application calls, using the identity of the user who asked. This is the safest default because it inherits everything your product already enforces: authorization, tenant scoping, business rules and rate limits. It fits live record lookups ("what is the status of this order") and every action that changes data. APIs are usually shaped for screens, so you may need a few narrow, read-only endpoints. Change only what the capability requires.

A read replica. For questions that scan or aggregate many records, and for building a retrieval index, a replica keeps that load away from the primary database. On a managed service such as Amazon RDS for PostgreSQL, replicas are read-only and are kept current with PostgreSQL's native replication. Replication lag means a replica can be slightly behind, so it suits reporting and trend questions better than "what is the balance right now".

A direct query. Querying the primary database directly is rarely right for AI. When it is used, limit it to narrow, parameterized, read-only queries written by your engineers, with timeouts and row limits. Model-written SQL should never run against the primary.

Whichever path you choose, give each one its own database role or service identity, scoped to the views and endpoints it needs.

03What data actually needs to be indexed for retrieval?

Less than most teams expect. Structured fields that your database can already filter, sort and join do not need embeddings. If a user asks for "customers with overdue invoices in March", the right design is a tool that runs a query through an approved path, not a vector search over invoice rows.

Retrieval indexes earn their place for unstructured text where meaning matters more than exact values: help center articles, contracts and uploaded documents, case notes, support tickets, emails and policy files.

Scope the index from the feature, not the database. Write down the real questions users will ask, trace each one to the source that answers it, and index only those sources. For each chunk you store, keep:

• the tenant identifier, so retrieval can be filtered before anything reaches the model
• the source record identifier and version, so answers can cite and link back
• permission labels that match your existing roles
• a timestamp, so stale content can be detected and refreshed

Leave out fields the feature does not need, and redact identifiers where the feature works without them. A smaller index is less to secure, sync and pay for.

04Can embeddings live in our current database?

Often, yes. If your product runs on PostgreSQL, the open-source pgvector extension stores vectors alongside the rest of your data, with exact and approximate nearest neighbor search. Its documentation describes two index types, HNSW and IVFFlat, with HNSW offering a better speed and recall tradeoff at the cost of slower builds and more memory. It also supports filtering with a normal WHERE clause, iterative index scans when filters remove many rows, and hybrid search with PostgreSQL full-text search.

When embeddings sit next to the source rows, the tenant filter is a WHERE clause on the same table and existing backups cover the index. A JOIN also brings back the current record rather than a copy that may have drifted.

Two cautions apply. First, a read replica cannot accept writes, so embeddings usually live in the primary, in a separate schema, or in a small dedicated PostgreSQL instance in the same cloud account. Second, if your primary database is not PostgreSQL, a vector store in the same network and account, keyed back to your source identifiers, gives the same outcome without moving the system of record.

05How do we keep sensitive data inside our network or cloud account?

Choose where the model runs. Use model endpoints hosted in your own cloud account and region, or self-hosted models for data that cannot leave your network at all.

Check permissions before retrieval, not after generation. The AI should only see what the asking user could already see in your product. If you use PostgreSQL row-level security, note what the documentation spells out: once row security is enabled on a table with no policy, a default-deny policy applies. The table owner is typically not subject to these policies, and superusers and roles with the BYPASSRLS attribute always bypass them. The role your AI service uses must be neither.

Treat prompts and logs as customer data. Retrieved context, prompts and responses often contain the same sensitive fields as the source. Keep them in the same account, under the same retention rules and access limits.

Run extraction inside your environment. If documents cannot leave, OCR and document parsing cannot leave either.

Fence off generated queries. If plain-language questions become SQL, run them through a read-only role against curated views, with allowlisted tables, row limits, timeouts and logging. Anything that writes goes through your API with human approval.

06A decision table: access pattern by data type and sensitivity

07How do we start AI data integration without a migration?

1. 01

   Pick one feature and write down the real questions or tasks it must handle.

2. 02

   Map each question to its source

   an API, a replica view, or a document set.

3. 03

   Create a scoped service identity and read-only views for the AI, with tenant filtering enforced in the database or the API.

4. 04

   Index only the unstructured sources the feature needs, tagged with tenant, source and version.

5. 05

   Build an evaluation set from real cases and measure answer quality, citation accuracy, permission leakage, latency and cost per successful task.

6. 06

   Release to a small group behind a feature flag, with every retrieval and response logged.

7. 07

   Only then decide, from the measurements, whether any data needs to move.

08When does moving data to a new platform make sense?

Sometimes it does, and the trigger should be a measured requirement rather than a vendor's reference architecture. Signals we take seriously:

• The feature needs to join data across systems that do not share keys, and doing it at request time is slow or unreliable.
• Analytical workloads are degrading the replica or the primary, even after query tuning.
• The feature needs history that the operational database purges or archives.
• Retrieval quality or latency targets are still missed after tuning the index, filters and chunking.
• A data residency or retention rule requires a separate, governed store.

Even then, move the slice the feature needs, not the whole database, and keep the operational system as the source of truth. We have done both. In one dealership engagement, visitor, location and customer data from three legacy products genuinely needed to be unified into one platform. In a photography platform that already ran a lakehouse on AWS and Databricks, we brought order data from two product lines into one shared order model.

Where the AI runs matters too: see Bedrock vs Azure AI vs Vertex AI for an existing SaaS.

09What we see in production

A document-management software company serving public-sector archives. Archive staff needed to ask questions over large collections of sensitive records in more than one language and get answers they could trace. They also needed extraction of people, places and acronyms, and writing tools that never alter names, dates or figures. The documents could not go to an outside AI service. We settled architecture ownership first, simplified the backend stack, and treated the client's prototype as a guide. We chose a self-hosted OCR engine after side-by-side tests on real documents, including handwriting, and kept model serving inside the environment.

A regulated investment platform. Issuers copied deal data by hand from pitch decks, memos and financials into long forms. We kept the existing .NET product as the system of record and built AI as services around it, so document, onboarding and compliance workflows could evolve without replacing the regulated application. Each agent got one narrow job and its own API, and only values a person accepts can reach a deal.

An agentic data copilot for businesses. Business users wanted answers from their own databases and documents in plain language, but the first customer would not let its data leave its network or reach a public model provider. That constraint was the starting point for the design, not an afterthought.

The same pattern shows up elsewhere in our work: reliability and regression coverage before AI retrieval in a cloud file-management product, and data synchronization with no core replacement in an investor CRM. It also shows up in AI connected to existing practice-management systems inside the customer's cloud for a practice communications platform. In each case, the hard part was not the model. It was permissions, tenancy, extraction and audit around the data that was already there.

Frequently asked questions