Knowledge graph vs RAG: what does an AI agent need?

01Knowledge graph vs RAG vs tools for agent context

When an agent answers a question or takes an action, it needs context it was not trained on. There are three common ways to give it that context, and they solve different problems.

Vector RAG. Documents are split into chunks, embedded and stored in a vector index. At query time, the most similar chunks are retrieved and passed to the model. The original research on retrieval augmented generation showed that pairing a model with a retriever makes generated answers more factual. It works well when the answer sits in one or two passages.

Knowledge graph. Entities such as companies, people, products and events are stored as nodes, and their relationships as edges. The agent can follow those links to gather connected facts. Microsoft Research's work on GraphRAG showed where this helps. Baseline vector retrieval struggles when an answer requires connecting separate pieces of information through shared attributes, or when a question asks about themes across a whole dataset. A graph built from the data handles those cases better.

Tools. The agent calls defined functions: search with filters, look up a record, create a lead, book an appointment. Each tool reads from or writes to your live systems. Tools give the agent fresh, structured data and the ability to act.

02Why we usually start with tools

In an established product, the most valuable context already lives in your systems, with structure and permissions attached. A product catalog has prices and filters. A CRM has leads and owners. An inventory feed has stock levels that change every hour. Copying all of that into an embedding index loses structure and goes stale. Exposing it as tools keeps it live and keeps your permission checks in place.

Anthropic's guide to building effective agents makes the same point from a different angle: start with the simplest solution that works, add complexity only when it clearly improves results. It also advises putting as much care into tool design as into prompt design. Clear tool names, well documented parameters and tools that are hard to misuse matter more than an elaborate orchestration graph.

That leads to our usual order of decisions:

1. 01

   List the questions and actions the agent must handle.

   Use real user prompts, not invented ones.

2. 02

   Map each one to a source of truth.

   A database, an API, a search index or a set of documents.

3. 03

   Expose live, structured sources as tools.

   Search with filters, record lookups and actions.

4. 04

   Add vector search for unstructured content.

   Website pages, help articles and documents, usually as one more tool.

5. 05

   Add a knowledge graph only for relationship questions.

   When users ask how things connect, and the tools cannot answer without many chained calls.

6. 06

   Keep the orchestration as simple as the tasks allow.

   Every extra node in an agent graph is one more place for context to be lost.

03What we see in production

Take a white-label AI sales assistant for dealership websites serving RV, marine and vehicle dealers. The agent has to talk like a salesperson, recommend real units from a live inventory feed, explain why, qualify the buyer and hand a complete lead to the dealer's CRM. Inventory changes hourly, and answers must never invent stock or prices. Buyers ask in natural language, with follow-ups, towing questions, payment budgets and location limits.

The architecture changed as we learned from real dealer tests. Early on, inventory was handled by an agent reading a CSV file. We replaced it with inventory search backed by Elasticsearch, with an hourly sync from each dealer's feed.

The first conversation flow was a multi-node LangGraph graph. Early dealer tests showed context loss and wrong vehicle classes. In the second sprint we simplified the flow by removing nodes and adding a search tool call. Later we rebuilt it around a tool-using agent with three kinds of tools: a lead capture tool, vector similarity search, and Elasticsearch filter tools for exact constraints such as price, class and location. Conversation state is kept in a PostgreSQL checkpointer, and every answer is grounded in the dealer's own feed and website. The assistant has been live for client dealerships in the U.S. and Canada since about January 2026, with leads and appointments flowing into the client's CRM.

No knowledge graph was needed there. The questions were about matching a buyer to units under exact constraints, which is a filter and search problem.

A retail sales assistant we built needed no graph either. Its flow searches document embeddings, runs CSV agents and queries the client's database at the same time. An aggregator node merges the results into one answer, because shoppers asked for product facts and side by side comparisons.

On an AI market intelligence app for retail investors, which we built for a fintech venture, the questions were different. Users wanted to know how news, filings, macro indicators and companies relate to each other and to their portfolio. There, a knowledge graph links companies, news and market factors so answers carry context, specialist agents for research, signals, trading and portfolios sit behind one conversational agent, and chat answers cite their sources. The graph was worth its cost because relationships were the product.

The contrast is the useful lesson. Same team, same agent framework, two different answers, chosen by the shape of the questions.

04Signs you need a knowledge graph

• Users ask how things are connected, not just what something is.
• Good answers require combining facts from many documents or records that share an entity.
• Users ask for themes or patterns across a whole dataset.
• You can name the entities and relationships clearly, and they are stable enough to maintain.

05Signs you need tools first

• The data is already structured in your product's database or APIs.
• The data changes often.
• Users need exact filters, such as price, date, size or location.
• The agent must take actions, such as creating a lead or booking a meeting.
• Answers must respect each user's permissions.

For most SaaS products adding their first agent, the second list is longer. Our article on how to add AI agents to existing SaaS without rebuilding explains how we turn existing APIs into safe agent tools. Our article on why AI agents fail in production covers what happens when the orchestration gets ahead of the data.

06Trade-offs to plan for

• Graphs need upkeep. Entity extraction, deduplication and relationship updates are ongoing work. Budget for it before you build one.
• Vector search misses exact constraints. A buyer who wants a unit under a set budget needs a filter, not a similarity score.
• Tools depend on API quality. Undocumented or inconsistent APIs make poor tools. Sometimes the first job is cleaning up the API.
• More nodes means more failure points. Simplifying an agent graph often improves accuracy, because there are fewer handoffs where context can be dropped.

Frequently asked questions