Beyond Vector Search: Why We Built with GraphRAG
The Promise and the Problem
Vector databases changed everything about how we build AI applications. The ability to encode meaning into mathematical space and find semantically similar content felt like magic when we first implemented it. Type a question in plain English, and the system retrieves relevant documents based on meaning rather than keywords. For many use cases, this approach works brilliantly.
We built several client projects on this foundation. Document search, FAQ matching, content recommendation systems. The results impressed stakeholders and delivered genuine value. Vector search became our default approach for any retrieval problem.
Then we hit a wall.
A client asked us to build an internal knowledge system for their organisation. The requirement seemed straightforward: help employees find information about projects, people, and processes. We deployed a vector database, embedded their documentation, and demonstrated the initial results. Everyone was pleased. Then someone asked a simple question that exposed a fundamental limitation.
“How is Sarah connected to the Birmingham expansion project?”
The system returned nothing useful. It found documents mentioning Sarah. It found documents about the Birmingham project. But it could not answer the actual question because the connection existed across multiple documents and required reasoning about relationships, not similarity.
Understanding Why Vector Search Falls Short
Vector search operates on a simple principle: convert text into numerical representations called embeddings, then find other embeddings that sit nearby in mathematical space. When you search for “project management tools,” the system finds documents about Trello, Asana, and similar topics because their embeddings cluster together.
This works wonderfully for questions with direct answers. “What is our refund policy?” returns the refund policy document. “How do I reset my password?” surfaces the relevant help article. The semantic similarity between question and answer ensures accurate retrieval.
The approach fails for questions requiring multiple reasoning steps. Consider what happens when you ask about connections between entities. The system must first identify that Sarah is a senior architect, then find that she authored the technical specification for the Birmingham project, then discover that specification influenced the procurement decisions. No single document contains this complete chain. Vector search finds isolated pieces but cannot assemble them into understanding.
We call these “multi-hop” reasoning questions because answering them requires hopping between multiple pieces of information. Traditional retrieval treats each document as an island. Real organisational knowledge exists as a connected web where meaning emerges from relationships as much as content.
GraphRAG: Connecting the Dots
GraphRAG stands for Graph-based Retrieval Augmented Generation. The name sounds technical, but the concept mirrors how humans actually think about connected information. Rather than treating documents as isolated items, GraphRAG builds a map of entities and their relationships.
Think of it as the difference between a library catalogue and a detective’s investigation board. The catalogue tells you which shelf holds each book. The investigation board shows you how every person, place, and event connects to every other. When someone asks about Sarah and the Birmingham project, GraphRAG traverses the relationship map rather than searching for similar text.
The implementation adds a knowledge graph layer alongside your vector database. As documents enter the system, natural language processing extracts entities (people, projects, locations, concepts) and relationships (authored, approved, mentioned, collaborated). These become nodes and edges in a graph database. The vector database still handles semantic similarity. The graph database handles structural relationships.
When a question arrives, the system determines whether it requires simple retrieval or relationship traversal. Simple questions go straight to vector search. Complex questions trigger graph queries that walk relationship paths before gathering relevant context for the language model.
The language model receives richer context because it sees not just similar documents but the actual relationship chain connecting query entities. “Sarah is connected to Birmingham through her authorship of Technical Specification 47, which established requirements that influenced Procurement Decision 892, which approved the Birmingham expansion scope.” The model can now provide a coherent answer that no single document contains.
The Technical Stack We Use
Pinecone
Pinecone serves as our vector database layer. We chose Pinecone for its managed infrastructure, excellent query performance, and straightforward scaling. Documents are chunked, embedded using models like OpenAI's text-embedding-3-large, and indexed for semantic search. When questions require similarity matching rather than relationship traversal, Pinecone handles retrieval directly.
Neo4j
Neo4j provides our graph database layer. Neo4j's Cypher query language makes relationship traversal intuitive, and its visualisation tools help during development and debugging. As documents enter the system, our extraction pipeline identifies entities and relationships, creating nodes and edges that mirror the organisational knowledge structure. Graph queries can find paths between entities, identify clusters, and surface indirect connections.
Microsoft's GraphRAG Framework
Microsoft's GraphRAG Framework offers an alternative approach we deploy for certain use cases. Rather than building custom extraction pipelines, Microsoft's framework automatically constructs knowledge graphs from document collections and provides pre-built query strategies. This accelerates deployment when project timelines are tight, though custom Neo4j implementations offer more control for complex requirements.
Measuring the Difference
GraphRAG achieved 78% accuracy on the same questions. The relationship traversal provided structural context that enabled coherent multi-hop answers. Even when answers were incomplete, they typically provided useful partial paths rather than disconnected fragments.
When to Choose GraphRAG
GraphRAG adds complexity and cost. The additional infrastructure, extraction pipelines, and maintenance overhead mean it should not be the default choice for every retrieval application.
Choose vector search alone when your questions have direct answers contained within single documents, when your content lacks strong entity relationships, or when latency requirements preclude graph traversal. FAQ systems, documentation search, and content recommendation often work excellently with vectors alone.
Choose GraphRAG when users ask questions about connections between entities, when answers require synthesising information across multiple sources, when your domain has rich relationship structures (organisational hierarchies, project dependencies, regulatory frameworks), or when reasoning quality matters more than response speed.
Many production systems benefit from a hybrid approach that routes queries to the appropriate retrieval strategy based on question analysis. This provides vector search speed for simple questions while enabling graph reasoning when complexity demands it.
We have found GraphRAG particularly valuable for professional services firms tracking client relationships, manufacturing companies managing supplier networks, and any organisation where understanding "how things connect" delivers business value beyond "what documents exist."
Moving Forward
Building effective GraphRAG systems requires careful attention to entity extraction quality, relationship schema design, and query routing logic. The technology decisions matter less than the knowledge modelling work that precedes them.If your organisation struggles with questions that span multiple documents, or if your users express frustration that the AI “knows” information but cannot connect it, GraphRAG may provide the architectural shift you need. We are happy to discuss whether this approach fits your specific requirements.
Ready to Activate Your Digital Advantage?
See for Yourself, Chat to Our Intelligent Assistant Now to See What We Can Do for You.
Discover more AI Insights and Blogs
From experimentation in chat windows to production-grade AI pipelines: why enterprises are moving beyond prompts to build repeatable, auditable agentic systems.
Large Language Models are remarkably coherent. They sound confident. They construct grammatically perfect sentences with
flawless logic.
When you flip a light switch, you don’t think about which power station generated the electricity. You simply expect the light to turn on. The same principle should govern how ...
When business leaders hear about fine-tuning large language models, many picture something like uploading a company handbook and expecting the AI to memorise every policy, procedure
One agent writes, one edits, one SEO-optimizes, and one publishes. How we build autonomous content teams inside WordPress that scale your marketing without scaling your headcount
One model doesn't fit all. We break down our strategy for routing tasks between heavy reasoners (like GPT-4) and fast, local SLMs to cut business IT
costs by 60%
Don't rewrite your old code. How we use Multi-Modal agents to "watch" and operate your legacy desktop apps, creating modern automations without touching the source code
The argument for cloud-first AI infrastructure is straightforward: you pay only for what you use, you avoid capital expenditure, and you can scale instantly when demand spikes.
There is a particular species of regret that lives in boardrooms across Britain. It arrives about eighteen months after the champagne corks popped for the new ‘Head of AI’ hire.
There is a persistent myth in business technology circles that AI agents, once deployed, simply run themselves. The pitch sounds compelling:
configure once,
Last month, we built what we believed was a bulletproof AI automation for a client. Custom system prompts. Input validation. Output filtering. The works. We were confident.
When OpenAI announced GPT-4 Turbo with a 128k context window, and Google followed with Gemini’s 1 million token capacity, the message
seemed clear: