The End of “Context Stuffing”
Treating LLM Prompts Like CI/CD Data Pipelines
The Prompt Is a Runtime Environment, Not a Text Box
There is a comfortable lie circulating through every AI integration team right now: that bigger context windows have solved the data problem. The reasoning goes something like this. If a model can accept 128,000 tokens, you should fill it. Dump in the entire knowledge base, the full conversation history, every document that might be relevant, and let the model figure it out. This approach has a name. It is context stuffing, and it is the single largest source of wasted compute, hallucinated outputs, and unreliable agent behaviour in production systems today.
The alternative is not a minor tweak. It requires treating the LLM prompt the same way a software engineer treats a compiler environment. You would never pass every file in your repository to a build step. You pass the exact dependencies the current compilation unit requires, nothing more. The same principle applies to language models. The prompt is not a container for everything the model might need. It is a strict, scoped runtime environment where every token must earn its place.
This article breaks down the architectural shift from context stuffing to context engineering: managing the attention budget, loading data just in time, quarantining poisoned context, and routing the entire flow through deterministic pipelines borrowed directly from standard software practice.
The Attention Budget and Context Rot
Context rot is the downstream consequence. As an agent conversation extends over dozens of turns, the working context accumulates tool outputs, intermediate reasoning, and superseded instructions. Without active management, the model begins referencing stale data as though it were current. It hallucinates connections between unrelated earlier steps. The prompt has not grown more informed. It has grown noisier, and noise compounds.
The worst part is that context rot is invisible to most monitoring systems. The agent continues producing outputs that look structurally correct. The JSON is valid. The function calls are well-formed. But the reasoning behind those outputs is increasingly built on a foundation of outdated assumptions and orphaned data fragments. By the time the degradation becomes obvious in the final output, the root cause is buried twenty turns back in a context window that nobody is inspecting.
The practical takeaway is blunt: filling the context window is not a strategy. It is the absence of one. Effective context engineering starts with the assumption that every token injected into the prompt must justify its presence against every other token competing for that same finite attention.
Just-In-Time Context: Loading Data When the Agent Needs It
The amateur pattern is to pre-load. Before the agent begins work, the developer retrieves every document, every schema, every piece of reference material and packs it into the system prompt or the first user message. This approach feels thorough. In reality, it burns through the attention budget before the agent has taken a single meaningful action.
Just-In-Time context inverts this completely. Instead of front-loading data, the agent carries lightweight identifiers: file paths, database indices, API endpoint references, or structured pointers into a vector store. When the execution flow reaches a step that requires specific information, the agent calls a tool to retrieve exactly that chunk, injects it into the current turn, uses it, and then allows it to be compacted or dropped from subsequent turns.
The Model Context Protocol from Anthropic provides a standardised interface for exactly this kind of interaction. Rather than each tool integration inventing its own format for injecting context, MCP defines a consistent contract: the agent requests a resource, the server responds with structured content, and the agent incorporates it into the current reasoning step. This standardisation matters because it makes JIT context composable. You can swap data sources, add new tools, or restructure your retrieval layer without rewriting the agent’s core logic.
Consider a practical example. An agent tasked with reviewing a codebase does not need the entire repository in its context to answer a question about a single function. It needs the function definition, its call sites, and the relevant type signatures. A JIT approach maintains an index of the repository structure and retrieves only those specific files when the reasoning step requires them. The next step might need entirely different files, so the previous retrieval is compacted or dropped, and fresh context is loaded. The agent’s working set stays small and relevant at every stage.
The result is a dramatically leaner prompt at every step. The model reasons over a focused, relevant working set rather than sifting through a haystack for the one needle it needs right now. Token costs drop. Latency improves. And critically, the model’s reasoning quality goes up because every token in the window is pulling its weight.
Context Quarantine: Stopping Poisoned Memory
The defence is twofold. First, context quarantine: before appending any tool output to the working history, the system validates it against a schema or a set of structural expectations. If the output fails validation, it is flagged and either retried or excluded, never silently merged into the context. Second, context compaction: a smaller, cheaper model periodically summarises older conversation turns into compressed state objects. The original verbose history is dropped, replaced by a tight summary that preserves decisions and outcomes while shedding the raw noise. This keeps the working memory both clean and current.
Together, these two techniques form a defensive layer around the agent’s memory. Quarantine prevents bad data from entering. Compaction prevents good data from becoming stale clutter. The prompt stays lean, accurate, and focused on the present task rather than dragging along every artefact of every previous step.
The Pipeline Architecture: DAGs, Schemas, and Deterministic Routing
Everything described above — attention budgeting, JIT retrieval, quarantine, compaction — only works if the orchestration layer is rigorous. This is where standard software engineering principles enter the picture. We structure agent workflows as Directed Acyclic Graphs, where each node represents a discrete operation: retrieve data, validate schema, call the model, evaluate output, route to the next step. The graph enforces execution order and prevents circular dependencies, exactly as a CI/CD pipeline prevents a deploy step from running before tests pass.
Every data handoff between nodes passes through strict type validation. Tool inputs and outputs conform to JSON schemas. If a node produces output that does not match its declared schema, the pipeline halts or routes to an error-handling branch. This is not optional hardening. It is the minimum viable architecture for any system where a probabilistic model sits in the middle of a deterministic workflow.
Deterministic routing sits on top. Based on the validated output of each step, the orchestrator follows explicit conditional logic to decide the next node. There is no ambiguity, no hoping the model figures out what to do next. The model handles reasoning within its node. The pipeline handles flow between nodes. Keeping these responsibilities separate is what makes the system debuggable, testable, and predictable — three qualities that context stuffing actively destroys.
This separation also makes the system observable. Each node boundary is a natural logging point. You can inspect exactly what data entered the model, what it produced, whether that output passed validation, and which branch the router selected. When something goes wrong — and in production, something always goes wrong — you have a clear trace. Compare this to a monolithic prompt where a hallucination could have originated from any of the fifty documents crammed into the context. The pipeline tells you exactly where the chain broke.
The shift from context stuffing to context engineering is not a philosophical preference. It is a measurable architectural decision with direct impact on output quality, token cost, and system reliability. Every token in the prompt either contributes to the current reasoning step or it competes against the tokens that do. There is no neutral middle ground.
The patterns described here are not theoretical. They are production techniques used in agent systems that run thousands of tasks daily without human oversight. Attention budgeting prevents signal loss. JIT retrieval keeps the working set relevant. Context quarantine and compaction defend against poisoned and stale data. And deterministic pipeline routing makes the whole system inspectable, testable, and reproducible.
The tooling exists now. Structured retrieval protocols like MCP give agents a clean interface for JIT data loading. JSON schema validation makes every data handoff auditable. DAG-based orchestration brings the same rigour to agent workflows that CI/CD brought to software deployment a decade ago. The question is not whether these patterns work. It is whether your team is still hoping that a bigger context window will solve problems that only disciplined engineering can address.
Engineering the Prompt, Not Filling It
Ready to Activate Your Digital Advantage?
See for Yourself, Chat to Our Intelligent Assistant Now to See What We Can Do for You.
References
Anthropic Engineering Blog. “Effective Context Engineering for AI Agents.” https://www.anthropic.com/engineering
Anthropic. “Model Context Protocol (MCP) Documentation.” https://modelcontextprotocol.io
Liu, N., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. “Lost in the Middle: How Language Models Use Long Contexts.” Stanford University / UC Berkeley. https://arxiv.org/abs/2307.03172
Discover more AI Insights and Blogs
Large Language Models are remarkably coherent. They sound confident. They construct grammatically perfect sentences with
flawless logic.
When you flip a light switch, you don’t think about which power station generated the electricity. You simply expect the light to turn on. The same principle should govern how ...
When business leaders hear about fine-tuning large language models, many picture something like uploading a company handbook and expecting the AI to memorise every policy, procedure
One agent writes, one edits, one SEO-optimizes, and one publishes. How we build autonomous content teams inside WordPress that scale your marketing without scaling your headcount
One model doesn't fit all. We break down our strategy for routing tasks between heavy reasoners (like GPT-4) and fast, local SLMs to cut business IT
costs by 60%
Don't rewrite your old code. How we use Multi-Modal agents to "watch" and operate your legacy desktop apps, creating modern automations without touching the source code
The argument for cloud-first AI infrastructure is straightforward: you pay only for what you use, you avoid capital expenditure, and you can scale instantly when demand spikes.
There is a particular species of regret that lives in boardrooms across Britain. It arrives about eighteen months after the champagne corks popped for the new ‘Head of AI’ hire.
There is a persistent myth in business technology circles that AI agents, once deployed, simply run themselves. The pitch sounds compelling:
configure once,
Last month, we built what we believed was a bulletproof AI automation for a client. Custom system prompts. Input validation. Output filtering. The works. We were confident.
When OpenAI announced GPT-4 Turbo with a 128k context window, and Google followed with Gemini’s 1 million token capacity, the message
seemed clear:
Vector databases changed everything about how we build AI applications. The ability to encode meaning into mathematical space and find semantically similar content felt like