Architecting Production LLM Agents: Tools, Memory, and Guardrails
Every team can build an LLM agent that demos well. Far fewer can build one that survives contact with real users, real data, and real edge cases. The gap between those two things is almost entirely architecture.
We've shipped agents that take actions against production systems - updating records, calling internal APIs, routing work to humans. This is how we structure them so they stay reliable, observable, and safe to change.
An Agent Is a Distributed System, Not a Prompt
The most common mistake is treating an agent as a clever prompt with some tools attached. That framing leads to a single, sprawling system prompt, ad-hoc tool calls, and no way to tell why the agent did what it did.
A production agent is better understood as a distributed system with one non-deterministic component. It has state, it calls external services that fail, it retries, and it makes decisions you need to audit. The moment you frame it that way, the right architecture follows: typed interfaces, explicit state, structured logging, and failure handling at every boundary.
The model is the least reliable part of your system. Everything around it exists to contain that unreliability.
Tools: The Most Important Surface
An agent is only as good as the tools it can call. We treat tool design as the core API design problem it actually is.
Every tool has a typed schema. Inputs and outputs are validated against a schema before and after the call. If the model hallucinates an argument - and it will - the call fails loudly at the boundary instead of corrupting downstream state.
Tools are idempotent wherever possible. Agents retry. If a tool that creates a record runs twice, you get duplicates. We design mutating tools to accept an idempotency key so a retry is safe by construction.
Tools return structured errors the model can reason about. A tool that fails should return a message the agent can act on - "customer not found, ask for a valid ID" - not a stack trace. Good error messages turn dead ends into recoverable steps.
Permissions live in the tool, not the prompt. Never rely on the system prompt to stop an agent from doing something dangerous. The tool itself enforces what's allowed - which customers it can touch, what actions it can take - so a jailbreak can't escalate privileges.
Memory and State
"Memory" gets overloaded. In practice we separate three things:
-
Working state - the current task: what the agent is doing, what it has tried, intermediate results. This belongs in an explicit state object managed by the orchestration layer, not stuffed into conversation history.
-
Conversation history - the dialogue with the user, trimmed and summarized to fit the context window without losing the thread.
-
Long-term knowledge - facts the agent retrieves on demand from a vector store or database. This is retrieval, not memory, and it should be evaluated like any retrieval system.
Conflating these is how agents end up with bloated context windows, slow responses, and unpredictable behavior. Keeping them separate is how you keep the agent fast and debuggable.
Orchestration: Why We Reach for Graphs
Simple agents can run as a loop: call the model, run a tool, repeat. Real workflows need branching, retries, human-in-the-loop steps, and durable execution that survives a crash. That's where an explicit graph - LangGraph, Google ADK, or a hand-rolled state machine - earns its place.
Modeling the workflow as a graph of nodes and transitions makes the agent's behavior inspectable. You can see which path it took, replay a run, and add a node without rewriting the whole thing. It also makes durable execution natural: if the process dies mid-task, you resume from the last committed state rather than starting over.
We don't always reach for a framework. For a single linear workflow, a framework can add more concepts than it removes. We choose based on whether the workflow's branching and durability needs justify the abstraction.
Guardrails
Guardrails are the difference between an agent you can put in front of customers and one you can only show in a sandbox.
Input guardrails screen for prompt injection and out-of-scope requests before the model ever runs. Output guardrails validate that the response is well-formed, on-policy, and doesn't leak data. Action guardrails - the most important - sit in the tools and enforce what the agent is permitted to do.
On top of these, we route on confidence. When the agent is uncertain, it escalates to a human rather than guessing. The cost of a wrong autonomous action is almost always higher than the cost of asking.
Observability and Evals
You cannot operate what you cannot see. Every agent we ship traces every model call, every tool call, and every decision, with enough structure to answer "why did it do that?" weeks later.
And every agent ships with an evaluation suite. Without evals, "we improved the prompt" is a vibe, not a fact. With evals, a change is scored against a labeled set on accuracy, escalation rate, and latency, and run in CI so a regression is caught before it reaches users. Evals are what let you change an agent at all without fear.
Where to Start
If you take one thing from this: design the tools and the evals first. The tools define what the agent can do; the evals define how you know it works. Get those right and the rest of the architecture has something solid to stand on.
If your team is building an agent that needs to run reliably in production, our AI Agent Development practice exists for exactly this - turning a promising prototype into a system you can trust.
The agents that make it to production aren't the ones with the cleverest prompts. They're the ones with the most boring, disciplined engineering around the model.