#185 — Your AI agent has a memory problem. Here's how to fix it.

RAG retrieves external documents to answer a question. Agent memory stores and retrieves what happened inside the conversation itself — decisions made, facts established, attempts that failed, and user preferences. RAG is about knowledge; agent memory is about continuity and state. In production, you'll almost certainly need both: RAG for grounding answers in your knowledge base, agent memory for preserving what was decided in the thread.

This is one of the most underrated failure modes in production agents. When layers disagree — say, your rolling summary preserves an old plan but structured memory has the final decision — you need explicit conflict rules: prefer structured decisions over summaries, prefer newer records over older ones, and prefer scoped memory (user- or project-specific) over generic memory. Tag every memory record with source, timestamp, scope, and status (active, superseded, rejected, archived). Without this, your agent will confidently hallucinate based on stale evidence.

The moment your agent needs to resume a task, avoid repeating a failed approach, or answer 'what did we try before?' — which happens earlier than most founders expect. A coding assistant that forgets the global patch already failed will suggest it again. A customer success bot that doesn't remember a client escalated last month will miss the tone entirely. Episodic memory is what makes an agent feel like it tracks progress, not just text. Start logging it as soon as your agent takes actions or makes recommendations across multiple turns.

LangChain memory classes are useful for demos and short sessions. For production, they fall short on durability, conflict handling, scoping, and governance — they don't give you queryable structured facts, they don't preserve episodic history across deployments, and they don't let you audit which memory surfaced a wrong answer. Use LangChain for orchestration and retriever interfaces, but back your memory with a real database that supports per-user scoping, access controls, and timestamp-aware freshness policies.

The answer is almost always in the context package assembled for that specific turn. Inspect it directly: Was the rolling summary stale? Did vector retrieval return an outdated chunk? Was the authoritative structured decision missing? Did episodic memory omit a failed attempt? The single most valuable architectural decision you can make is to log the full context package on every turn — what was assembled, from which layer, and with what priority. Without that, you're debugging a black box and guessing.

If done right, it should reduce your token costs, not increase them. The whole point of layered memory is to replace a giant transcript with a minimal, targeted context package. A sliding window of 6 turns plus a 200-token rolling summary plus 3 structured facts is almost always smaller and more useful than 40 raw conversation turns. The expensive failure mode is stuffing everything into one prompt — layered memory with a good memory manager is the fix, not the cause.

For a B2B SaaS agent serving real users across sessions, the practical floor is: (1) store every message per user and thread, (2) rolling summary for threads over 10 turns, (3) structured memory for account preferences, decisions, and configuration state, and (4) episodic logs for any action the agent took or recommended. Vector retrieval can come later. The goal at MVP is to never make a customer repeat themselves and to never recommend something you already tried.

All three — and mixing them up is one of the most common production bugs in multi-tenant AI systems. A user's tone preferences belong to their profile. A specific debugging decision belongs to that thread. A product roadmap fact belongs to the project. Without explicit scoping on every memory record, you'll bleed context between customers or between sessions — which is both a product failure and a potential data privacy liability. Enforce scope at write time, not retrieval time.

Shared memory in multi-agent systems requires a centralized, governed memory layer — not per-agent in-process state. Each agent should read from and write to the same structured and episodic memory store, with clear ownership rules: which agent can update which memory type, and which records are read-only once committed. Without this, agents will contradict each other or act on stale facts. This is where a database-backed memory layer with access controls becomes non-negotiable rather than just convenient.

Apply the same bar you'd hold any other production system to. Memory is production-ready when it's inspectable (you can see what was stored and why), updateable (decisions can be superseded, not just appended), scoped (records belong to a user, thread, or task — not globally), governed (access controls exist), and debuggable (when the agent gives the wrong answer, you can trace which memory layer was responsible). If you can't trace a bad answer back to its source, your memory system is not production-ready.

Conversation summary memory compresses what was said — it's a lossy representation of the dialogue over time. Episodic memory records what happened and why it mattered — specific events, outcomes, and failed attempts with timestamps. A rolling summary might tell you the team discussed a deployment issue. Episodic memory tells you the global patch was explicitly rejected, the EU-only patch passed staging, and the rollout was agreed for Thursday. For task resumption and avoiding repeated mistakes, episodic memory is the layer summaries cannot replace.

Vector retrieval fails when semantic similarity doesn't equal relevance or correctness. A retrieved chunk can be topically related but outdated, from a different project scope, or superseded by a later decision. It also struggles with precise structured facts — asking 'what timeout value did we set for EU checkout?' is better answered by structured memory than a similarity search. Use vector retrieval for paraphrased follow-ups and long-range semantic recall, but always rank it below explicit structured decisions in your memory manager's priority order.

The key is separating state from dialogue. Dialogue can be summarized and compressed. State — current task phase, open decisions, pending actions, known constraints — needs to be stored as structured memory with a clear schema and update rules. Episodic memory should log every meaningful event and outcome with a timestamp. When the user returns after two weeks, your memory manager should be able to reconstruct 'where we left off' from structured state and recent episodes — not by replaying the entire transcript.

At minimum, you need four tables: conversation_memory (thread_id, role, content, turn_id, timestamp), structured_memory (thread_id, memory_type, memory_key, memory_value, scope_json, status), episodic_memory (thread_id, event_type, description, outcome, timestamp, status), and vector_memory (thread_id, chunk_id, text, embedding, source, timestamp). Every record should carry scope, timestamp, source, and a status field (active, superseded, rejected, archived) so your memory manager can enforce freshness and conflict rules without deleting historical context.

A reliable priority order is: (1) current user message, (2) recent conversation turns, (3) active structured decisions and project state, (4) relevant episodic events, (5) rolling summary, (6) vector-retrieved chunks, (7) archived/superseded memory only when needed to explain why a path should not be repeated. The goal is the smallest useful context package that is current, scoped, and explainable — not the largest context window you can fill. Structured decisions should always rank above summaries and vector results when they conflict.