Agent Memory Architecture

A 6-layer memory system for AI coding agents, running daily in the reference setup — a 7-node Proxmox + Kubernetes homelab managed via ArgoCD GitOps. Reference implementation uses Claude Code, but the architecture is model-agnostic and maps cleanly to Cursor, Aider, Continue, GitHub Copilot, Cline, or any custom agent built on the Anthropic / OpenAI / Gemini APIs.

Maturity honest-statement: Layers 1-4 are validated and boring — copy freely. Layer 5 is standard practice with mature tooling. Layer 6 is research-grade and opinionated. See docs/getting-started.md for which layers to actually install.

Who this is for

If any of this sounds familiar, you're in the right place:

• Your CLAUDE.md (or .cursorrules or CONVENTIONS.md) is past 200 lines and the agent increasingly ignores the back half.
• You keep re-explaining the same project facts every session — you think "didn't we cover this yesterday?"
• You've tried MCP memory servers and either (a) weren't sure which one to pick or (b) the one you picked didn't obviously help.
• You have infrastructure knowledge that doesn't fit in a system prompt, and RAG feels like overkill for it.
• You're building a homelab / dev platform and want the agent to actually learn your environment, not just take instructions.

Heads up: this isn't a drop-in product. It's a reference implementation I actually run every day — opinionated, with templates you edit. You read it, apply what fits, throw out what doesn't.

Try it in 30 seconds:

curl -sSL https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/bootstrap.sh | bash -s -- --layer=2 --dry-run

(drops --dry-run to actually install; backs up any existing files).

How this came together

I started where most people start: a single CLAUDE.md at the root of every repo. It worked for a few weeks. Then it started failing in the same boring way every time — the file grew past 200 lines, instructions started getting ignored, the agent re-learned the same infrastructure facts every session, and I'd find myself pasting the same context into the chat again.

I added a wiki. The wiki worked, but the agent didn't always find the right article. So I added semantic search over the wiki. That worked, until I noticed I was also accumulating session-specific learnings ("we tried X, it failed because Y") that didn't belong in compiled wiki articles. So I added a separate vector collection for those. Then I noticed those learnings stayed forever — useful ones and stale ones treated identically — so I added cognitive memory with activation decay.

Six layers later, the agent stopped relearning. Sessions that used to start with 20 minutes of context-loading start with the right context already loaded. The token cost is lower, not higher, because each layer is targeted: the cheap layers handle the common case, the expensive layers only fire when the cheap ones miss.

This repo is what that ended up looking like — sanitized, templated, and documented enough that you can take any layer (or all six) and apply it to your own setup.

What six more months of running it taught me

Been running this daily since I published the repo, and the architecture itself
held up fine — I haven't changed the shape of it. What I got wrong was how much
I trusted my own measurements.

The thing I kept running into is that none of this fails loudly. When something
breaks in a memory stack it doesn't throw an error, it just quietly returns
results from a smaller corpus than you think you have, and everything downstream
looks normal. I found a semantic index that had never embedded a single article
body, so nothing written inside a document was retrievable, only titles. It
had been like that for months. The nightly eval was passing the whole time,
because the passing bar had been set from a measurement taken while the index
was already broken. There was also a health check pointing at a collection that
got renamed during a migration, reporting the same failure every day until the
message stopped registering as information.

Chunking the article bodies moved hit@5 from 0.70 to 0.90 on the same set of
questions, without touching the embedding model or the reranker.

I wrote seven of these up in docs/failure-modes.md,
with the measurements and what to check in your own setup. If you've already
built something like this and the numbers look fine, that's the page I'd read.

The Six Layers

Most setups stop at Layer 2. Some go to Layer 4 (Karpathy's "LLM Wiki" insight). This repo shows what Layers 5 and 6 look like in practice.

Architecture

graph TB
    subgraph "Session start (always loaded)"
        A[System instructions<br/>~120 lines]
        B[Memory file index<br/>~50 lines, hub-and-spoke]
        C[Path-scoped rules<br/>conditional on file path]
    end

    subgraph "On-demand retrieval (lazy)"
        D[Wiki knowledge base<br/>~120 markdown articles]
        E[Vector store<br/>semantic search over wiki]
        F[Vector store<br/>session learnings + work summaries]
    end

    subgraph "Cognitive memory"
        G[MSAM / Zep / Letta<br/>activation decay + knowledge graph]
    end

    A --> B
    A --> C
    B -. on demand .-> D
    D -. fallback .-> E
    A -. on demand .-> F
    A -. on demand .-> G

    style A fill:#5e6ad2,color:#fff
    style D fill:#2d5f3f,color:#fff
    style G fill:#FACC15,color:#000

Latency profile:

See docs/architecture.md for the full breakdown of each layer.

How this maps to your agent

The wiki (Layer 4) is completely tool-agnostic — it's just markdown. Layers 5-6 are exposed via MCP servers (Claude Code and Cursor support natively), or via direct REST API calls from any HTTP-capable agent.

See docs/adapting-to-other-agents.md for the full integration guide including a 30-line custom-agent example.

Layer 4 in practice: the wiki + interactive graph

Layer 4 is the most underrated piece. A well-curated wiki of compiled knowledge — markdown articles with [[wikilinks]] and YAML frontmatter — handles ~95% of what naive RAG does at near-zero operational cost. Most teams jump to vector search too early; a disciplined wiki solves the problem first, and RAG becomes the fallback for when the index doesn't surface the right article.

To make a 100+ article wiki actually navigable, you want an interactive force-directed graph of the [[wikilinks]]. Two paths:

1. Cosma (by Arthur Perret, GPL 3.0) — mature tool, generates a standalone HTML "Cosmoscope" with a built-in article panel. This is what the reference setup actually uses today. Install via npm install -g @graphlab-fr/cosma.

2. scripts/build-wiki-graph.py (this repo, MIT) — lightweight Python alternative that emits a graph.json you can render with D3 / vis.js / Cytoscape. Use it when you want more control or don't want a Node.js dependency. Not as polished as Cosma out of the box.

What's in this repo

templates/
  global/CLAUDE.md          Sanitized global system instructions
  project/CLAUDE.md         Sanitized project-level instructions
  rules/
    kubernetes.md           Path-scoped: loads only for kubernetes/**
    terraform.md            Path-scoped: loads only for terraform/**
    dockerfiles.md          Path-scoped: loads only for dockerfiles/**
    wiki.md                 Path-scoped: loads only for wiki/**
  memory-files/
    project_example.md      project-type memory file template
    reference_example.md    reference-type memory file template
    feedback_example.md     feedback-type memory file template
scripts/
  rebuild-memory-index.py   Audit memory files: orphans, stale, oversized, credentials
  build-wiki-graph.py       Generate interactive knowledge graph from [[wikilinks]]
  msam-mcp-wrapper.py       FastMCP wrapper for cognitive memory REST API
  check-sanitization.sh     Pre-publish scanner for secrets / leaked IPs / personal data
  hooks/
    pre-compact-save.sh     Preserve working state before context compaction
docs/
  README.md                 Reading order and "by role" index
  getting-started.md        Decision tree: which layer do you need?
  architecture.md           Deep dive on each of the 6 layers
  adapting-to-other-agents.md   Mapping this to Cursor / Aider / Copilot / custom
  layer-6-backends.md       Comparison of MSAM, Zep, Letta, Mem0, Basic Memory
  failure-modes.md          How a memory stack lies to you — 7 silent failures, measured
  sanitization.md           Pre-publish checklist if you fork this
examples/
  project_webapp-example.md       Filled-in project-type memory file
  reference_deploy-example.md     Filled-in reference-type memory file
  feedback_testing-example.md     Filled-in feedback-type memory file
  mcp-config-sample.json          Sample MCP config for Layers 5–6

Want to see the structure without installing anything? Run bash bootstrap.sh --demo and it creates a small example project in ./memory-arch-demo/ — nothing touched outside that directory.

Quick start

One-line installer that auto-detects your agent (Claude Code, Cursor, Aider) and installs templates at the right paths:

# From your project root — installs Layer 2 (system instructions)
curl -sSL https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/bootstrap.sh | bash -s -- --layer=2

# Or any other layer (1-6 or 'all'):
curl -sSL https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/bootstrap.sh | bash -s -- --layer=4

# Dry-run first if you want to see what it'd do:
curl -sSL https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/bootstrap.sh | bash -s -- --layer=all --dry-run

Or use this repo as a GitHub template: click the green "Use this template" button at the top of the repo to fork it as your own.

For the full decision tree (which layer to install when), see docs/getting-started.md.

Standalone scripts

Each script in scripts/ works on its own — no need to clone the whole repo.

# Audit your existing memory files (orphans, stale, oversized, credentials)
curl -O https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/scripts/rebuild-memory-index.py
CLAUDE_MEMORY_DIR=~/.claude/projects/<your-project>/memory python3 rebuild-memory-index.py

# Build an interactive graph from any markdown wiki with [[wikilinks]]
curl -O https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/scripts/build-wiki-graph.py
WIKI_DIR=./wiki python3 build-wiki-graph.py --output graph.json

# Sanitize a fork before publishing
curl -O https://raw.githubusercontent.com/futhgar/agent-memory-architecture/main/scripts/check-sanitization.sh
bash check-sanitization.sh

Sanitization

This repo was built fresh — never a fork of the private origin. Every IP, domain, UUID, and credential has been stripped or replaced with placeholders.

If you fork it, run the bundled checker before publishing your changes:

./scripts/check-sanitization.sh

See docs/sanitization.md for the full checklist.

License

MIT.

Acknowledgments

This reference implementation stands on the shoulders of open-source tools and public ideas. Specifically:

• Andrej Karpathy — the "LLM Wiki" insight that Layer 4 is built on: structured flat files with good cross-references outperform naive RAG at a fraction of the operational cost. This whole repo is a practical expansion of that idea.
• Cosma by Arthur Perret et al. (GPL 3.0) — the standalone HTML graph viewer used by the reference setup. Our scripts/build-wiki-graph.py is a JSON-emitting alternative for Python-only stacks.
• FastMCP — the Python framework scripts/msam-mcp-wrapper.py is built on.
• Qdrant — the vector store at Layer 5, for hybrid dense + sparse retrieval with server-side fusion. The reference setup has moved through several embedding models since this repo was first published; docs/architecture.md explains how to choose one rather than naming a winner, because re-embedding your corpus is the real cost of that decision.
• Alternative Layer 6 backends: Zep / Graphiti, Letta / MemGPT, Mem0, Basic Memory. Each linked where relevant in the docs; all battle-tested options beyond the MSAM wrapper included here.
• Path-scoped rules, CLAUDE.md hierarchy, /compact survival rules — documented by Anthropic in the official Claude Code memory docs. The rules format (paths: YAML frontmatter) is Anthropic's convention, adopted here.
• MCP (Model Context Protocol) — the Anthropic-initiated, now Agentic AI Foundation-governed open protocol that makes Layers 5-6 swappable across agents.

If you use this as a starting point and build something, please link back — that's the only "contribution" this repo asks for.

Next steps

A few things if this was useful:

• The deep-dive blog post this repo is based on: The 6-Layer Memory Architecture I Run for Claude Code
• Follow @jmrlad on X — I post updates there as the setup evolves
• Starring the repo helps me see what's resonating and what to write about next
• If you want help rolling this out in your org, Guatu Labs does exactly this kind of engagement

Open an issue if something's broken or unclear. PRs welcome for docs fixes, adapter guides for agents that aren't documented yet, or template improvements.

About

Built by futhgar at Guatu Labs — we help companies roll out AI agent infrastructure: memory systems like this one, agent fleets running under budget governance, and the operational tooling to keep them running without babysitting.

If this architecture solves a problem you're wrestling with and you want help implementing something similar in your org, get in touch.