RLM Memory MCP Server

Persistent memory + semantic file discovery for ANY AI coding agent.

Works with Claude Code, OpenAI Codex, Gemini CLI, Cursor, Windsurf — anything that speaks MCP.

What is this?

AI agents forget everything between sessions. This MCP server fixes that:

• 🧠 Memory — after every task, the agent records what it changed and why. Next session, it remembers.
• 🗺️ File map — a semantic index of your codebase ("this file is the login form", "this is the checkout API"), so the agent finds the right files without grepping the whole repo.
• 🔄 Bi-directional — the agent asks the MCP ("user wants to fix the submit button — which files?") and the MCP answers with files, history, and suggestions.

The core idea (Recursive Large Model): the agent stays intentionally "blind" to the filesystem and uses the MCP as its eyes and memory — making it faster, cheaper, and more focused.

┌────────────────────────────────────────────────────────┐
│  AI Agent (Claude Code, Codex, Gemini CLI, Cursor...)  │
│                                                        │
│  "User wants to fix login" ──► rlm_query               │
│                            ◄── relevant files+history  │
│  [does the work]                                       │
│  "Here's what I changed"   ──► rlm_smart_memory        │
└────────────────────────────────────────────────────────┘
                          │ stdio (MCP)
┌────────────────────────────────────────────────────────┐
│  RLM Memory MCP Server                                 │
│  • JSON storage per project (projects/<name>/.rlm/)    │
│  • AI-powered matching via OpenRouter or Gemini        │
│  • Web UI for you at http://localhost:3848             │
└────────────────────────────────────────────────────────┘

Setup in 3 Steps

1. Install & build

git clone https://github.com/jumpino27/RLM-Memory-MCP-Server.git
cd RLM-Memory-MCP-Server
npm install
npm run build

2. Add an API key

Copy .env.example to .env and set one key:

# Option A (recommended): OpenRouter — one key, any model
# https://openrouter.ai/keys
OPENROUTER_API_KEY="sk-or-..."

# Option B: Google Gemini direct — https://aistudio.google.com/
# GEMINI_API_KEY="..."

• With OpenRouter the server uses google/gemini-3.5-flash by default — fast, cheap, near-Pro quality.
• With Gemini direct it uses gemini-3.5-flash.
• No key at all? Everything still works using keyword matching (just less smart).

Want a different model? Set LLM_MODEL (e.g. anthropic/claude-haiku-4.5 or openai/gpt-4o-mini on OpenRouter). See .env.example for all options.

3. Connect your AI agent

Replace C:\\path\\to with where you cloned the repo.

Claude Code (one command):

claude mcp add rlm-memory -- node C:\\path\\to\\RLM-Memory-MCP-Server\\dist\\index.js

OpenAI Codex CLI — add to ~/.codex/config.toml:

[mcp_servers.rlm-memory]
command = "node"
args = ["C:\\path\\to\\RLM-Memory-MCP-Server\\dist\\index.js"]

Gemini CLI — add to ~/.gemini/settings.json:

{
  "mcpServers": {
    "rlm-memory": {
      "command": "node",
      "args": ["C:\\path\\to\\RLM-Memory-MCP-Server\\dist\\index.js"]
    }
  }
}

Any other MCP client: launch node dist/index.js over stdio.

💡 Tell your agent how to use it: copy the rules from example_agents.md into your agent's instructions file (CLAUDE.md, AGENTS.md, .cursorrules, …).

The Workflow

First time on a project — index it once:

rlm_init → rlm_index_codebase → rlm_verify_index

Every task after that — three steps:

1. rlm_query          "User wants X — which files?"   → files + history + tips
2. (agent does the actual work)
3. rlm_smart_memory   "Here's what I changed"         → remembered forever

That's it. The more the agent works, the smarter the memory gets.

The Tools

Daily drivers

Project setup

Maintenance & extras

Example: rlm_query

{
  "project_name": "my-app",
  "user_request": "The user wants to fix the submit button color on the login form"
}

Returns:

{
  "relevant_files": [
    { "path": "src/components/LoginForm.tsx", "description": "Login form with submit button",
      "component_type": "form", "feature_area": "auth",
      "recent_changes": ["Added hover state to submit button"] }
  ],
  "relevant_memories": [
    { "summary": "Changed submit button to theme primary color", "date": "..." }
  ],
  "ai_analysis": "The submit button lives in LoginForm.tsx and uses theme.ts colors...",
  "suggestions": ["Check theme.ts for the color tokens"]
}

Example: rlm_smart_memory

{
  "project_name": "my-app",
  "user_prompt": "Fix the submit button color",
  "changes_context": "Changed the submit button in LoginForm to use the primary theme color instead of hardcoded blue. Added hover state.",
  "files_modified": [
    { "path": "src/components/LoginForm.tsx", "change_type": "modified",
      "change_summary": "Button color now uses theme.primary, added hover state" }
  ],
  "affected_areas": ["auth", "ui"]
}

The Web UI (for you, the human)

npm start   # → http://localhost:3848

• Browse all projects, memories, and the semantic file map
• Test every MCP tool from the browser
• Delete stale memories / file entries
• See live AI provider status (e.g. openrouter · google/gemini-3.5-flash)

Configuration Reference

All settings live in .env (see .env.example):

Scripts: npm start (web UI) · npm run mcp (MCP server directly) · npm run build · npm test (end-to-end smoke test) · npm run dev (UI with auto-reload) · npm run typecheck

How data is stored

Everything is plain JSON — no database needed:

RLM-Memory-MCP-Server/
└── projects/
    └── my-app/.rlm/
        ├── config.json       # project info
        ├── memory_log.json   # every recorded task
        └── file_map.json     # the semantic file index

Back up by copying projects/. Inspect with any text editor or the web UI.

FAQ

Does this work without an API key?
Yes — all tools fall back to weighted keyword matching. AI matching is just smarter.

Why store data centrally instead of in each repo?
One place to back up, browsable across projects in the UI, no .rlm clutter in your repos, survives repo deletion.

Which agent works best?
Any MCP-capable agent. The tool descriptions teach the agent how to use them, and example_agents.md has drop-in instructions.

rlm_query vs rlm_recall_memory?
rlm_query searches files + memories + edit history and adds AI analysis. rlm_recall_memory only searches memories by keyword. Use rlm_query.

How much does the AI cost?
Helper calls are small and run at low reasoning effort. With google/gemini-3.5-flash ($1.50/M input, $9/M output) typical queries cost fractions of a cent.

Project structure (for contributors)

src/
├── index.ts            # MCP server entry (stdio) — registers all tools
├── ui/server.ts        # Web UI (Express) at localhost:3848
├── services/
│   ├── llm.ts          # Multi-provider AI layer (OpenRouter / Gemini + fallbacks)
│   └── database.ts     # JSON file storage
├── tools/              # MCP tool implementations
├── schemas/index.ts    # Zod input validation
├── types.ts            # Shared types
└── constants.ts        # Paths, models, limits

License

MIT