localagents

Hand Claude Code's grunt work to a model running on your own hardware.

localagents is an MCP server that gives Claude Code a run_agent tool. Each call
starts a full headless Claude Code session — same tools, same CLAUDE.md, same
working tree — except its API traffic goes to a llama.cpp or vLLM server you run
instead of to Anthropic. Claude writes the brief, the local model does the work,
Claude reviews the result. Your Anthropic token budget goes on the parts that
need it.

A 27B Qwen on one GPU is perfectly capable of "add a CLI for this module and
tests to match"; Opus is better spent on the design conversation than on
watching pytest run. Two local agents in parallel can build the two halves of a
package against a pinned interface.

Status: early. It works, I use it daily, and the interface will move. It
targets llama.cpp and vLLM specifically; Ollama isn't a goal.

How it works

Claude Code (your session)
   │  MCP: run_agent(task, model=...)
   ▼
localagents ── spawns ──▶ headless `claude` (Agent SDK)
   │                          │  ANTHROPIC_BASE_URL
   │                          ▼
   └──── in-process shim ◀────┘   normalises requests, logs them,
              │                   translates backend errors
              ▼
   llama-server / vllm   (/v1/messages, on your machine or your LAN)

Three things make this more than an environment variable:

1. A registry that is probed live. models.yaml lists where servers are and a
   menu of model names. What each server is actually serving right now, its real
   context window, and how many slots are busy are discovered on every call. You
   bring models up and down by hand — the server never launches anything — and
   when Claude needs a model that isn't running it asks you for it by name.

2. A shim between Claude Code and the backend. Claude Code sends things local
   chat templates reject, and local servers fail in ways Claude Code doesn't
   recognise. The shim fixes both directions (details below) and writes a
   requests.jsonl per job so you can see exactly what went over the wire.

3. The same isolation model as Claude's own subagents. By default a job works
   in your tree, like the Agent tool does. isolation: worktree gives it a fresh
   git worktree on a local-agent/<job> branch, kept only if it changed something,
   with a diffstat in the job record so Claude can review it as a diff.

Requirements

• Python 3.12+ and uv
• Claude Code. The Agent SDK bundles its own claude binary, so nothing else to install.
• A server that speaks Anthropic's /v1/messages:
  • llama.cpp llama-server — start it with --jinja; add --slots --metrics
    to get occupancy and cache stats in the tool output.
  • vLLM with --enable-auto-tool-choice --tool-call-parser <parser>.
• A model that can actually drive Claude Code: solid native tool calling and a
  context window of 128k or more per request. Qwen3.8-27B works well.
  Smaller windows work but compact constantly; see Context windows.

Install

git clone https://github.com/ccebelenski/localagents.git && cd localagents
uv tool install -e .                  # `localagents` on PATH; editable, so repo edits apply
cp models.example.yaml models.yaml    # edit for your servers (gitignored)
claude mcp add --scope user local -- localagents --config "$PWD/models.yaml"

User scope means every project gets the local server. It inherits the cwd of the
Claude Code session that launched it, so run_agent defaults to that project's
tree. A project can carry its own ./models.yaml to override the registry.

If you'd rather keep it to one project, put this in that project's .mcp.json:

{"mcpServers": {"local": {"command": "localagents", "args": ["--config", "/path/to/models.yaml"]}}}

Restart Claude Code (or /mcp → reconnect) after adding it; MCP servers load at startup.

Using it

Claude picks it up like any tool. Ask for it by name and it'll do the right thing:

Use the local agent to add a --json flag to the CLI and cover it in the tests.

What Claude does behind that: list_models to see what's up, run_agent(task=…)
which returns a job id, then wait_job / job_status / job_log until it's done,
then reads files_touched (or the worktree diff) and checks the work. Jobs that
outlive Claude Code's 2-minute tool timeout get backgrounded and picked up later;
you don't have to do anything.

If nothing suitable is running you'll be asked to start one:

qwen3.8-27b is not running anywhere. Ask the user to bring it up.
Notes: default mid-size coder on llama.cpp; run with --reasoning on

Start it however you normally do, say "it's up", and Claude retries.

Tools

Job records live in ~/.local/state/localagents/jobs/<job>/: transcript.txt
(what the agent said and did), events.jsonl (every SDK message), requests.jsonl
(every backend request with timing, size and usage), and requests_full.jsonl if
you turn on request dumping.

Config: models.yaml

Start from models.example.yaml. It's re-read on every call, so edits take effect
immediately, and the server never rewrites it — register_* write to a sidecar
models.local.yaml that is merged on top.

endpoints:
  llamacpp:
    base_url: http://127.0.0.1:8080
    backend: llama.cpp
  gpu-server:
    base_url: http://gpu-server.lan:8000
    backend: vllm
    host: gpu-server

models:
  qwen3.8-27b:
    notes: default mid-size coder on llama.cpp; run with --reasoning on
  deepseek-v4-flash:
    host: gpu-server
    notes: vllm needs --enable-auto-tool-choice --tool-call-parser deepseek_v3

• endpoints are places that serve /v1/messages. What they serve is probed.
• models are just names. A name is fuzzy-matched against served ids
  (qwen3.8-27b finds unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL), so an entry needs
  nothing but notes and maybe a host to relay when asking you to start it.
  served_name (exact id or glob), endpoint, context (fallback window) and
  bring_up (a start command) exist as overrides if you want them. Start
  commands go stale quickly; a name and a note usually age better.
• defaults cover the default model, permission_mode (acceptEdits), allowed
  and disallowed tools (subagents can't spawn subagents), which Claude settings to
  load, max_turns, timeout_s, and a system-prompt suffix telling the agent it's
  a delegate and how to report back.

What the shim does

Both llama.cpp and vLLM speak /v1/messages natively, so pointing
ANTHROPIC_BASE_URL at them almost works. The shim closes the gaps:

System messages mid-conversation. Claude Code puts role: system entries
inside messages — the skills listing, a token-budget marker, and one more per
turn. Qwen's chat template refuses: "System message must be at the beginning".
The shim folds each one into the adjacent user message as a <system>…</system>
text block, in place. Hoisting them into the top-level system field instead
changes the start of the prompt every turn, which invalidates the server's
KV-cache prefix and re-evaluates the whole ~35k-token prompt each time (21–47 s
per turn on a 27B). Folding in place keeps the prompt append-only: f_sim_best
0.88–0.99 in llama-server's log, 2.5–14 s per turn.

Context overflow. Claude Code assumes a 200k window for any model it doesn't
recognise; with a smaller slot it runs into llama.cpp's
exceed_context_size_error, which it doesn't understand, and the job dies. See
the next section.

Everything the shim does is a no-op when it isn't needed, and every request is
logged with its timing, message count, byte size and reported usage.

Context windows

Two layers keep a session inside the real window:

1. The probe reads it — llama.cpp /props n_ctx (per slot: -c divided by
   --parallel when unified KV is off), vLLM max_model_len — and the session
   gets CLAUDE_CODE_MAX_CONTEXT_TOKENS. Claude Code's own auto-compact then fires
   at the right point. Below 128k the output budget is also shrunk to n_ctx/8,
   because the compact threshold is window − max_output and would otherwise sit at zero.
2. If a request still overflows, the shim rewrites the backend's error into
   Anthropic's prompt is too long: N tokens > M maximum, which Claude Code
   answers by compacting and retrying.

At 64k this works but thrashes: Claude Code's ~20k of fixed prompt and tool
schemas, plus a ~7k-token compaction summary (~55 s on a 27B) and the files it
re-attaches, refill the window within a few turns and its thrash guard ends the
job. Give each slot 128k or more.

Backend notes

• llama.cpp: llama-server -hf <gguf> --jinja -fa on --slots --metrics, plus
  --reasoning on for thinking models and --parallel N for concurrent jobs. With
  /slots on, list_models shows {total, busy, free} so Claude knows whether a
  second agent will run now or queue. With /metrics on, each job records prompt
  tokens processed vs. cached, cache hit ratio, prompt and generation tok/s, and
  speculative-decode acceptance — the counters are server-wide, so overlapping jobs
  share the delta.
• vLLM: vllm serve <model> --served-model-name <alias> --enable-auto-tool-choice --tool-call-parser <parser>. The context window comes from max_model_len in
  /v1/models; occupancy (requests_running/waiting, kv_cache_usage) and the per-job
  cache stats come from its always-on /metrics. vLLM reports context overflow on the
  Anthropic endpoint as HTTP 500 internal_error; the shim translates that too. The session
  is started with CLAUDE_CODE_ATTRIBUTION_HEADER=0 because the per-request attribution
  hash defeats prefix caching. Verified with DeepSeek-V4-Flash: thinking blocks replay with
  their signatures, prefix cache hits on every turn after the first.
• The first turn of a job costs about 20k tokens of prompt on a cold slot (system
  prompt plus tool schemas), ~10 s on a 27B. Everything after that is a cache hit
  plus the delta.

Development

uv sync --dev
uv run pytest -q

See CONTRIBUTING.md for the layout and how to test changes
against a real server.

License

MIT. See LICENSE.

Copyright © 2026 Chris Cebelenski