llmman

Run any agent on any model.

Claude Code, Codex, OpenCode and friends, pointed at a model running on your own machine, or at any hosted provider, in one command.

llmman launch claude --model qwen3.8

That starts a local inference server, downloads a llama.cpp build matching your
GPU, loads the model, and execs an agent against it.

Models are OCI images, so moving one takes no tooling you don't already have:

llmman transfer hf.co/unsloth/Qwen3.5-0.8B-GGUF docker.io/owner/model:latest

That copies from Hugging Face into your own registry directly, without a copy
landing in your local store.

Why llmman?

• Provider agnostic. The same launch, run and list commands work
  against a local model or any hosted provider (--provider baseten,
  --provider groq, ...). One endpoint, one agent config, local or hosted.
• Registry agnostic. llmman run qwen3.8 pulls straight from Docker
  Hub; llmman run hf.co/org/model pulls straight from Hugging Face. Or
  package a model as a plain OCI artifact and push it to GHCR, quay, Harbor
  or a self-hosted mirror, then llmman run it from there. No curated
  library, no account with llmman, no gatekeeper.
• Vanilla everything. Upstream llama.cpp releases (or the
  llama-server already on your PATH), vllm and mlx-lm as-is, serving
  unmodified GGUF and safetensors files. No fork to wait on, no import step,
  no private blob format: the store is a standard OCI Image Layout that all
  can read.
• One-step transfer. Any source pull understands paired with any OCI
  registry destination, optionally signed, without a copy in your local store.
  That is the shape air-gapped and compliance-bound environments need.
• Pool your machines. Several llmman serve daemons form an
  aggregation: name the others, and a request to any of
  them runs on whichever node has the model loaded or the most room for
  it. A laptop, a workstation and a Spark look like one endpoint.

Install

Linux, macOS:

curl -fsSL https://raw.githubusercontent.com/llmmanorg/llmman/main/install.sh | sh

Windows (PowerShell):

irm https://raw.githubusercontent.com/llmmanorg/llmman/main/install.ps1 | iex

Quick start

Three commands cover most of it:

llmman launch claude --model qwen3.8  # an agent on a local model
llmman run qwen3.8                    # just chat with a model
llmman serve                          # an Ollama/OpenAI/Anthropic-compatible endpoint

Run llmman launch with no arguments to see the supported agents and whether
each is installed. Want a hosted model instead of a local one? Every command
above takes --provider; see Hosted providers.

Commands

Models are OCI artifacts

Models are packaged as standard OCI artifacts and stored in any compatible
registry: Docker Hub, GHCR, quay, self-hosted. There is no curated library and
no gatekeeper: push a model anywhere you can push a container image, and anyone
can llmman run it straight from there.

Pull a model

llmman pull qwen3.8

Transfer a model between locations

Transfer an image directly from a source to a destination without adding it
to your local store, e.g. HuggingFace straight to an OCI registry:

llmman transfer hf.co/unsloth/Qwen3.5-0.8B-GGUF docker.io/owner/model:latest

Any source llmman pull understands (an OCI registry, hf://, ms://, ...) can be paired with any OCI registry destination.

Sign a model, and verify what you pull

No gatekeeper means nothing vouches for a model implicitly, so llmman
verifies the artifact rather than trusting the hub. Signatures are
cosign-format, so cosign verify
reads what llmman writes and the check works the same on Docker Hub,
GHCR, quay, or an air-gapped mirror.

llmman push docker.io/myorg/mymodel:v1 --sign-key signing.key
llmman verify docker.io/myorg/mymodel:v1 --key signing.pub

A [verify] trust policy turns that into an automatic check on every
pull, warning or refusing outright per repository. Off by default —
there is nothing to check against until you have said whom you trust.
See docs/verification.md.

Serve

llmman serve

One endpoint on 127.0.0.1:17434 that speaks Ollama, OpenAI and Anthropic:

So any existing client works unchanged:

OLLAMA_HOST=127.0.0.1:17434 ollama run unsloth/Qwen3.5-0.8B-GGUF

Models load on demand, each in its own backend process, and unload after
five idle minutes (keep_alive, as in Ollama). GGUF is served by upstream
llama.cpp, downloaded to match
your GPU if no llama-server is on PATH; safetensors by
vllm, or by
mlx-lm on Apple Silicon. Tool
calling, vision, structured output, embeddings (GGUF) and the Responses
API (what Codex speaks) all work; there is a web UI at / and an
optional Prometheus /metrics.

The full endpoint list and per-API notes are in docs/api.md;
backend selection in docs/backends.md; bind address,
context length, keep-alive and the other daemon settings in
docs/configuration.md.

Aggregation

Several machines each running llmman serve can pool their hardware — a
group of manatees is an aggregation. Name the others on each node and a
request to any of them is served by whichever has the model loaded, or
the most room to load it:

llmman config set aggregation.peers asahi,spark
LLMMAN_HOST=0.0.0.0 llmman serve

llmman ps, /api/tags and /v1/models then show the whole
aggregation, and llmman stop reaches a model wherever it was loaded.
Nothing is elected and nothing is shared: every node is a whole llmman
that knows the others' addresses. See docs/aggregation.md.

Launch an integration

Point an integration at a model in one step. llmman launch starts serve
in the background if it isn't already running (preloading a local
model), then sets the right environment variables and execs the
integration:

llmman launch claude --model qwen3.8

Run llmman launch with no arguments to list the supported integrations
(Claude Code, OpenCode, Codex, Aider, Qwen Code, Gemini CLI, ...) and
whether each is installed. Any extra arguments after -- are forwarded to
the integration's own CLI. Short names work wherever a model reference is
accepted.

Hosted providers

--provider points llmman at a model it doesn't serve itself, from
launch, from run, and from list:

export OPENROUTER_API_KEY=...
llmman providers                                    # which providers, and is the key set
llmman list --provider openrouter                   # its models, and $/Mtok in and out
llmman run --provider openrouter qwen/qwen3-coder   # chat with one directly
llmman launch opencode --provider openrouter --model qwen/qwen3-coder

The provider list comes from models.dev, the same
catalog opencode uses, so a new provider works without an llmman
release. Requests still go through llmman serve; --provider changes
where the daemon forwards them, not who the client talks to, so a local
and a hosted model are one endpoint and one integration config. The key
travels per request; it is never written into an integration's config.

Which integrations support --provider, where the key comes from, and
why it is only ever sent to a loopback daemon are in
docs/providers.md.

Documentation