strix-halo-inference

A Strix Halo as a measured inference machine: a local coding agent in
production — and text-to-image, speech and video under the same memory
authority. One configuration — Ryzen AI Max+ 395 (gfx1151), 128 GB of
shared memory — and every number here was taken on it, with the date and
method beside it.

bash setup/preflight.sh              # is this repo for your machine?
bash setup/install.sh                # once — writes ~/.config/llm-stack.env
bash setup/switch-model.sh halogen-qwen38flash # (alias: halogen) serve the high-speed Halogen container
# or: bash setup/switch-model.sh halogen-qwen38  # serve 27B via Halogen
# or: bash setup/switch-model.sh qwen38         # serve GGUF via llama-server
bash tests/run.sh                    # the gate (1449 tests, ~20 s, no GPU)

The name you fetch is the name you serve. There is no pull command, because
a model is its profile in setup/env/ — and the profile
carries where the weights come from, what a token of context costs in KV, and
the measurement behind every flag on its command line.

Does this fit your machine?

bash setup/preflight.sh answers it in five seconds, without root, changing
nothing:

Nothing is scaled down to change that — a scaled number is a guess. Not a
generic stack, on purpose; preflight.sh says where you stand before you
spend an afternoon.

The engines: Turnkey performance on Strix Halo

Instead of fighting ROCm compilation flags or manual container mounts, the
stack provides turnkey configurations for the fastest inference engines on
this silicon:

• Halogen Flash Server (setup/halogen/): A rootless
  container backend running Qwen 3.8 Flash-Next with MTP speculative decoding.
  The fastest solution on this hardware for daily agent workloads, delivering
  top token throughput.
• llama.cpp (setup/scripts/build-llama.sh):
  Upstream master plus a small curated set of hardware patches
  (setup/patches/) for the broad GGUF model
  ecosystem.

Switching between them is one command:

bash setup/switch-model.sh halogen-qwen38flash # switch to Halogen Flash Server (or: halogen)
bash setup/switch-model.sh halogen-qwen38      # switch to Halogen Server (27B)
bash setup/switch-model.sh qwen38              # switch back to llama.cpp

The gateway shields your clients completely: changing the underlying engine
requires zero configuration changes in your editor or harness.

The Gateway: Universal interface and instant turns

The LLM gateway (setup/gateway/) sits between your clients
and the serving engine:

• Dual-dialect translation: Speaks Anthropic (/v1/messages) and OpenAI
  (/v1/chat/completions) in-process. Claude Code, DeepSeek Harness, Cursor,
  and scripts talk to whichever backend is running without separate bridge
  daemons.
• True prompt and prefix caching: Agent prompts often carry 20k–40k tokens
  of system instructions and tool definitions. The gateway tracks prefix hashes
  and KV states, turning a ~120-second cold prefill into a 1.3-second
  follow-up turn (>90% cache reuse).
• Edge-stable streaming: Incremental token-by-token parameter streaming for
  tool calls and 10-second SSE keepalive heartbeats prevent Cloudflare Tunnel
  and proxy dropouts (500 / 524) on long generations.
• Stable model aliases: Use local-low, local, or local-medium in your
  clients. They resolve to the active engine's equivalent mode, so switching
  models never breaks your client configuration.
• Dedicated CPU Vision Sidecar: Exposes qwen3-vl-4b and vision alongside
  the primary GPU model. Runs on 8 dedicated Zen 5 CPU cores (CCD1: cores 8–15)
  via llama-vision.service with an independent admission gate (VISION_GATE),
  protecting Halogen GPU decoding from memory bus contention. Auto-starts on
  demand and stops after 10 minutes of inactivity (VISION_IDLE_TIMEOUT=600) to reclaim RAM for the Linux
  page cache. Fully compatible with Unity Asset Inventory, DeepSeek Harness,
  and OpenAI multimodal clients.

Memory authority: Never freeze the machine

On unified memory architectures like Strix Halo, there is no discrete VRAM.
The GPU allocates host RAM through GTT, and that allocation is pinned.
A workload that exceeds available memory does not page out and does not get
OOM-killed: it hard-freezes the entire machine, taking down all processes
without writing to kernel logs.

setup/lib/budget.py weighs every profile before start
and actively refuses if it does not fit:

REFUSING TO START qwen38: it needs about 70.1 GiB and it does not fit.
    the host has 44.2 GiB available, 12 must stay free

For the same reason, never start a second model — or a media workload — by
hand: python3 bench/sideserver.py is the only safe way, stopping production,
metering memory ceilings, and putting production back.

Not only a language model: Multimodal tenants

Since 01.09.2026 the same machine renders images, speaks and films under the
same memory authority. Measured on this box, n=3 each, idle machine, every
output machine-judged:

Each declares its measured footprint in setup/workloads/,
guarded by budget.py. The base install remains torch-free (the ~20-second
test gate proves it); torch workloads stay contained behind media/.

Why not a generic runner (like ollama)?

Generic runners get an endpoint running quickly, but they are built for
standard discrete GPUs or generic CPU fallback. On unified-memory APUs like
Strix Halo, that leaves crucial gaps:

We measure, not claim

Nothing in this repository is based on estimates or marketing claims:

• Every flag is measured: Speeds, context windows, and KV costs carry
  their date and measurement method directly beside them in the profile
  comments.
• Defects as testable data: Silent hardware corruptions (e.g. ////
  degeneration) and upstream bugs are recorded as data in
  setup/defects.json. python3 setup/lib/defects.py
  verifies whether your running build is affected.
• Rigorous test gate: bash tests/run.sh runs 1449 tests in ~20 seconds
  without needing a GPU, verifying parser integrity, dialect conversions,
  budget calculations, and guardrails before anything touches production.

Where to start