shura

A 35B coding model on one gaming GPU, on a PC you keep using

English · Русский

Shura (شورى) is Arabic for "consultation, council". In a Mixture-of-Experts model every token is
decided by a council: a router consults 8 of 256 experts. This project is about seating that council
on modest hardware.

What is Shura

A 35B Mixture-of-Experts coding model runs on an ordinary home PC with one gaming graphics card. No
server, no cloud. And while the model works, the computer stays yours: the browser, video, a call and
the IDE keep running on the same GPU without stutter.

How that is done, in one breath (the working build; the Status section says what is in
main today): the model's hot experts live in VRAM and the rest are computed from
system RAM; a governor inside the inference server watches free VRAM every 200 ms and gives memory
back to the desktop the moment the desktop needs it; when it has to stop, it checkpoints and continues
the same answer after a restart; the KV cache is compressed so a 200k-token window fits; the
model's own MTP head drafts tokens; a small gateway loads the model on demand, keeps sessions in
RAM and protects your SSD; ShuraCode, an offline coding agent, sits on top. The engine is our fork of
llama.cpp, hikkian/llama.cpp-shura.

[!IMPORTANT]
These numbers come from one machine and from the working build that is being prepared for 1.0
(see Status). Other GPUs, CPUs and memory speeds will differ, and the setup
needs re-tuning there. Raw measurements and the method are published with the 1.0 benchmark set.

Where it's going: a model that grows with you

Today every local model starts from zero. A new one comes out, you switch, and everything the old one
learned from working with you is gone. Shura is meant to work the other way: one model, on your own
PC, for years, and it keeps getting better at your work, mostly at how it reasons and solves your tasks,
not just at remembering facts.

It grows in four layers, from cheap to hard:

The rules we hold to: nothing gets worse (every change is measured on your tasks before and after, and
rolled back if anything regresses); it learns only from what you confirmed, from the conversation, never
from watching you; everything stays local; it is gentle with the SSD and the desktop; and when a better
base model fits your card, your memory, rules and examples move with you. We publish the numbers either
way, including when an idea does not work.

The longer story and the progress live on the blog:
I'm building a local LLM that actually gets better the longer I use it.

Status: preparing 1.0

This README describes the project as it is measured today. Not all of it is in the published main
branch yet. To be exact:

What this README does not claim: a side-by-side number against upstream llama.cpp of the same day
(that measurement is planned for the 1.0 benchmark set), desktop smoothness as a number (the earlier
metric turned out to be insensitive; a better one is in the works), and anything about hardware we have
not run ourselves.

Results

All numbers: working build, one machine (RX 7800 XT 16 GB, Ryzen 5 5600, 32 GB DDR4, ROCm 7.2.4), with a
live KDE desktop holding 1.3–1.6 GiB of VRAM during the runs. Speeds are server-side, wall-clock tokens
per second of real requests through the gateway (not llama-bench), averaged over three alternating
pairs against the previous build; MTP speculative decoding is on in the short layout.

Speed (2026-10-24)

The 100k numbers come from the long-context layout (200k window, more experts on the CPU, no MTP:
measured, speculative decoding does not pay off there). Switching between the short and the long layout
keeps the session and takes 20–28 s.

The desktop comes first (2026-10-25)

The mechanism is an elastic expert cache: the hot experts in VRAM are copies of experts that also live in
RAM, so giving them back costs speed, never correctness. The governor runs inside llama-server, polls
the card every 200 ms and resizes the cache between compute steps.

Sessions survive (2026-10-17, 2026-10-25)

It runs for days (2026-10-24)

The agent (2026-10-24)

ShuraCode on Tiel, run the way the product runs (serve mode through the gateway, short layout), on the
dev split of our agentic task set: 73 of 111 runs solved (66 %): bug fixes 86 %, tests-only tasks
100 %, implement 41 %, refactor 36 %. The harness is in bench/agentic-eval; the task set used
here (224 verifiable tasks in 56 families) is published with 1.0.

Choosing the models (2026-10-28)

The same agent and the same 56 tasks × 2 runs for four local models on the same card. Tiel stays the default: the
fastest, and the best at fixes and tests. Sage-A1.5 becomes the second model, picked with /model for bigger
implement and refactor work, where it solves clearly more at about twice the time. Occamy matched Tiel without
adding anything and is no longer shipped; Gemma 4 is kept for a later general-purpose role, not for code.

Quality gate. Every change to kernels or layouts is accepted only if the KL divergence of the output
distribution to the reference layout stays within the gate (0.0385 on a fixed corpus) and the top-1 token
agreement does not drop; levers that failed the gate are off. The method and the full tables are
published with the 1.0 benchmark set; the September measurements on the 12 GB card are in
docs/BENCHMARKS.md.

How it works

The experts of the first layers stay in system RAM and the CPU computes the cold ones on 6 threads; the
most-routed experts of every layer sit in VRAM, chosen by a routing profile traced from generated tokens,
and the two halves are merged exactly, so the output is the same as without the cache. The KV cache is
compressed (TurboQuant) so that a 200k window fits next to the weights. The model's own MTP head drafts
tokens for speculative decoding. The gateway starts the model on demand, survives crashes, switches
text/vision automatically, keeps the prompt cache in RAM and parks the session in RAM when the desktop
needs the VRAM. In the working build the size of the expert cache is no longer fixed: the governor in
llama-server sizes it to whatever VRAM the desktop is not using, down to zero (see
Status). Design and endpoints: docs/ARCHITECTURE.md.

What we learned

• A fixed VRAM split cannot be both fast and safe. The desktop's GPU memory swings by hundreds of
  MiB (calls, video, the lock screen, a new tab). The answer is not a bigger reserve but a cache that
  shrinks in a quarter of a second and grows back when the desktop is quiet.
• The expert cache is the biggest lever, and too many slots fail silently. Oversubscribed VRAM does
  not raise an error, it slows down run over run.
• Prompt experts ≠ generation experts. A routing profile must be traced from generated tokens.
• TurboQuant is not a speedup by itself. Its value is turning saved KV VRAM into more expert slots.
  It also needs a fix on head_dim-256 models.
• Hybrid models need context checkpoints, and slot save/restore does not keep them. Without them any
  change in the prompt prefix forces a full re-read.
• MemAvailable lies about your browser. Loading 14 GB of weights through the page cache pushes the
  browser into swap while the kernel still reports plenty of "available" memory. The model has to load
  without touching the page cache and count only honest free memory.
• Speculative decoding does not always pay. With many experts on the CPU, verifying a draft costs
  almost as much as generating the tokens; in the long-context layout MTP is off, by measurement.
• A GPU page fault is usually a bug, not the driver. Ours was an out-of-bounds read in the MoE
  matmul path, visible only on long, growing sessions.

All experts on CPU with a large expert cache · n-gram drafting on top of MTP · fusing the small kernels of
a decode step (a few percent at most) · a separate 100k "normal" profile with more slots · more than 24
expert slots at 200k context on the 12 GB card · Vulkan for this stack (the compressed KV cache produced
garbage and MTP accepted nothing). Details: docs/BENCHMARKS.md.

Quick start

On Fedora, Ubuntu/Debian or Arch with an NVIDIA GPU and its driver installed (the published installer
path; the AMD/ROCm path used for the numbers above arrives with 1.0):

curl -fsSL https://raw.githubusercontent.com/hikkian/shura/main/setup.sh | bash
# or: git clone https://github.com/hikkian/shura.git && cd shura && ./setup.sh

The installer picks the Tiel-Coder quant that fits your RAM and VRAM, builds llama.cpp for your GPU,
measures a few layouts on your card and installs everything. It asks before each sudo step. Options
(--quick, --quant, --model, --no-search) and the list of everything it changes:
docs/INSTALLER.md. Manual setup: docs/INSTALL.md.

Daily use

Everything starts with the desktop session; nothing needs to be launched by hand.

• App menu → Shura. Pick a project folder and ShuraCode opens there in its own window. Right-click the
  icon for Load model now / Unload model. Works with one of the supported terminals
  (SHURA_TERMINAL chooses one; only Ghostty has been tested).
• Terminal: cd into a project and type shura. The model starts loading in the background.

shura status    # model state, free RAM/VRAM, idle timer
shura warm      # load the model now          shura unload   # free RAM/VRAM now
shura off / on  # disable / enable the AI     shura logs     # follow logs
shura window    # new Shura window for the current folder

The model unloads itself after 30 idle minutes; the last long session is saved and restored on the next
start.

ShuraCode is the coding agent: permanent memory shared by all sessions, the commands /remember
/forget /memory /test /commit, a read-only Plan mode next to Build (Tab), and a live model
status in the footer. It is our customization layer on top of the OpenCode engine, in its own repository:
hikkian/shuracode; the installer sets it up.

Will it run on my computer?

• Backend. By default the installer uses Vulkan (it works on NVIDIA, AMD and Intel) and falls back to the CPU if that
  fails. Another backend can be chosen with --backend.
• Memory. The catalog model (35B-A3B, MoE) needs about 20 GiB of memory on the CPU alone. A machine with 16 GB of RAM and no
  graphics card will not fit, and the installer says why. A graphics card with 6 GB of memory and 8 GB of RAM will not fit
  either. A graphics card with 8 GB of memory and 32 GB of RAM fits: some layers stay in RAM (a hybrid layout).
• Speed. Before the first run the installer does not promise a number. The prediction is an estimate and can differ a
  lot from reality; for MoE models on Vulkan it is not calibrated yet. Trust the measurement on your machine, which the
  installer runs itself.
• If it did not work, see docs/TROUBLESHOOTING.md.
• Other machines. What has been verified is in the table below; how to join the pilot programme:
  CONTRIBUTING-hardware.md.

Run it on your hardware

Shura chooses the model and the settings from what your machine can do, not from its brand: how fast its memory
really is (measured), how much of it there is, what the GPU has, and which backend actually works.

git clone https://github.com/hikkian/shura && cd shura
./scripts/shura check             # what your machine can run and how fast (installs nothing, sends nothing)
./scripts/shura install --dry-run # what would be downloaded, how much disk it needs (downloads nothing)
./scripts/shura install           # download, test and set up; then `shura start` / `shura stop`
./scripts/shura report --issue    # an anonymous report for a GitHub issue: helps everyone with your hardware

On Windows use scripts\shura.cmd (or py installer\universal\cli.py). Windows cannot measure RAM speed without a compiler: add --ram-gbs 40 (dual-channel DDR4-3200; about 80 for DDR5-6000). NVIDIA on Linux gets the fast tier (our
CUDA fork) through setup.sh; shura install points you to it. Every other machine is installed by shura install
itself, without compiling anything and without root.

shura install is built so that a bad day costs you nothing: it asks before it downloads; it checks the free disk space
first; a dropped download resumes; every file is verified (size and SHA-256); it runs the real server with the planned
settings to prove they work; if the machine runs out of memory it plans again with more room left free and retries; and
if nothing works it leaves a report without personal data and keeps the downloads, so the next run does not start over.
shura uninstall removes the Shura home folder (engines, models, logs, state). The setup.sh install also writes files outside it (INSTALLER.md); uninstall leaves those.

[!NOTE]
Where it can run (Vulkan on Linux, Metal on Apple Silicon, CUDA on Windows with NVIDIA), shura install also tries a
third-party llama.cpp fork with a turbo KV cache, an expert cache and MTP
(TheTom/llama-cpp-turboquant, pinned release, archive checked against a
SHA-256 recorded in this repository). It measures it against upstream llama.cpp on your machine and keeps the fastest. Each vendor's own stack (ROCm on AMD, CUDA on NVIDIA, SYCL on Intel, Metal on Apple) stays the main
engine unless another is 10% faster on your machine. shura install --no-turbo never uses the third-party build.

• Classes by resources, not by device type: a GPU with system RAM, unified memory (Apple), or a CPU-only machine.
  CPU-only does not mean small models: a many-channel server can run models no 12 GB card can hold.
• Backends: NVIDIA (CUDA, Vulkan), AMD (ROCm or Vulkan, whichever measures faster), Intel (SYCL, Vulkan, OpenVINO),
  Apple (Metal), or the CPU. The prebuilt llama.cpp builds are used, nothing is compiled on your machine.

What is verified, and what is not

[!WARNING]
Two machines have run Shura for real, both the author's: an RX 7800 XT 16 GB (today, the working build) and an
RTX 4070 SUPER 12 GB (until 2026-10). Everything else below is written, tested in software and believed to work, but
nobody has run it on that hardware yet. You may be the first, and your result is the most useful thing you can send.

What is checked automatically on every commit (CI): unit tests on Linux, macOS and Windows; the real llama.cpp build
is downloaded and run on a 19 MB model (the CPU build on Linux and Windows, the Metal build on macOS, inside a CI virtual machine); the Vulkan build runs on a software driver (experimental). The flags shura install passes to llama-server were checked against the real --help of the pinned release (b11301); no CI step repeats that check.

What no one has checked: a real GPU other than the author's two; loading the real model on other hardware; the speed
predictions (estimates, fitted to one machine, shown as a range, and the plan is tested by really starting the server);
Windows start/stop of the server; a load slower than 15 minutes (the installer gives up waiting).

Help us check it

1. ./scripts/shura check and look at the plan: does it make sense for your machine?
2. ./scripts/shura install --dry-run, then ./scripts/shura install.
3. Open a Hardware report issue whichever way
   it went. ./scripts/shura report --issue prints a pre-filled link; the report has no names, paths or IDs and nothing is
   sent until you press the button on GitHub.
   • It worked: say the speed you really saw (tokens/s) and the context size. The report already has what we predicted;
     the gap between the two is how the predictions get fixed.
   • It did not: attach logs/verify.log from the Shura folder (~/.local/share/shura, ~/Library/Application Support/shura or %LOCALAPPDATA%\shura) and say where it stopped. A failed install is as useful as a successful one.
   • You fixed it yourself: describe what was wrong and send the change.
4. A machine class moves from unverified to verified in the table above when a report with a real measurement arrives.

How the decision is made: docs/HARDWARE.md. To add your hardware, a model or a backend:
CONTRIBUTING-hardware.md.

Tested environment and limits

[!NOTE]
Everything here was developed and tested on one machine at a time. Anything outside this list is supported by
design but untested. Reports and fixes are welcome.

What this is not:

• It is a 35B-A3B-class model at 3–4 bits. Expect a capable local coding assistant, not a frontier model.
• Tested on Linux only. The expert cache, the governor and the compressed KV cache live in the CUDA/HIP backend of the
  fork; Intel and Apple get upstream llama.cpp without them.
• Not validated on other hardware: other distributions (the installer's package steps were checked in
  Ubuntu 24.04, Debian 12, Arch and Fedora 44 containers only; a full fresh install has not been run there), XFCE and
  tiling WMs, terminals other than Ghostty (dry-run only), the kdialog/yad dialogs, and GPUs other than the author's two.

AMD is now the author's main machine (ROCm/HIP), but only one card has been run. Other AMD cards should work with the
same build; please send a hardware report. Intel is not supported out of the box. You can port it yourself or with an AI
coding agent (ShuraCode can do this work too). Upstream llama.cpp already runs on Intel (SYCL, Vulkan). Two features
of the fork are missing there:

1. Works today, slower: build upstream llama.cpp for your backend and use the same layout ideas: the
   first layers' experts in RAM (--n-cpu-moe), q8_0/q4_0 KV cache and MTP. The gateway, ShuraCode and
   the rest of this repository do not depend on the fork.
2. Full speed: port the fork's expert-cache and compressed-KV kernels to SYCL or Vulkan. Then
   re-run the benchmarks in bench/ and the correctness probes, because silent output corruption is the
   failure mode to watch for (see docs/TURBOQUANT-FIX.md).

Pull requests with measured results on other cards are very welcome.

Repository layout

setup.sh     one-command installer (distro packages, CUDA, build, model, auto-tune)
installer/   hardware planner, GGUF header reader, model download, auto-tune
tests/       installer and gateway unit tests + distro package test (containers)
gateway/     ai_gateway.py - on-demand llama-server manager + OpenAI-compatible proxy (stdlib only)
config/      example configs + agentic MoE routing profile for Tiel-Coder
patches/     patches for the llama.cpp fork (TurboQuant head_dim-256 fix, persistent slot checkpoints, ...)
scripts/     shura (daily CLI), build-llama.sh, install.sh, capture-moe-trace.sh
desktop/     app-menu launcher template
assets/      logo (light/dark), app icons, social preview, source artwork
bench/       agentic-eval task set and harness, deep-context benchmark, depth sweep, correctness probes
systemd/     user service unit
searxng/     private SearXNG settings template
docs/        benchmarks, architecture, harness comparison, TurboQuant fix, install, troubleshooting

Documentation

Acknowledgements

• ggml-org/llama.cpp, the foundation of everything here, and the
  thecodacus/llama.cpp perf fork (MoE expert cache, TurboQuant), from which
  our fork descends
• wilky2005 for the TurboQuant fix (thecodacus/llama.cpp#12)
• The authors of the upstream fix for persistent context checkpoints
  (llama.cpp #25913,
  PR #26004), which our implementation builds on
• Tiel-Coder-35B-A3B-MTP by peculiar-ragdoll
• OpenCode, SearXNG,
  Playwright MCP

License

MIT for everything in this repository. The patches in patches/ are derived from llama.cpp (MIT).
Model weights are not included and are covered by their own license.