cluaiz

A local AI engine in Rust: chat, vision, speech and embeddings on your own hardware, with the placement on GPU, RAM and CPU decided for you

Documentation  |  Website  |  Discord  |  Reddit  |  LinkedIn

[!WARNING]
Pre-release. v0.1.0 is the first public build. Expect breaking changes in settings, APIs and file layout
until the stable release.

About Cluaiz

Cluaiz is one engine process that runs local models and serves them to apps, scripts and agents:

• GGUF models through a fork of llama.cpp linked into the engine:
  chat, vision, audio input, embeddings.
• ONNX models through ONNX Runtime: text-to-speech (Kokoro and others), speech-to-text (Whisper), embeddings
  and vision encoders.
• A placement planner that measures the free GPU memory and RAM on every load and chooses where the model
  runs, how large the context is and which KV cache type fits. You can override any of it.
• MoE models larger than the GPU, and larger than free RAM: attention stays on the GPU, the experts sit in
  RAM, and whatever does not fit RAM is read from the SSD when it is used.
• APIs that existing clients already speak (OpenAI-compatible and Ollama-style routes), plus CEP, one
  JSON-RPC 2.0 protocol for everything engine-specific, over HTTP, MCP or a C library.
• Tools for agents: skills, plugins and MCP servers, with permission levels, approvals and an audit log.

On the same model file, placement and settings, Cluaiz generates as fast as llama.cpp's own llama-bench, and
reads prompts up to 2x faster when a model is split across GPU and RAM (see Benchmarks).

[!TIP]
A 26B model on a 4 GB laptop GPU. Gemma 4 26B A4B (MoE, 13.3 GB file) on an RTX 3050 Laptop with 4 GB
VRAM: 16–18 tokens/s with about 8 GB of free RAM, and still 7.4 tokens/s with only about 3 GB free,
when more of the experts are read from the SSD. The planner picks this placement by itself (n_cpu_moe); no flags
needed. Dense models that do not fit RAM + VRAM are refused with the reason instead of crawling from the disk.
Details.

Quick Start

1. Install (downloads the release for your system and the GPU pack for your GPU, and checks every file
against the release's checksums):

# Windows (PowerShell)
irm https://cluaiz.com/install.ps1 | iex

# Linux and macOS
curl -fsSL https://cluaiz.com/install.sh | sh

2. Chat with a model (an installed model, a registry name or a Hugging Face repository; it is downloaded on
first use):

cluaiz run unsloth/Qwen3-4B-GGUF
cluaiz run unsloth/Qwen3-4B-GGUF "Explain a KV cache in two sentences."

3. See what the planner chooses on this machine before loading:

cluaiz show unsloth/Qwen3-4B-GGUF

4. Serve it to other apps and call the OpenAI-compatible API:

cluaiz serve

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $(cat ~/.cluaiz/session.token)" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-4b-gguf-Q4_K_M", "messages": [{"role": "user", "content": "Hello"}]}'

cluaiz list shows the installed model IDs; cluaiz token show shows the API token. The port and host come
from permission.json (cluaiz permission port <n>).

Features

Models and formats

Families and the models tested with each are listed in RELEASE.md.

Documents and ingest

POST /v1/ingest and cluaiz ingest <file> turn documents and images into text, chunks or vectors. A model is
optional: without one the engine extracts the text itself.

Not included yet: XLSX / PPTX parsing and a persistent vector index. Details:
docs/api/inference/ingest.md.

Placement and memory

• Candidates on every load: full GPU, GPU + RAM split, MoE experts in RAM (or read from the SSD when
  they do not fit RAM), CPU only. The plan picks the context size and the KV cache type (F16, then Q8_0) and
  keeps a VRAM buffer free.
• The plan is checked against llama.cpp's own dry run, and the VRAM a load really takes is remembered per model,
  so the next plan is closer.
• A model that fits nowhere is refused with the amount missing, instead of thrashing the disk.
• cluaiz show <model> prints every candidate with its memory numbers; the server log prints the same plan on
  each load.
• Every value can be set by hand with llama.cpp's flag names (-ngl, -c, -ctk, -fa, -ncmoe, ...) or in
  a request's llama object; a user setting is passed to llama.cpp as it is.

Speed

• Prompt cache reuse: the shared beginning of a chat stays in the KV cache, and shared chunks after it are
  moved instead of recomputed (llama-server's --cache-reuse).
• Prompt chunks shaped for split models: a short last chunk is evened out with the one before it, so layers in
  RAM do not compute it on the CPU.
• Chat templates parsed once per model: about 0.1 s to the first token on a warm prompt.
• The CPU backend picks the best build for the processor at start (SSE4.2 to AVX-512, NEON on ARM).

APIs

Reference: docs/api/README.md and docs/api/cep/README.md.

Tools, agents and security

• Skills, plugins and MCP servers installed from the cluaiz-tools
  registry (cluaiz skill install, cluaiz plugin install, cluaiz mcp install). Plugins run as separate
  processes.
• Chat memory: earlier messages that no longer fit the prompt are recalled when the new message needs them
  (details).
• Execution policy: full_access or sandboxed (default) per engine and per tool; actions outside the
  workspace ask for approval; destructive commands (disk wipes, rm -rf /) and credential paths are blocked.
• API tokens for every route (cluaiz token), and an audit log of every write, exec and admin call
  (parameters kept only as a hash).

Command line

Every command and flag: cluaiz --help and cluaiz <command> --help.

Supported Hardware

What release v0.1.0 ships (core + GPU packs, chosen by the installer) and where it has run. Details:
docs/deployments/platforms.md.

ROCm/HIP, OpenVINO, SYCL and CANN builds are not part of the release yet.

Benchmarks

Every number comes from test/engine_regression/run.py: the same model file,
placement and settings on Cluaiz (through its chat API) and on llama.cpp's llama-bench built from the same
llama.cpp commit. Higher is better.

Laptop: RTX 3050 4 GB, Windows 11 (2026-10-12)

Empirical regression test data measured on AMD Ryzen 7 7435HS (8C/16T), 24 GB RAM, NVIDIA GeForce RTX 3050 A Laptop (4 GB VRAM), Windows 11, CUDA 13.2:

Generation speed (Tokens / Second)

Prompt processing speed (~530 tokens prompt)

[!NOTE]
Why Split Prompt is 1.4x–2.2x Faster: Upstream llama.cpp chunks prompts into fixed 512-token batches (leaving trailing tokens under 32 to run on CPU). Cluaiz balances chunk sizes evenly (e.g. 265 + 265), keeping all prompt chunks on the accelerated path.

Time To First Token (TTFT): 0.09s - 0.10s (warm prompt with persistent template caching, down from 0.31s).

Complete Reproducible Benchmark: Full test tables, methodology, and reproduction scripts (python test/engine_regression/run.py) are documented in test/engine_regression/RESULTS.md.

Cloud: Tesla T4 16 GB, Ubuntu 24.04 (2026-10-28)

Same test (test/engine_regression/run.py) on Google Cloud n1-standard-8: Intel Xeon 2.3 GHz (4C/8T), 29 GB RAM,
NVIDIA Tesla T4 (16 GB VRAM), Ubuntu 24.04, driver 580. Cluaiz is release v0.1.0 installed with install.sh
(CUDA 12 pack); llama.cpp is llama-bench built on the same machine from the same llama.cpp commit and given the
placement Cluaiz chose.

Generation speed (Tokens / Second)

Prompt processing speed (~535 tokens prompt)

Measured with the prompt fixes on main (commits 6120d017, 4228752a; not in release v0.1.0). Before them the
T4 showed llama.cpp 7–22% ahead in the cases marked below.

CPU only, llama.cpp also without a GPU (-dev none): Qwen3 4B 27 vs 29, Gemma 3 12B 10 vs 10, Mistral 24B 5 vs 5,
Gemma 4 31B 4 vs 4 t/s.

Less RAM, and a model bigger than RAM + VRAM (same VM with 15 GB RAM)

[!NOTE]
Reading the T4 numbers: generation equals llama.cpp in every test (-3% to +8%; the T4's clock moves a few
percent between runs). Prompt processing equals llama.cpp on full GPU after the two fixes and stays up to 1.6x
faster when a model is split across GPU and RAM. A dense model larger than RAM + VRAM is refused by the planner
instead of thrashing the disk; streaming from the SSD is used for MoE experts (Gemma 4 26B above).

Full Report: methodology, every setting, the exact llama-bench commands and all logs are in
test/cloud/runs/2026-10-28-run5-linux-t4-regression/RESULTS.md.

Architecture

Technical Specification

• Purpose: One local engine process that serves chat, embeddings, speech and transcription through
  OpenAI-compatible routes and its own JSON-RPC protocol (CEP), and decides by itself where each model runs
  (GPU, GPU + RAM, CPU) from live hardware measurements.
• Backends: llama.cpp (cluaiz fork, linked into the engine) for GGUF models; ONNX Runtime for speech and
  vision models.
• Platforms: Windows x64 (MSVC), Linux x64 / ARM64 (glibc 2.38+), macOS Apple Silicon.
• Delivery: a CPU core plus optional GPU packs (CUDA 12, CUDA 13, Vulkan); the engine loads the GPU backend
  libraries it finds next to its executable at start.

Architectural Flow

flowchart TB
    subgraph CLIENTS["Clients"]
        CLI["cmd/<br/>cluaiz command line"]
        APP["Cluaiz App<br/>(desktop GUI)"]
        HTTP["Any OpenAI-compatible client"]
        MCPC["MCP clients"]
        EMB["Embedding apps<br/>(C ABI)"]
    end

    subgraph GATEWAY["Gateway: api/ (Axum HTTP + SSE)"]
        AUTH["Session token auth<br/>host and port from settings"]
        ROUTES["/v1/chat/completions · /v1/embeddings<br/>/v1/audio/speech · /v1/audio/transcriptions<br/>/v1/models · /v1/ingest · /health"]
        CEPR["/v1/cep and /mcp"]
    end

    FFI["ffi/<br/>cep_call · cep_free"]
    CEP["cep/: Cluaiz Engine Protocol<br/>JSON-RPC 2.0 envelope, method registry,<br/>principals, approvals, tasks, discovery"]

    subgraph SERVICES["Request services"]
        SEC["security/<br/>permissions, filesystem jail, audit"]
        CTX["context/<br/>pure context planning: fit, compaction,<br/>memory cards, episodes, recall"]
        AGT["agents/<br/>tools, skills, plugins, MCP client,<br/>tool search, sandboxed execution"]
        MEDIA["engine/media<br/>bounded sources, image / audio / video<br/>decoding, resampling"]
    end

    subgraph RUNTIME["engine/runtime: single engine host"]
        HOST["Model host<br/>load / unload, keep_alive,<br/>one model transition at a time"]
        CHATMEM["Chat memory and KV cache reuse"]
        SPEECH["Speech, transcription and<br/>embedding hosts"]
        VRAMUSE["Measured VRAM use<br/>(learned margins)"]
    end

    subgraph PLANNING["Placement planning"]
        PLAN["engine/plan<br/>candidates: full GPU · GPU + RAM split ·<br/>MoE experts in RAM / SSD · CPU only<br/>context size, KV type, VRAM buffer"]
        HW["engine/hardware<br/>CPU, RAM, GPU (NVML), disk<br/>free memory measured live"]
        MODELS["engine/models<br/>GGUF reader, model registry,<br/>Hugging Face fetcher, capabilities"]
        CORE["engine/core<br/>settings snapshot, paths, units, contracts"]
    end

    subgraph EXEC["Execution"]
        LLAMA["engine/llama<br/>FFI to the llama.cpp fork: chat templates, tokenizer,<br/>prompt chunks, sliding-window checkpoints,<br/>sampling, tool-call parsing, token streaming"]
        ONNX["engine/onnx<br/>ONNX Runtime: Kokoro and other TTS,<br/>Whisper STT, vision encoders"]
    end

    subgraph BACKENDS["Backends loaded at start (core + GPU packs)"]
        GCPU["ggml CPU<br/>(best build for the processor)"]
        GCUDA["ggml CUDA 12 / 13<br/>+ cudart, cuBLAS"]
        GVK["ggml Vulkan"]
        GMTL["ggml Metal<br/>(inside the macOS core)"]
        ORTP["ONNX providers<br/>CUDA (+ cuDNN fetched) · DirectML · CPU"]
    end

    subgraph HARDWARE["Hardware"]
        NV["NVIDIA GPU"]
        AMDI["AMD / Intel GPU"]
        APL["Apple Silicon"]
        CPUH["CPU and RAM"]
        SSD["SSD<br/>(memory-mapped weights)"]
    end

    STORE[("CLUAIZ_HOME<br/>models, settings, session token,<br/>hardware profile, VRAM margins")]

    CLI --> ROUTES
    APP --> ROUTES
    APP --> CEPR
    HTTP --> ROUTES
    MCPC --> CEPR
    EMB --> FFI
    AUTH --- ROUTES
    AUTH --- CEPR
    CEPR --> CEP
    FFI --> CEP
    ROUTES --> SERVICES
    CEP --> SERVICES
    ROUTES --> HOST
    CEP --> HOST
    SERVICES --> HOST
    HOST --> PLAN
    PLAN --> HW
    PLAN --> MODELS
    PLAN --> CORE
    VRAMUSE --> PLAN
    HOST --> CHATMEM
    HOST --> SPEECH
    HOST --> LLAMA
    SPEECH --> ONNX
    LLAMA --> GCPU
    LLAMA --> GCUDA
    LLAMA --> GVK
    LLAMA --> GMTL
    ONNX --> ORTP
    GCUDA --> NV
    ORTP --> NV
    GVK --> AMDI
    GMTL --> APL
    GCPU --> CPUH
    LLAMA -. mmap .-> SSD
    MODELS --- STORE
    HW --- STORE
    VRAMUSE --- STORE

One Chat Request, Step by Step

sequenceDiagram
    autonumber
    participant C as Client
    participant A as api/
    participant X as context/ + agents/
    participant H as engine/runtime
    participant P as engine/plan
    participant L as engine/llama
    participant G as llama.cpp (ggml backend)

    C->>A: POST /v1/chat/completions (stream)
    A->>A: check the session token, validate the request
    A->>X: fit the chat into the context, attach tools and memory
    A->>H: generate(model, messages, settings)
    alt model not loaded, or load settings changed
        H->>P: plan(model geometry, live free RAM and VRAM, settings)
        P-->>H: placement (n_gpu_layers, n_cpu_moe), n_ctx, KV type, VRAM buffer
        H->>L: load the model and context with that plan
        L->>G: llama_model_load, llama_init_from_model, warm-up decode
        H->>H: compare the measured VRAM with the plan, learn the margin
    end
    H->>L: render the chat template, tokenize, reuse the cached prefix
    L->>G: prefill in chunks of 512 (a small tail evened out when weights sit in RAM)
    loop each token
        L->>G: decode one token, sample
        L-->>A: token (answer or reasoning, tool calls parsed)
        A-->>C: SSE chunk
    end
    L-->>A: timings (prompt and generation tokens per second)
    A-->>C: final chunk with usage and engine timings

Workspace Crates

Failure & Recovery Logic

Documentation

Ecosystem

Troubleshooting

Windows SmartScreen

The release executables are not signed with a commercial certificate yet, so Windows may show "Windows
protected your PC". Choose More info, then Run anyway.

License

Cluaiz is released under the Apache License 2.0. See LICENSE.