mlx-serve

ddalcu.github.io/mlx-serve

Native Zig server that runs MLX-format language models on Apple Silicon and exposes OpenAI-compatible and Anthropic-compatible HTTP APIs. No Python. Comes with MLX Core, a macOS menu bar app with built-in chat, agent mode, and model management.

Download MLX Core.app — latest release for macOS (Apple Silicon)

Install via Homebrew

brew tap ddalcu/mlx-serve https://github.com/ddalcu/mlx-serve
brew install --cask mlx-core   # GUI menu bar app
brew install mlx-serve          # CLI server only

Features

• OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/models)
• Anthropic-compatible API (/v1/messages) — works with Claude Code
• Streaming and non-streaming responses
• Tool calling (function calling) with automatic detection
• KV cache reuse across requests for fast multi-turn conversations
• Sampling: temperature, top-k, top-p, repeat penalty, presence penalty
• Reasoning/thinking mode support
• Chat templates via Jinja2 (Jinja_cpp) with fallback formatting
• TUI status bar with CPU, memory, and GPU metrics

MLX Core (macOS App)

Menu bar app that wraps the server with a full UI:

• Model browser -- download models from HuggingFace with resumable downloads
• Chat interface -- multi-session chat with markdown rendering
• Agent mode -- 8 built-in tools (shell, readFile, writeFile, editFile, searchFiles, browse, webSearch, saveMemory) with automatic tool calling loop
• Editable system prompt -- customize agent behavior via ~/.mlx-serve/system-prompt.md (Agent menu → Edit System Prompt)
• Persistent memory -- agent can save memories across sessions to ~/.mlx-serve/memory.md
• Prompt-based skills -- drop .md files in ~/.mlx-serve/skills/ to teach the agent custom capabilities
• Server management -- start/stop server, view logs, configure max tokens

Supported Models

Any quantized MLX model using one of the above architectures should work.

Prerequisites

• macOS with Apple Silicon (M1/M2/M3/M4)
• Zig 0.15+
• mlx-c:

brew install mlx-c

Quick Start

Download a model

The MLX Core app can download models directly, or use the CLI:

pip install huggingface-hub
huggingface-cli download mlx-community/gemma-4-e4b-it-4bit --local-dir ~/.mlx-serve/models/gemma-4-e4b-it-4bit

Build and run

zig build -Doptimize=ReleaseFast
./zig-out/bin/mlx-serve --model ~/.mlx-serve/models/gemma-4-e4b-it-4bit --serve --port 8080

Build the app

cd app && SKIP_NOTARIZE=1 bash build.sh
open "MLX Core.app"

Requires APPLE_DEVELOPER_ID and APPLE_TEAM_ID environment variables for code signing.

Usage

Interactive mode

./zig-out/bin/mlx-serve --model /path/to/model --prompt "What is 2+2?"

HTTP server

./zig-out/bin/mlx-serve --model /path/to/model --serve --port 8080

CLI options

API

POST /v1/chat/completions

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Write a haiku about programming."}],
    "max_tokens": 256,
    "stream": true
  }'

Supports messages, max_tokens, temperature, top_p, top_k, stream, tools, repetition_penalty, presence_penalty, and logprobs.

POST /v1/messages (Anthropic)

curl http://localhost:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "mlx-serve",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Write a haiku about programming."}]
  }'

Compatible with Claude Code (ANTHROPIC_BASE_URL=http://localhost:8080 claude) and Anthropic SDKs. Supports streaming, tool calling, and extended thinking.

Other endpoints

• GET /health -- health check
• GET /v1/models -- list loaded models
• POST /v1/completions -- text completions
• POST /v1/messages -- Anthropic Messages API

Performance

Benchmarked on Apple M4 (16 GB unified memory):

Matches mlx-lm (Python) generation speed while using less memory and starting 3x faster. Key optimizations: fully-lazy async pipeline with reordered eval (submit-first pattern), JIT-compiled activations (GELU, GeGLU, softcap via mlx_compile), and GPU memory wiring.

# Prefill (~840 token prompt):
./zig-out/bin/mlx-serve --model ~/.mlx-serve/models/gemma-4-e4b-it-4bit \
  --prompt "$(python3 -c "print('Explain the following topics in extreme detail: ' + ', '.join([f'topic {i} about science and technology and its impact on human civilization throughout history' for i in range(1,50)]))")" \
  --max-tokens 1

# Decode (256 tokens, temp=0):
./zig-out/bin/mlx-serve --model ~/.mlx-serve/models/gemma-4-e4b-it-4bit \
  --prompt "Write a detailed essay about quantum computing" \
  --max-tokens 256

Run 3 times and take the average of runs 2-3 (run 1 includes model loading from disk).

License

MIT