Motivation

Commercial AI APIs process every request on infrastructure the customer neither owns nor can inspect. Training-data extraction, prompt logging, and retroactive policy changes are documented incidents. The European regulatory framework --- in particular GDPR Articles 25 and 32 --- mandates data protection by design, a requirement difficult to discharge with an opaque black box in a foreign jurisdiction.

MoE Sovereign is a fully self-hosted multi-model orchestrator with template-based expert routing that runs entirely on your own hardware. No data leaves your network. No cloud dependency. No vendor lock-in.

Architecture

flowchart TD
    subgraph Clients["Client Layer"]
        CC["Claude Code"]
        OW["Open WebUI"]
        API["Any OpenAI Client"]
    end

    subgraph Orchestrator["MoE Orchestrator (LangGraph)"]
        Cache{"L0/L1 Cache<br/>Valkey + ChromaDB"}
        Planner["Planner<br/><i>phi4:14b</i>"]
        
        subgraph Experts["Parallel Expert LLMs"]
            E1["Code Reviewer"]
            E2["Researcher"]
            E3["Domain Expert"]
        end

        MCP["28 MCP Tools<br/><i>AST-Whitelist + PPTX</i>"]
        GraphRAG["Neo4j GraphRAG"]
        Judge["Judge / Merger<br/><i>llama3.3:70b</i>"]
        GapCheck{"Gap Detector<br/><i>COMPLETE?</i>"}
        Replan["Agentic Re-Plan<br/><i>up to 3 rounds</i>"]
    end

    subgraph Storage["Persistence Layer"]
        Neo4j[("Neo4j<br/>Knowledge Graph")]
        Chroma[("ChromaDB<br/>Vector Cache")]
        Kafka["Kafka<br/>Event Stream"]
        Valkey[("Valkey<br/>State & Sessions")]
        PG[("PostgreSQL<br/>Users & Checkpoints")]
    end

    Clients -->|"/v1/chat/completions<br/>/v1/messages<br/>/v1/responses"| Cache
    Cache -->|Miss| Planner
    Cache -->|Hit| Response
    Planner --> E1
    Planner --> E2
    Planner --> E3
    E1 --> MCP
    E2 --> MCP
    E3 --> MCP
    MCP --> GraphRAG
    GraphRAG --> Judge
    Judge --> GapCheck
    GapCheck -->|"COMPLETE"| Response["Response"]
    GapCheck -->|"NEEDS_MORE_INFO"| Replan
    Replan -->|"inject gap context"| Planner
    Response -->|Ingest| Kafka
    Kafka --> Neo4j
    Response -->|Cache Write| Chroma
    Judge -.->|"Retry on<br/>low score"| Planner

    style Orchestrator fill:#f0f4ff,stroke:#4a6fa5
    style Experts fill:#e8f5e9,stroke:#388e3c
    style Storage fill:#fff8e1,stroke:#f9a825

Pipeline Stages

Module Structure

The orchestrator codebase is organised into focused packages. main.py is a thin entry point (~1 500 LOC) holding the FastAPI app, lifespan, middleware, and graph wiring. All domain logic lives in dedicated packages:

moe-infra/
├── main.py                    # FastAPI app, lifespan, middleware, graph wiring (~1 500 LOC)
├── config.py                  # All os.getenv() — typed config constants
├── state.py                   # Shared mutable globals (redis_client, _userdb_pool, …)
├── prompts.py                 # Static prompt text + routing detection regexes
├── metrics.py                 # Single Prometheus registry
├── parsing.py                 # Stateless parsers: JSON extraction, confidence, history truncation
├── context_budget.py          # Per-model context-window estimation
│
├── routes/                    # FastAPI APIRouters (one per concern)
│   ├── health.py              # /health, /metrics
│   ├── watchdog.py            # /api/watchdog/*, Starfleet feature toggles
│   ├── mission_context.py     # /api/mission-context
│   ├── graph.py               # /graph/*
│   ├── feedback.py            # /v1/feedback, /v1/memory/ingest
│   ├── admin_*.py             # Benchmark, ontology, stats admin endpoints
│   ├── models.py              # /v1/models
│   ├── ollama_compat.py       # /api/* (Ollama protocol)
│   └── anthropic_compat.py    # /v1/messages, /v1/responses, /v1/chat/completions
│
├── services/                  # Business logic — no FastAPI imports
│   ├── auth.py                # OIDC + API key validation + budget enforcement
│   ├── tracking.py            # Usage logging, request lifecycle, budget counters
│   ├── routing.py             # Expert template + per-template prompt resolution
│   ├── templates.py           # Expert template + Claude Code profile loading
│   ├── llm_instances.py       # ChatOpenAI singletons (judge, planner, ingest, search)
│   ├── inference.py           # Node selection, fallback chain, Thompson sampling
│   ├── helpers.py             # Progress reports, semantic memory, self-evaluation
│   ├── skills.py              # Server-side skill resolution + ADMIN_APPROVED hard-lock
│   ├── healer.py              # Ontology gap-healer (one-shot + dedicated subprocess)
│   ├── kafka.py               # Fire-and-forget Kafka publish helper
│   └── pipeline/              # OpenAI / Anthropic / Ollama / Responses API handlers
│       ├── chat.py            # OpenAI chat completions
│       ├── anthropic.py       # Anthropic Messages API + tool/MoE/reasoning handlers
│       ├── ollama.py          # Ollama-protocol streaming wrappers
│       └── responses.py       # OpenAI Responses API
│
├── graph/                     # LangGraph node implementations
│   ├── router_nodes.py        # cache_lookup, semantic_router, fuzzy_router, _route_cache
│   ├── tool_nodes.py          # mcp_node, graph_rag_node, math_node_wrapper
│   ├── planner.py             # planner_node + plan sanitization + topological levels
│   ├── expert.py              # expert_worker (parallel expert execution)
│   ├── research.py            # research_node + research_fallback + domain extraction
│   └── synthesis.py           # merger_node, thinking_node, resolve_conflicts_node, critic_node
│
├── pipeline/
│   ├── __init__.py            # LangGraph graph builder — assembles nodes into the pipeline DAG
│   └── state.py               # AgentState TypedDict (67 fields across 3 categories)
│
├── web_search.py              # SearXNG integration with domain-reliability scoring
├── math_node.py               # SymPy-backed math node (solve, integrate, differentiate)
├── graph_rag/                 # GraphRAG query, entity linking, ontology, corrections
├── federation/                # Push / pull federation client to MoE Libris hubs
├── mcp_server/                # 28 MCP precision tools (AST-whitelisted)
├── admin_ui/                  # Admin backend: experts, users, budgets, cleanup manager
├── prompts/systemprompt/      # 15 expert system prompts (English, "Respond in German.")
├── tests/                     # 195 unit + integration + smoke tests (all green)
└── benchmarks/                # Overnight benchmark suite, GAIA runner, result injection

The orchestrator started as an 11 190-line monolith in main.py. A 14-phase split (Q2 2026) decomposed it into the structure above without a single behavioural change — every phase ended with the full test suite green. See docs/ARCHITECTURE.md for the detailed module map.

Key Capabilities

A) Core AI & Orchestration

B) Security, Sovereignty & Admin

C) Enterprise Data Management (moe-codex Extension)

This feature group requires the optional moe-codex
enterprise stack (Apache NiFi, Marquez/OpenLineage, lakeFS). It is not part of the
moe-sovereign core and is deployed as a separate compose stack.
See the moe-codex repository for setup instructions.

Federated Knowledge Ecosystem

MoE Sovereign goes beyond a local RAG system. With community knowledge bundles, deployments exchange domain knowledge (law, Kubernetes, React, medicine) without sharing proprietary data or source code.

flowchart LR
    subgraph Instance_A["Deployment A<br/><i>Banking</i>"]
        GA[("Neo4j<br/>3 150 entities")]
    end
    subgraph Instance_B["Deployment B<br/><i>Healthcare</i>"]
        GB[("Neo4j<br/>2 800 entities")]
    end
    subgraph Instance_C["Deployment C<br/><i>DevOps</i>"]
        GC[("Neo4j<br/>1 200 entities")]
    end

    GA -- "Export Bundle<br/>(privacy-scrubbed)" --> Bundle["JSON-LD<br/>Knowledge Bundle"]
    GB -- "Export Bundle" --> Bundle
    Bundle -- "Import<br/>(trust-capped)" --> GC
    Bundle -- "Import" --> GA

    style Bundle fill:#e3f2fd,stroke:#1565c0,stroke-width:2px

Privacy protection: Metadata stripping • Regex detection of PII/secrets • Sensitive relation-type filter • ⚠ Human-in-the-loop responsible for contextual/structural PII (see Privacy Scrubber limitations)

Import safety: Entity MERGE (no duplicates) • Trust ceiling (0.5) • Contradiction detection via moe.linting

Every new installation enriches the collective knowledge graph. Every bundle import accelerates all participants. This is the network effect for open-source AI.

Benchmarks

⚠ GAIA evaluation integrity note: One evaluation run inadvertently used non-sovereign (cloud) inference for the Planner/Judge nodes. The results above reflect that run. A controlled sovereign GAIA run is pending. See whitepaper Section 14 for full disclosure.

Quick Start

One-Line Install

Debian / Ubuntu:

curl -sSL https://raw.githubusercontent.com/moe-sovereign/moe-sovereign/main/install.sh | bash

macOS (Docker Desktop or rootless Podman; run as a regular user, never with sudo):

curl -fsSL https://raw.githubusercontent.com/moe-sovereign/moe-sovereign/main/install-macos.sh | bash

Manual Setup

git clone https://github.com/moe-sovereign/moe-sovereign.git
cd moe-sovereign
cp .env.example .env
nano .env                      # Set credentials and inference server URLs
sudo docker compose up -d
curl http://localhost:8002/v1/models

Deployment Targets

flowchart LR
    OCI["One OCI Image<br/><i>multi-stage, non-root</i>"]

    OCI --> Solo["<b>Solo</b><br/>LXC / single VM<br/>~1.5 GiB RAM"]
    OCI --> Team["<b>Team</b><br/>Docker Compose<br/>~6 GiB RAM"]
    OCI --> Ent["<b>Enterprise</b><br/>Helm / K8s<br/>HA, HPA, PDB"]

    Solo --> LXC["LXC / Proxmox"]
    Team --> DC["Docker Compose"]
    Team --> Pod["Podman (rootless) ✓"]
    Ent --> K3s["K3s / Kubernetes"]
    Ent --> OCP["OpenShift"]

    style OCI fill:#e8eaf6,stroke:#3f51b5,stroke-width:2px

All targets use the same OCI image --- no code forks, no feature loss.

Services

Agent Integration

Ecosystem & Integrations

• openDesk Edu —
  Educational digital workplace for universities, with planned
  integration of MoE Sovereign as a self-hosted AI orchestration layer.

Competitive Landscape

Note on Palantir comparison: The table above compares technical feature presence, not product maturity or enterprise support depth. Palantir AIP is a commercially mature platform with thousands of engineers, extensive certifications, and a global support organisation. MoE Sovereign is an open-source project addressing the same architectural problem space — with full data sovereignty, zero licence cost, and complete code auditability as its differentiating properties. See Palantir Comparison for a detailed assessment.

Hardware Requirements

The orchestrator runs on CPU. GPU VRAM is only needed on inference nodes (Ollama).
The moe-codex Enterprise Data Stack (NiFi 4 GB, lakeFS 512 MB, Marquez 1.5 GB + 2× Postgres)
adds significant overhead — plan a dedicated host or at least 8 GB additional RAM.

Documentation

Full documentation: docs.moe-sovereign.org

Local preview: pip install mkdocs-material && mkdocs serve

Publications

Funded Research: EuroHPC LUMI-G

In June 2026, proposal EHPC-DEV-2026D06-XXX was awarded under the EuroHPC
Development Access programme (award notice received 2026-06-05): 4,500
node-hours (18,000 GPU-hours) on the LUMI-G supercomputer (AMD MI250X,
128GB HBM2e/node, ROCm stack, 2TB storage), 6-month duration.

The grant funds a distillation programme covering the five highest-impact
LLM-dependent decision points in the orchestration graph, replacing
cloud-routed inference with locally executable Small Language Models (SLMs)
where decision quality can be preserved:

The 18,000 GPU-hours are budgeted across synthetic data generation, encoder/
reward-model training, planner SFT+DPO with ablations, offline RL, and a
final RLHF pass over six months. Rollout is gated behind a PLANNER_MODE
flag (llm / slm_local / hybrid) with confidence-based fallback to the
cloud teacher — the same fail-open principle already used for expert routing.
Details: CHANGELOG.md (v2.7.1) and the
arXiv paper, Section
"Funded Research: EuroHPC LUMI-G SLM Distillation Programme".

Training Results (August 2026): The complete empirical campaign — including
Judge training (deployed 2026-07-19), Planner v4 six-job SLURM chain, and the
teacher-student distillation pipeline — is documented in the
LUMI-G Training Paper
(PDF, 23 pages).
Key finding: reducing max_seq_len 8192→4096 yields only 1.2× speedup with
--no-packing (not the expected 4×); packing must be enabled to realise the
quadratic attention savings.

Research Basis

The formal logic state layer (capabilities #25–28) is grounded in peer-reviewed
mathematical logic research. We gratefully acknowledge the foundational contributions
of Prof. A. de Vries, whose unified algebraic hierarchy of logics provides the
theoretical framework for paraconsistent, intuitionistic, and fuzzy logic within
this system:

A. de Vries, "Algebraic hierarchy of logics unifying fuzzy logic and quantum logic",
arXiv:0707.2161 [math.LO], 2007. https://arxiv.org/abs/0707.2161

Additional classical results used: Gödel t-norm (1932), Łukasiewicz t-norm (1920),
Kolmogorov algorithmic information content (1965), Chaitin complexity bound (1966),
Ratcliff/Obershelp string similarity (1988). Full attribution in
docs/ARCHITECTURE.md.

Contributing

See CONTRIBUTING.md for the extension model (MCP tools, expert templates, Admin UI).

Please read CODE_OF_CONDUCT.md before opening issues or pull requests.

Disclaimer

MoE Sovereign is a research and productivity tool. AI-generated output may be inaccurate or misleading.
Medical and legal expert outputs do not constitute professional advice.
All AI output should be verified independently.
See PRIVACY.md for data handling details.

License

Apache License 2.0

See THIRD_PARTY_NOTICES.md for bundled component licenses.