RLE — RimWorld Learning Environment

A harness × model benchmark: swappable agent harnesses manage a RimWorld colony under uncertainty and are scored on the same footing against an unmanaged baseline. Think FLE (Factorio Learning Environment) but stochastic, multi-agent-capable, and with the harness — not just the model — as a first-class variable.

What makes this different

• Harnesses are swappable like models — --harness felix (the original 7-agent Felix SDK stack), --harness baseline (unmanaged colony), or any harness package installed from PyPI/GitHub (rle-harness-<tool>); the core never imports an agent framework
• Harness-agnostic scoring — process metrics read the writes that reached the game, not any harness's internal messaging; scenarios, saves and the composite are identical for every harness
• Spatial awareness — deterministic terrain analysis from the game map gives every harness verified build/farm/stockpile coordinates (MAP_SUMMARY)
• Stochastic environment — raids, plague, mental breaks, weather. Harnesses adapt, not just optimize
• Paired against a real baseline — every run is compared to RimWorld's own pawn AI on the same save
• Provider-agnostic — runs on a free local 4B model or a cloud 30B, same environment

Architecture

RimWorld (game)
    ↕ Harmony patches
RIMAPI mod (C# REST :8765 + SSE events)
    ↕ httpx async + SSE
RLEGameLoop (environment: pause → state → harness.step → execute → score → unpause)
    ↕ Harness protocol (rle.harness) — discovered via the `rle.harnesses` entry-point group
    ├─ felix     MapAnalyst → 6 role agents over CentralPost, merged by ActionResolver   [in tree, extra `felix`]
    ├─ baseline  unmanaged colony                                                        [in tree]
    └─ <tool>    external coding agents attached over the RimAPI MCP server (rle-mcp)    [own repos]
    ↕ OpenAI-compatible / Anthropic / local API (provider + model are strings; the harness interprets them)
LLM

python scripts/run_benchmark.py --harness list          # what is installed
python scripts/run_benchmark.py --harness felix --harness baseline --smoke-test

Writing a harness: see docs/harness-plugins.md. Third-party harnesses (OpenCode, Grok Build, ...) live in their own AppSprout-dev/rle-harness-* repos and are installed with pip, never committed here.

Harnesses

The benchmark has two axes. --model picks the LLM; --harness picks the decision architecture around it. Every harness gets the same scenarios, saves, neutral scenario brief (goals, state, MAP_SUMMARY, action catalog) and scoring, and is paired against the same unmanaged baseline. Leaderboard rows are harness/model.

python scripts/run_benchmark.py --harness list                                  # installed plugins + availability
python scripts/run_benchmark.py --harness felix --harness opencode --model openai/gpt-4o --runs 4
python scripts/run_scenario.py crashlanded --harness grok-build --model grok-4.6 --tick-interval 30
python scripts/run_scenario.py crashlanded --harness felix --harness-opt no_think=true --harness-opt parallel=false

Coding-agent harnesses attach to RLE over MCP: RLE hosts a RimAPI tool server in-process (rle.mcp, extra mcp), the agent calls get_brief, then write tools (work_priority, blueprint, ...), then end_turn; the writes that reached the game are what gets scored. --smoke-test needs none of the binaries — each plugin ships a scripted stand-in that plays the same round trip.

The Felix harness: 7 agents

Prerequisites

You need four things set up:

1. RimWorld (Steam) with Harmony and RIMAPI mods subscribed and enabled in the Mods menu. Load order: Harmony → Core → (DLCs) → RIMAPI.
2. LLM provider — LM Studio (local, free) or OpenRouter (cloud)
3. Python 3.14+ with uv
4. Save file — rle_crashlanded_v1 in RimWorld's save folder (the scenario auto-loads it)

RIMAPI note: The Workshop version may not have our contributed endpoints yet. See CLAUDE.md for instructions on building and deploying our fork DLL.

Verify

# Start RimWorld, load into a colony, then:
curl http://localhost:8765/api/v1/game/state   # RIMAPI (must be in-game, not main menu)
curl http://localhost:1234/v1/models            # LM Studio (if using local)

Quick Start

Install

git clone https://github.com/AppSprout-dev/RLE.git
cd RLE
uv sync --extra dev --extra felix     # core + the Felix harness
# add --extra mcp for the RimAPI MCP server used by external coding-agent harnesses

Core is framework-free; felix-agent-sdk is an optional extra. Without it, --harness felix shows as unavailable in --harness list and everything else still runs.

Configure .env

cp .env.example .env

Edit .env with your setup:

# LM Studio (local, free)
OPENAI_API_KEY=lm-studio
PROVIDER=openai
MODEL=unsloth/nvidia-nemotron-3-nano-4b
PROVIDER_BASE_URL=http://localhost:1234/v1

# -- OR --

# OpenRouter (cloud)
OPENAI_API_KEY=sk-or-v1-your-key-here
PROVIDER=openai
MODEL=nvidia/nemotron-3-super-120b-a12b
PROVIDER_BASE_URL=https://openrouter.ai/api/v1

Important: For OpenRouter, OPENAI_API_KEY must be your OpenRouter API key. The OpenAI SDK reads this env var directly. CLI flags (--provider, --model, --base-url) override .env values.

Run a live scenario

# Local (LM Studio, Nemotron Nano 4B)
python scripts/run_scenario.py crashlanded \
  --provider openai \
  --model unsloth/nvidia-nemotron-3-nano-4b \
  --base-url http://localhost:1234/v1 \
  --no-think --no-pause --visualize --ticks 10 \
  --output results/live --tick-interval 30

# Cloud (OpenRouter, Nemotron 30B)
OPENAI_API_KEY=<your-openrouter-key> \
python scripts/run_scenario.py crashlanded \
  --provider openai \
  --model nvidia/nemotron-3-nano-30b-a3b \
  --base-url https://openrouter.ai/api/v1 \
  --no-think --no-pause --visualize --ticks 10 \
  --output results/live --tick-interval 30

The scenario will:

1. Load the save file (rle_crashlanded_v1)
2. Wait for the game to be ready
3. Unforbid all starting items
4. Unpause and start running agents

Key flags

Dashboard (optional, 3 terminals)

# Terminal 1: Run scenario with --output
python scripts/run_scenario.py crashlanded --output results/live ...

# Terminal 2: Serve tick data (CORS-enabled file server on :9000)
python scripts/serve_dashboard.py results/live

# Terminal 3: React dashboard on :3000
cd ../rimapi-dashboard && bun run start
# Open http://localhost:3000, add the 5 RLE widgets

Other commands

# Smoke test: mock game state + each harness's mock agent (no LLM, no RimWorld)
python scripts/run_benchmark.py --smoke-test --ticks 10 --harness felix --harness baseline

# Harness matrix against a live game, 4 paired runs each
python scripts/run_benchmark.py --harness felix --harness opencode --runs 4 --output results/matrix/

# List scenarios / harnesses
python scripts/run_scenario.py --list
python scripts/run_benchmark.py --harness list

# Visualize CSV results
python scripts/visualize_results.py results/ --all

# Cross-model leaderboard vs the pinned baseline (after a spread)
python scripts/analyze_spread.py --spread-dir results/spread

Benchmark Results

11-model spread, felix harness, scoring 1.1 — Crashlanded, 10 ticks, seed 42, 2026-06-11. N=1, content-first — not statistically valid (no confidence intervals). Winners advance to N=4; N=4 is not published. Ranked by mean composite across the run. Featured numbers live on the HF card and rle.appsprout.dev (same site_data.json payload).

These rows predate the harness axis and scoring 1.2: they were all produced by the Felix harness and include the since-removed coordination / communication_efficiency metrics, so they are not comparable to 1.2 runs. The next published spread will be a harness × model matrix at scoring 1.2.

Measured against a pinned no-agent baseline (4 seeds, mean time-to-end 8.0 days). 1 of 11 models beat the no-agent baseline (GLM-5.1). Highest mean composite is not the same as beating baseline. Costs marked ~ are token-count estimates (subscription-billed); unmarked costs are OpenRouter billed.

Scenarios

Scoring

9 metrics, weighted composite (SCORING_VERSION = "1.2"; scenarios can override weights):

Both process metrics (efficiency, plan_coherence) are computed from the writes that actually reached RIMAPI, so any harness is scored the same way, and both return a neutral 0.5 for ticks with no writes so the unmanaged baseline earns no free points. The pre-1.2 coordination / communication_efficiency metrics were removed because they were ≈1.0 by construction (issue #51).

plan_coherence is measured after whatever coordination a harness does internally, so it is a floor a competent harness clears rather than a way to rank harnesses against each other; the colony outcome metrics (86% of the weight) are what separate them. Harness-internal process data (Felix's CentralPost traffic, resolver conflicts, a coding agent's tool-call count) is recorded in the event log as diagnostics, not scored.

Every run also records harness, harness_options, harness_versions, per-tick step latency and cost, and a harness_failed flag when RIMAPI plumbing errors (null-ref cascades, invalid plant defs) occurred; quarantined runs are excluded from leaderboard means.

Development

uv sync --extra dev --extra felix --extra mcp
pytest                                      # Run all tests (Felix-only modules skip without the extra)
ruff check src/ tests/ scripts/             # Lint
mypy src/                                   # Type check
python scripts/check_harness_boundary.py    # felix confined to harness/felix; no third-party harness code in tree

CI runs the suite twice — with and without the felix extra — plus a contract job that installs rle-harness-template from GitHub and checks it appears in --harness list and passes --smoke-test.

Related Repos

License

MIT

Built by AppSprout with Claude Code