EvalHub

Local-first Benchmark evaluation for LLMs and coding agents.

Public datasets, local models, persistent workflows, sample-level evidence, and six-dimension capability reports on one reproducible evaluation path.

English · 简体中文

Features · Quick Start · Screenshots · Model Report · Agent Report · MiniClaw Report · How It Works · Benchmarks · Documentation

EvalHub is a unified evaluation platform for local LLM development. It runs real public benchmarks
and also puts complete coding agents such as Pi and MiniClaw through the same tasks and the same
hidden verifier. Every run keeps sample results, node states, resource metrics, and audit evidence —
not just a final score.

_{Pick the benchmark, model, and sample range from one console, and follow the whole evaluation. Click the image for the full-size version.}

Features

Quick Start

Prerequisites

• Python 3.11+
• Node.js 20+ and npm
• Ollama for local model inference
• Docker Desktop for code benchmarks such as HumanEval and MBPP

git clone https://github.com/NEDONION/evalhub.git
cd evalhub
python3 -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
npm --prefix frontend install
./scripts/start_local.sh

Then open http://127.0.0.1:8000. The script builds the React console,
prepares the pinned benchmark runtime, and connects to or starts Ollama when it is available locally.
For full installation, Agent Runtime, CLI, and troubleshooting details, see the
local setup guide.

To evaluate a MiniClaw checkout outside the adjacent directory, point EvalHub at the project root
before starting. EvalHub never reads or stores MiniClaw API keys — models, providers, tools,
policies, and memory stay under MiniClaw's own control:

export EVALHUB_MINICLAW_ROOT=/absolute/path/to/miniclaw
./scripts/start_local.sh

Headless evaluation grants one-time approval only for write_file / edit_file calls that already
passed MiniClaw's policy check and that EvalHub re-confirms are inside the current temporary sample
workspace. Commands, network access, memory, and anything outside the workspace continue to follow
MiniClaw's own approval boundary.

Screenshots

Model benchmark workflow Inspect the task DAG, node progress, resource metrics, and execution audit.	Agent failure audit Replay agent messages, tool calls, file changes, and failure classification.
Agent six-dimension result Read the capability profile by planning, comprehension, implementation, tool use, verification, and robustness.	Model and dataset assets Manage Ollama model downloads and the cache of real benchmark data.

Model Benchmark Report

The results below are a measured snapshot taken on 2026-08-05 on the same pinned
evalhub-hexagon-v1 1.2.0 suite: 9 local Ollama models and 2 hosted DeepSeek models each
completed 30/30 samples, for 330 model evaluations in total. The overall score is passed samples
divided by 30; each dimension holds 5 samples, so a single question moves that dimension by 20
points.

Every hexagon uses the same 0–100 scale, with the axis order matching the EvalHub console:
knowledge, instruction following, math, reasoning, code, safety. Each capability polygon is a closed
path, so shapes and weak spots can be compared directly.

In this round deepseek-v4-pro has both the highest total and the most complete shape; among local
models gemma4:12b scores highest. deepseek-v4-flash shows a clear strong-code, weak-math profile.
Instruction following is already saturated on most mid-size and larger models, while code and
reasoning separate them more clearly.

[!NOTE]
This is a Mini Suite capability profile, not a complete official score on the upstream benchmarks.
Plain Ollama, thinking models with an explicit think=false, and the OpenAI-compatible DeepSeek
API form three generation-protocol groups; they share the same samples and graders, but rankings
across protocol groups are observational only. Generation temperature is 0 throughout, and the
maximum generation budgets for MMLU/IFEval/GSM8K/BBH/HumanEval/TruthfulQA/BBQ are
256/1024/512/512/1024/256/256 tokens respectively. HumanEval runs its hidden tests in a pinned
Docker image. The baseline commit for this round is 4e9107c.

Agent Benchmark Report

The results below are a measured snapshot taken on 2026-08-05 on the same coding-mini-v3 suite:
the Pi agent shell is fixed and only the underlying model is swapped. 4 API models and 6 local
Ollama models each completed 6/6 official samples, for 60 agent tasks in total. Across official
questions there were 23 passes, 311 tool calls, and 60 tool errors. The protocol pre-check runs
separately and counts toward neither the pass rate nor the tool-call totals.

The hexagons share a 0–100 scale, with the axis order planning, code comprehension, implementation
correctness, tool use, verification, robustness; every capability path is a closed polygon. Click the
image to open the original SVG from the repository and zoom in.

Process evidence explains why identical zeros have different causes: gemma4:12b and
granite4.1:3b passed the protocol pre-check and did call tools, but their final workspace failed
hidden verification; granite3.3:8b and deepseek-r1:1.5b never completed the structured tool
protocol and were marked incompatible. Both qwen3:14b and qwen3:4b are compatible, so their
low scores are not a protocol misclassification.

[!NOTE]
This is a 6-task everyday diagnostic set, not an official SWE-bench score. All runs use Pi CLI
0.74.1, scaffold d8a7cbaf0d432018, and hidden workspace verification; API latency and local
Ollama timing depend on hardware and network, so elapsed time should not be read directly as model
inference speed. The separate 6-task SWE-bench Verified Mini will only publish scores once the gold
patch reaches 6/6 in the official Docker harness — this round does not pad with zeros or mix
unready results into the ranking.

MiniClaw Agent Evaluation Report

The following are auditable results for the complete MiniClaw agent on the same coding-mini-v3
suite. For each model only the first official job that completed all 6 tasks without a task-level
infrastructure failure is used; there is no score-driven rerun or best-of selection. Models,
providers, tool calls, and the reasoning loop are managed by MiniClaw — EvalHub only supplies the
identical sample workspace, the write-approval boundary, and the hidden verifier.

The official Flash job keeps the original failure in async_worker_cleanup caused by one upstream
protocol exception; the later targeted retest with the limit relaxed to 180 seconds only confirms
that the tool loop can finish and does not replace the official 5/6 result. qwen3:4b passed the
protocol pre-check, but left the workspace untouched on 5 tasks and timed out on 1, modifying 0 files
in total — so 0/6 is this round's official capability result, not a compatibility misclassification.

Six-dimension results

The table converts persisted 0–1 scores to 0–100; the axis order matches the EvalHub Agent report.

Pi vs. MiniClaw on the same models

The Pi figures come from the frozen 2026-08-05 matrix; the MiniClaw figures come from the single
official jobs above. Both sides share coding-mini-v3, the same scaffold, and the same hidden
verifier, but the agents differ in prompts, tool policy, and loop implementation.

These three single 6-task runs give no sign that MiniClaw can lift qwen3:4b from Pi's 1/6 to 5/6 —
here it passed one fewer task. Pro and Flash match Pi's pass counts while MiniClaw uses fewer tool
calls and less time per task. Such differences are useful for locating agent behavior, but a single
small-sample run is not enough to infer a general causal advantage of the shell itself.

Excluded runs

After the compatibility fix, the first valid official job for the same model is job_08aa057bf2db
(5/6). The excluded run is kept only as protocol-boundary evidence and does not enter official
results or rankings.

[!NOTE]
All official jobs use MiniClaw 0.1.0, coding-mini-v3, and scaffold d8a7cbaf0d432018; no extra
permissions were granted for commands, network, memory, or writes outside the workspace. The hidden
verifier determines pass counts and six-dimension scores, while the protocol pre-check only
provides diagnostic evidence. This suite is a 6-task everyday agent diagnostic, not an official
SWE-bench ranking; model responses, keys, and workspace contents are not written into this report.

How It Works

flowchart LR
    Visitor["Visitor"] --> Entry{"Web Console / CLI"}
    Entry --> Engine["Evaluation Engine"]
    Dataset["Public benchmark data"] --> Engine
    Engine --> DAG["Persistent evaluation DAG"]
    DAG --> Model["Model Adapter + Ollama"]
    DAG --> Registry["Complete Agent Registry"]
    Registry --> Pi["Pi + EvalHub-managed model"]
    Registry --> MiniClaw["MiniClaw + self-managed runtime"]
    Model --> Evidence["Sample results, checkpoints, audit events"]
    Pi --> Evidence
    MiniClaw --> Evidence
    Evidence --> Report["Scores, failed samples, six-dimension profiles"]

The web console and the CLI reuse the same Python evaluation core. Model jobs and agent jobs enter
the same persistent job center: progress and resources are visible while running, and node evidence
can be replayed once finished. Model evaluations aggregate by model; agent evaluations aggregate by
complete candidate, such as Pi · qwen and MiniClaw — the two are never ranked together.

Benchmarks

[!IMPORTANT]
The Hexagon Mini Suite is a local capability profile, not a complete official score on the seven
upstream benchmarks, and must not be restated as a paper or leaderboard result. Code evaluation
requires Docker; the current Ollama adapter lacks prompt logprobs, so some individual official
protocols block explicitly rather than producing fake scores through a generative approximation.

Hexagon 1.2 freezes model transport and answer grading separately: registered thinking models use a
top-level think=false on Ollama /api/generate, and the seven benchmarks use multiple-choice,
verbatim IFEval rules, numeric, BBH subtask, and HumanEval code protocols respectively. An empty
final answer blocks the node instead of being stored as a zero; a non-empty but wrong answer is still
scored zero under the same standard.

Project Status

EvalHub is currently a Local MVP: the React console, Python service, SQLite-persisted jobs,
public dataset cache, and local Ollama inference are all in place. It suits local development and
reproducible experiments, but is not yet positioned as a multi-tenant production evaluation service.

Documentation

Note: the documents linked above are currently written in Chinese.

Contributing

Issues and suggestions are welcome through Issues, as
are small, focused pull requests. Before submitting, please run:

.venv/bin/python -m ruff check .
.venv/bin/python -m pytest

Development constraints and common commands are documented in AGENTS.md.