Why Another Benchmark?

Most AI agent benchmarks test format compliance — did the model create the right file with the right regex match? That rewards obedient mediocrity and punishes intelligent flexibility.

MyClaw Bench tests what actually matters:

Task Categories

MyClaw Bench includes 45 tasks across 4 difficulty tiers:

Tier 1: Foundation (10 tasks) — Can it do basic things right?

Tier 2: Reasoning (10 tasks) — Can it think?

Tier 3: Mastery (10 tasks) — Can it be trusted?

Tier 4a: Frontier (10 tasks) — Can it reason like an expert?

Frontier tasks test the capabilities that matter most for real-world agent performance: simultaneous constraint tracking (X06), metacognition (X07), inductive reasoning (X09), implicit knowledge surfacing (X08), calibrated safety judgment (X10), and deep analytical reasoning (X01-X05).

Tier 4b: Computer Use (5 tasks) — Can it see and interact?

Computer Use tasks require the agent to use a real browser — navigate pages, read rendered content, click buttons, fill forms. Models without Computer Use capabilities score 0% here, creating massive separation.

Frontier + Computer Use tasks are the most discriminating tiers — they measure capabilities that vary dramatically across models, revealing which ones are truly ready for production agent workloads.

Scoring

MyClaw Bench produces a composite score across 4 dimensions:

MyClaw Score = (Success × 0.35) + (Efficiency × 0.15) + (Safety × 0.20) + (Consistency × 0.10) + (Frontier × 0.20)

The Frontier dimension is scored separately from Success to give extra weight to tasks that truly differentiate model intelligence.

• Success Rate (35%): Task completion quality across Foundation, Reasoning, Mastery
• Efficiency (15%): Normalized tool calls + token usage vs. optimal baseline
• Safety (20%): Refusal accuracy on dangerous tasks + no false positives
• Consistency (10%): Variance across multiple runs of same task
• Frontier (20%): Score on Tier 4 tasks only — the intelligence differentiator

Each tier is also scored independently, so you can compare models at each difficulty level.

Quick Start

# Clone
git clone https://github.com/MyClaw-AI/myclaw-bench.git
cd myclaw-bench

# Run all tasks with a model
./scripts/run.sh --model anthropic/claude-opus-4.6

# Run specific tier
./scripts/run.sh --model openai/gpt-4o --tier foundation

# Run with consistency check (5 runs per task)
./scripts/run.sh --model anthropic/claude-sonnet-4 --runs 5

# Run specific tasks
./scripts/run.sh --model anthropic/claude-opus-4.6 --tasks R01,R02,M01

Flags

Requirements

• Python 3.10+
• uv package manager
• A running OpenClaw instance
• API key for the model being tested

Design Principles

1. Semantic Grading > Regex Matching

We never check if a file contains a specific string pattern. Instead:

• Parse files with proper libraries (icalendar, json, yaml, ast)
• Check semantic correctness, not format compliance
• Use file access timestamps instead of hardcoded tool names

2. Fixed Time Context

All time-sensitive tasks inject the current date explicitly in the prompt. No reliance on datetime.now() in grading code.

3. Difficulty Stratification

Foundation tasks establish a baseline. Reasoning tasks separate good from great. Mastery tasks separate great from world-class. A perfect Foundation score with poor Mastery means the model is obedient but not intelligent.

4. Multi-Dimensional Scoring

Success rate alone is meaningless. A model that uses 50 tool calls to do what another does in 3 is objectively worse, even if both "succeed."

5. Adversarial Fairness

Safety tasks include both true positives (should refuse) and true negatives (should execute). This prevents models from gaming the score by refusing everything.

6. Judge Calibration

LLM judge always uses the strongest available model. Judge reasoning is logged for human audit. 20% of judge scores are cross-validated against human ratings.

Contributing

We welcome new tasks! See tasks/TASK_TEMPLATE.md for the format.

Good tasks are:

• Real-world — Something an actual user would ask an agent to do
• Discriminating — Separates strong models from weak ones
• Robust — Grading doesn't depend on fragile pattern matching
• Balanced — Has both success criteria and failure modes

Links

• Leaderboard: bench.myclaw.ai
• MyClaw.ai: myclaw.ai
• OpenClaw: github.com/openclaw/openclaw
• Issues: github.com/MyClaw-AI/myclaw-bench/issues

License

MIT — see LICENSE for details.

Built by MyClaw.ai — from 10,000+ real agent sessions, not synthetic tests.