AgentBench

The Regression Testing Framework for AI Agents

Replay · Evaluate · Compare · Assert · Catch Regressions — in CI

Quick Start · Why · DSL · Ecosystem · Examples · vs Others · Documentation · Releases

🚀 Quick Start

npm install -g @agentbench/cli
agentbench init
agentbench test

AgentBench is to AI agents what Jest is to JavaScript.

Before Jest, JavaScript testing was fragmented — every team wrote their own harness, nobody trusted anyone else's results, there was no standard. Jest gave developers a unified way to write tests, run them fast, and catch regressions in CI. Playwright did the same for browser apps, bringing deterministic assertions to an inherently visual and interactive medium.

AgentBench does the same for AI agents — a domain that is non-deterministic, LLM-powered, and tool-calling by nature. Same philosophy: assertions, snapshots, replay, coverage, CI integration. Different domain: instead of expect(2 + 2).toBe(4), you write expect(agent).tool("search").toBeCalledWith({ query: "refund policy" }). Instead of deterministic pass/fail, you get LLM-aware scoring, regression detection, and cross-model replay.

You test your code. You test your UI. You test your API. Now test your AI agents with the same rigor. One CLI. Five minutes. Ship agents with confidence.

git clone [email protected]:1304674612/agentbench.git
cd agentbench && pnpm install
docker compose up -d
cp .env.example .env && pnpm db:push
pnpm dev

📖 Why AgentBench?

"I tracked my time. Coding was 10%. Testing was 90%. Not because I'm slow — because there was no tool."

— Read the full origin story

AI Agents are unpredictable. A prompt tweak, a model upgrade, or a tool swap can silently degrade your agent -- and most teams discover this only when users complain.

AgentBench gives you the same testing rigor for your AI agents that you expect for your software.

✨ Features

await expect(runResult)
  .tool("search").toBeCalled()
  .tool("search").toBeCalledWith({ query: "refund policy" })
  .output().toContain("30 days")
  .tokens().toBeLessThan(4096)
  .latency().toBeLessThan(5000)
  .score("correctness").toBeGreaterThan(7)
  .run()

🔌 Assertion DSL

The most fluent way to test an AI agent. Chainable, type-safe, reads like English.

import { expect } from '@agentbench/core'

const result = await expect(runResult)
  .status().toBeCompleted()                     // Agent finished successfully
  .tool("search_docs").toBeCalled()             // Called the right tool
  .tool("search_docs").toBeCalledWith({         // Called with correct args
    query: "refund policy"
  })
  .tool("hallucinate").not.toBeCalled()         // No forbidden tools
  .output().toContain("30 days")                // Output has correct info
  .output().toMatchRegex(/refund.*policy/i)     // Pattern validation
  .tokens().toBeLessThan(4096)                  // Token budget respected
  .latency().toBeLessThan(5000)                 // Under 5 seconds
  .score("correctness").toBeGreaterThan(7)       // Quality threshold met
  .score("safety").toBeGreaterThan(8)            // Safety threshold met
  .run()

if (!result.allPassed) process.exit(1)

📦 Ecosystem

Provider Plugins

SDK Packages

📚 Examples

14 production-quality reference implementations. Each demonstrates how to test a specific kind of AI agent with comprehensive test suites.

Each example includes:

• README with quick start, architecture diagram, and key takeaways
• 3-5 test suites with 8+ test cases
• At least 3 different assertion types (tool, output, score, latency, tokens)
• Replay test suite (zero-cost, deterministic)
• CI workflow file (.github/workflows/agentbench.yml)
• Dataset of 20+ test inputs
• 100% pass rate when run against their target agent

# Run any example
cd examples/customer-support-agent
cp .env.example .env   # Add your API key
npm install
agentbench test

🆚 Competitive Positioning

AgentBench is not an observability platform or an evaluation library. It is a testing framework -- the same category as Jest, Playwright, and Pytest, but purpose-built for the non-deterministic world of AI agents.

At a Glance

AgentBench vs The Alternatives

The "Why Not Just Use X" Answers

Why not just use LangSmith?
LangSmith is for observability -- seeing what happened. AgentBench is for testing -- asserting what should happen and gating on it. You use LangSmith to debug; you use AgentBench in CI to prevent broken agents from reaching production.

Why not just use Jest?
Jest expects deterministic outputs. AI agents are non-deterministic. AgentBench provides LLM-aware assertions (tool().toBeCalled(), score("correctness").toBeGreaterThan(7)), replay, and regression detection that Jest cannot.

Why not just use DeepEval?
DeepEval evaluates the output text. AgentBench tests the agent behavior -- which tools were called, in what order, with what arguments, how many tokens were used, whether the response was fast enough, and whether quality regressed from the last run.

📊 Project Status

🏗 Architecture

agentbench/
├── apps/
│   ├── web/                    Next.js 15 Dashboard + REST API + Benchmark Marketplace
│   ├── cli/                    Commander.js CLI (12 commands)
│   └── docs/                   VitePress documentation site (25+ pages)
├── packages/
│   ├── core/                   @agentbench/core -- Engine
│   │   ├── runner/             Agent Runner
│   │   ├── tracer/             Execution Tracer + LLM Interceptors
│   │   ├── evaluator/          Rule + LLM + Hybrid Judge
│   │   ├── assertion/          Chained Assertion DSL
│   │   ├── snapshot/           Snapshot Manager
│   │   ├── replay/             Replay Engine
│   │   ├── diff/               Diff Engine
│   │   ├── experiment/         A/B Testing Engine
│   │   ├── coverage/           Coverage Analysis
│   │   ├── reporter/           Report Generator (JSON/MD/HTML/JUnit)
│   │   ├── dataset/            Dataset Management (CSV/JSON/JSONL)
│   │   ├── storage/            Storage Abstraction (Postgres + Memory)
│   │   ├── types/              TypeScript Type Definitions
│   │   └── utils/              Token Counter + JSON Validator
│   ├── openai/                 @agentbench/openai
│   ├── anthropic/              @agentbench/anthropic
│   ├── gemini/                 @agentbench/gemini
│   ├── deepseek/               @agentbench/deepseek
│   ├── azure-openai/           @agentbench/azure-openai
│   ├── openrouter/             @agentbench/openrouter
│   ├── groq/                   @agentbench/groq
│   ├── mistral/                @agentbench/mistral
│   ├── cohere/                 @agentbench/cohere
│   ├── ollama/                 @agentbench/ollama
│   ├── vllm/                   @agentbench/vllm
│   ├── lm-studio/              @agentbench/lm-studio
│   ├── provider-utils/         @agentbench/provider-utils
│   ├── mcp/                    @agentbench/mcp
│   ├── adapter/                @agentbench/adapter
│   ├── langgraph/              @agentbench/langgraph
│   └── typescript-config/      Shared TSConfig
├── examples/                   14 official examples
├── sdk-python/                 Python SDK
├── vscode-extension/           VS Code extension
├── benchmark-registry/         Benchmark registry + DB
├── docs/                       Internal docs (architecture, schema, roadmap)
├── docker-compose.yml          PostgreSQL 16 + Redis 7
└── .github/workflows/          CI pipeline

📚 Documentation

Community

AgentBench is open source and community-driven. We welcome all contributions.