AI Evals Toolkit: LLM judges, eval datasets and rubrics

An open-source, practical toolkit for evaluating AI agents and LLM features. It covers LLM-as-a-judge prompt templates, code-based evaluators, a method for designing eval datasets (the User Input Grid), AI PRD and eval rubric templates, judge alignment (TPR/TNR) and Claude Code skills that automate each step.

It was built live for the Reforge AI Evals course, taught by Sandhya Hegde and Justin Bauer of Calibre. The methods apply to any AI product, from a single-turn prompt to a multi-step agent. Market Map, the example agent used throughout, is only the demo.

Who it's for: product managers, AI engineers and founders who want to go from "I have a prompt that works" to "I have an eval loop I trust", and then to monitoring the agent in production.

Contents

• Quick start
• What's in this repo
• The User Input Grid (UIG)
• Key ideas
• FAQ
• The example agent: Market Map
• Course sessions
• About
• License

Quick start

Prerequisites: a Braintrust account, an Anthropic API key for the LLM judges, and Python 3.10+.

git clone https://github.com/Calibre-Labs/reforge-ai-evals.git
cd reforge-ai-evals
pip install anthropic
bash install-skills.sh   # installs /uig, /eval-code, /eval-llm-judge, /llm-align, /ticket-to-eval

Then open Claude Code in your own project and run /uig to design an eval dataset, or /eval-llm-judge to write your first judge.

Run the evaluators locally:

from evaluators import company_count, has_sources, ranking_quality_judge

output = "your model output here"
input_query = "team chat"

print(company_count(output, input_query))          # 0.0, 0.5, or 1.0
print(ranking_quality_judge(output, input_query))  # {"score": 1.0, "metadata": {...}}

Run the Market Map demo in Braintrust:

1. Import datasets/week2-dataset-30.csv into Braintrust as a new dataset
2. Paste prompts/market-map-prompt.md into the Braintrust playground as the system prompt
3. Run an experiment against the dataset
4. Add the scorers in evaluators/ to the experiment

Optional: braintrust-write MCP server. This lets Claude insert dataset rows directly into Braintrust (/ticket-to-eval uses it). Run pip install -r mcp/requirements.txt; install-skills.sh prints the config snippet to add to ~/.claude/settings.json.

What's in this repo

skills/: Claude Code skills for AI evals

Install them with bash install-skills.sh, which copies each one to ~/.claude/commands/.

rubric-templates/: AI PRD and eval rubric examples

Corner and Ledger were built live during the Eval Rubrics that Drive AI Product Strategy webinar. They sit at opposite ends of the same framework. Corner is a consumer product where asking the user is expensive and mistakes are small and recoverable. Ledger is an enterprise product where asking is free and a mistake compounds.

evaluators/: code-based evaluators and LLM judges

evaluators.py contains every evaluator as a plain Python function, for local runs or as Braintrust scorer functions. The other files contain one evaluator each, written as a Braintrust custom scorer (handler function).

Code-based evaluators (deterministic, no LLM call):

LLM-as-a-judge evaluators (semantic):

Each judge prompt has 3 few-shot examples (a clear PASS, a clear FAIL and a BORDERLINE case) and returns structured JSON with a critique before the score.

datasets/: example eval datasets

All datasets are Braintrust-format CSVs and can be imported directly.

Metadata tags (JSON in the metadata column):

{
  "query_type": "direct_category | competitive_comps | acquisition_targets | historical_snapshot | future_speculative | segment_specific | validation | trend_evolution | edge_out_of_scope",
  "domain": "tech_saas | healthcare | financial | consumer_brand | industrial_other",
  "style": "well_specified | under_specified | multi_constraint | jargon_heavy | edge_out_of_scope",
  "temporal": "current | historical | future",
  "edge_case": true | false
}

The User Input Grid (UIG)

The User Input Grid (UIG) is a method for designing AI eval datasets that cover the real diversity of inputs an agent will face in production. You list the dimensions along which user inputs vary (such as query type, domain, phrasing style and time frame), list the values each dimension can take, and then check that your dataset hits every value several times. It replaces datasets built from engineer intuition, which overfit to what engineers know, or from raw user logs, which overfit to easy, common cases.

• General method: skills/uig-skill.md (or run /uig)
• Worked example: docs/uig-market-map.md

Market Map UIG: 4 dimensions

The Week 1 dataset (10 rows) had zero coverage of the Financial domain and the Segment-Specific query type. The Week 2 dataset was designed to fill those gaps deliberately.

Key ideas

Evaluator design

• Code-based evaluators check structure. LLM judges check semantics. Use both.
• Write one criterion per judge. A compound judge is uninterpretable when it fails.
• The borderline few-shot example is worth more than the pass example: it calibrates the gray zone.
• Validate judges against human labels (TPR/TNR) before trusting them in production.

Dataset design

• 30–80 queries that hit each dimension value 3+ times outperform 500 random queries.
• Tag every row with its dimension values. You need to slice by dimension to diagnose failures.

The eval loop

• Evals are only useful if they run on every prompt change. Wire them into your workflow.
• A score going up only means something if you know what changed and which eval category improved.

Judge calibration

• False negatives (judge too strict) and false positives (judge too lenient) have different root causes and need different fixes. Read the judge's reasoning for each disagreement before changing the prompt.
• /llm-align automates this for any Braintrust experiment.

FAQ

What is an AI eval?

An AI eval is a repeatable test that scores an AI feature's outputs against defined criteria, so you can tell whether a prompt, model or agent change made the product better or worse. A useful eval setup has three parts: a dataset of realistic inputs, evaluators that score each output, and a loop that reruns them on every change.

What is LLM-as-a-judge?

LLM-as-a-judge means using a language model to grade another model's output against a written criterion. It's for semantic qualities code can't check, such as whether the ranking is correct or whether an ambiguous query was handled well. This repo's judges each test one criterion, give binary PASS/FAIL definitions, include 3 few-shot examples (PASS, FAIL, borderline) and return JSON with a critique before the score. See skills/eval-llm-judge-skill.md.

When should I use a code-based evaluator instead of an LLM judge?

Use code for anything structural and deterministic: counts, required sections, citations, formats. It's fast, free and never drifts. Use an LLM judge for meaning: correctness, relevance, tone, edge-case handling. Most products need both. See skills/eval-code-skill.md.

How do I know whether my LLM judge is trustworthy?

Label 20–40 outputs by hand (roughly half PASS, half FAIL), run the judge on the same outputs, and compute TPR (how often the judge agrees when a human said PASS) and TNR (how often it agrees when a human said FAIL). Aim for both above 90%, and treat 80% as the minimum. Track them separately, because a judge with high overall accuracy can still be too lenient. /llm-align automates this.

How big should an eval dataset be?

Coverage matters more than size. 30–80 queries that hit every value of every UIG dimension at least 3 times will usually beat 500 random queries. Start small, tag every row, and add rows where the gap analysis or production failures show you're blind.

What is a User Input Grid?

A method for designing eval datasets around the dimensions along which real user inputs vary, so you cover them deliberately instead of by accident. See The User Input Grid.

What goes in an AI PRD and an eval rubric?

An AI PRD specifies the agent's behavior, not screens. It includes how much autonomy the agent has and the release thresholds that gate a launch. The eval rubric lists the criteria the agent is judged on, with the definition of success and known risks next to the criteria they justify. Write a v1 rubric from the PRD before you read any traces, then sharpen it into v2 after trace analysis. See rubric-templates/.

How do I turn production failures into regression tests?

Take the support ticket or trace, strip the PII, classify the failure mode, map it to your UIG dimensions, and add it to a regression dataset, plus a few generalized variants so you test the failure pattern rather than just the one query. /ticket-to-eval does this end to end (example).

Do I need Braintrust or Claude to use this?

No. The methods (UIG, judge design, TPR/TNR, rubrics) work with any eval platform and any model. The datasets and scorers are formatted for Braintrust, and the skills run in Claude Code.

The example agent: Market Map

The Market Map agent takes a market research query and returns a ranked list of the top 3 players in that market, with supporting metrics and citations.

• "team chat" → ranks Microsoft Teams, Slack, Google Chat
• "what AI security startups could Okta acquire in 2026 with its cash?" → a scoped acquisition-target analysis
• "search in 2003" → a period-accurate historical market map

It is a single-turn prompt with no tools or retrieval, so all its knowledge comes from the model's weights. That makes it a good eval target: the failure modes are predictable and the output format is fixed. Prompt: prompts/market-map-prompt.md.

Course sessions

About

Built by Sandhya Hegde and Justin Bauer, co-founders of Calibre, an applied AI research and consulting firm. They teach the AI Evals course on Reforge, and Calibre writes about AI product quality at blog.calibrelabs.ai.

License

Code (evaluators/, mcp/, install-skills.sh) is released under the MIT License. Everything else, including the skills, rubric templates, PRDs, datasets, prompts and docs, is released under Creative Commons Attribution 4.0 (CC BY 4.0): reuse and adapt it freely, with credit to Calibre.

If you use this toolkit, cite it with GitHub's "Cite this repository" button (see CITATION.cff).