evalkit

By InferEdge.

Make your AI agent better, end to end.

evalkit is a self-hosted system for improving AI agents. Real traffic goes
in; a better agent comes out, with evals your domain experts signed off on.

TL;DR Mine failures from your agent's real runs and turn them into
eval cases. Your domain experts review the cases in plain language, so
"right" means what the business means. Run them, read every trajectory,
and get file-cited fixes for your prompts, tools and skills. Apply,
re-run, repeat until green. Your coding agent drives the loop with
evalkit's skills; evalkit keeps the cases, reviews, runs and history.

flowchart LR
  T["Your agent's<br/>real runs"] -->|mine| C["New cases"]
  C -->|review| R["Approved cases"]
  R -->|run| S["Scores +<br/>trajectories"]
  S -->|diagnose| F["File-cited fixes"]
  F -->|apply| A["Better agent"]
  A -->|re-run| S
  A -.->|new runs| T

How evalkit improves your agent

1. Find what goes wrong. Pull a batch of your agent's app runs from
   production, staging or manual testing. Your coding agent analyzes each
   one, groups the failure patterns and drafts eval cases.
2. Agree on what right looks like. Domain experts read each case as a
   question and the facts a right answer must contain, then approve it,
   comment, or send it back with changes. Approved cases are promoted to
   your production suite.
3. Measure it. evalkit runs the cases against your agent, keeps the full
   trajectory of every run (messages, reasoning, tool calls, tokens, cost,
   timing) and scores it with LLM-judged rubrics, built-in tool-call checks
   and your own evaluators.
4. Fix it. Your coding agent pulls the failed runs, diagnoses each one,
   and proposes fixes that name the prompt, tool description, skill or data
   source to change.
5. Prove it. Apply the fixes, re-run, and compare the two experiments
   case by case: what got fixed, what broke. The cases stay as regression
   guards for every change after.

Steps 1, 4 and 5 run as agent skills (evalkit skills install) in Claude
Code or another coding agent. Alongside them are skills for writing cases
with a domain expert and connecting your traces.

Developers and domain experts, one loop

An agent is only as good as the definition of "right" it is tested
against. Developers know the prompts, tools and harness; domain experts
know what a correct answer is. evalkit gives each side its own view of the
same cases.

• Cases in plain language. A case is a question plus rubrics, one
  sentence each: "States 17,133 company-operated stores at FY2021
  year-end." An expert can read and judge it without reading code. The
  case-authoring skill interviews the expert in their own terms and writes
  the YAML.
• Review in the web UI. Unreviewed → Review requested → Approved, or
  Changes requested with a comment. Threaded comments sit on the case, and
  each case keeps a changelog of every sync and decision. Review never
  filters a run: every case runs, and approval decides which version is
  promoted.
• Workspaces. Each developer has a personal workspace to iterate in
  without touching anyone's results. Shared workspaces such as dev and
  prod hold the team's datasets, agents and experiments, with writer
  and reviewer members.
• Promotion. Link dev to prod, and approving a case in dev copies
  that exact version into prod. Your prod suite only ever holds reviewed
  cases.

It's that simple

# eval/cases/smoke/cases.yaml
cases:
  - id: store-count
    input:
      question: "How many company-operated stores did Starbucks have at FY2021 year-end?"
    evaluators:
      output:
        answer_rubrics:
          rubrics:
            - rubric: "States 17,133 company-operated stores at FY2021 year-end."
              type: factual_retrieval

# eval/run.py
from evalkit.sdk import run_experiment, tracing
from myagent import ask

tracing.register(agent="myagent")  # your agent's spans become its trajectory


def task(input: dict) -> str:
    return ask(input["question"])


result = run_experiment(dataset="smoke", agent="myagent", task=task)
print(result.url)  # the scorecard in the web UI

Why evalkit

• Failures you can read. Open a low score and see exactly what the agent
  did, step by step.
• Plug in your traces. Already emitting OpenTelemetry spans with
  OpenInference or GenAI conventions? evalkit maps them into trajectories.
  Any harness, any model.
• See your app runs too. Browse your agent's real runs with their
  latency, tokens, cost and tool calls, not just eval runs.
• Graders you choose. Any OpenAI-compatible model as the judge, and your
  own evaluators in Python or over HTTP.
• Cases are code. They live in your repository and sync from CI, like
  the rest of your code.

Run your own evalkit

1. Try it locally. pip install "evalkit-py[server]", then
   evalkit init-server against a local Postgres, or use Docker.
   Try locally starts the server on your machine,
   judges with your LLM, and runs a sample agent through an SDK run with
   traces.
2. Deploy it for your team. The
   evalkit-deployment skill covers the
   server image, Postgres, blob storage, the judge, migrations, users and
   workspaces. Config and deploy
   has the config reference and the compose setup.
3. Connect your agent.
   • SDK runs with Tracing
     and Mappings: the default path.
   • Harness API: when the server should drive
     your agent.

Use a team's evalkit

If your team already runs a server, open its /docs page: the same pages,
with your server's URL filled in. Start with the
Quickstart.

On GitHub, {{origin}} in the docs stands for your server's URL.

Reference

The docs and the Python SDK reference are published at
inferedge-ai.github.io/evalkit.

• Concepts: the terms
• Install, Login,
  Config, Workspaces
• skills/: agent skills for authoring cases, eval-driven development,
  connecting traces and day-to-day use. evalkit skills install copies them
  into your repository.

Feedback

• Bugs, feature requests and feedback: open an
  issue.
• Questions: ask in
  Discussions.
• Security vulnerabilities, or anything else: email
  [email protected]. Don't post
  vulnerabilities in a public issue.

evalkit doesn't take pull requests, but issues are very welcome: a clear
bug report or idea helps just as much.