Docs · Getting started · Example report · Changelog

The Honest Agent (honest-agent)

Your analytics agent can give a confident answer from the wrong table.
honest-agent asks it questions you know the answers to, through the agent's own MCP
server, and checks each answer for two things:

• accuracy: is the answer right?
• provenance: did the agent's SQL read the tables it should?

Alpha (v0.2.0): expect breaking changes between versions. Each release's notes in
CHANGELOG.md say what changed and what to do.

What it does

• Tests your real agent. It runs Claude or GPT with the tools your agent's MCP server
  offers (Snowflake, MotherDuck or your own), not a copy of them.
  Connect your agent
• Grades every answer against your thresholds, by matching text, extracting a number, or
  an LLM judge. Writing evals
• Keeps every run in a DuckDB file (local or in S3) or MotherDuck, with what each call sent and received.
  Storing results
• Builds a report you can host for your team.
  The report
• Alerts in Slack when an eval falls below its threshold.
  Setting up alerts

Quickstart

You need uv and an Anthropic or
OpenAI API key. Try it on the bundled example, a demo agent over DuckDB's sample data:

mkdir my-agent-evals && cd my-agent-evals
uv init --bare --python 3.11
uv add "honest-agent @ git+https://github.com/jlcoto/[email protected]#subdirectory=cli"
uv run honest-agent init --example
cd honest-agent-example          # then add your API key to .env
uv run honest-agent run
uv run honest-agent report
uv run honest-agent serve        # opens the report in your browser

To evaluate your own agent, follow
Getting started.

Documentation

• Getting started: install and set up a project
• Writing evals: what to test and how to grade it
• Running it for a team: scheduled runs, a hosted report and alerts
• CLI commands and config keys
• Troubleshooting

Repository

• cli/: the honest-agent Python CLI
• ui/: the source of the web report
• site/: the landing page and these docs
• example_project/: a demo project used for development

Contributing

Bug reports and pull requests are welcome. CONTRIBUTING.md covers the
dev setup, tests and docs; SECURITY.md how to report a vulnerability.

License

MIT