Skillget

the_honest_agent

mcp
Security Audit
Warn
Health Warn
  • License — License: MIT
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Low visibility — Only 5 GitHub stars
Code Pass
  • Code scan — Scanned 12 files during light audit, no dangerous patterns found
Permissions Pass
  • Permissions — No dangerous permissions requested

No AI report is available for this listing yet.

SUMMARY

Open-source evals for agentic analytics: test your AI analytics agent's accuracy and provenance over its own MCP server.

README.md

The Honest Agent: open-source evals for agentic analytics

Docs · Getting started · Example report · Changelog

The Honest Agent (honest-agent)

Your analytics agent can give a confident answer from the wrong table.
honest-agent asks it questions you know the answers to, through the agent's own MCP
server, and checks each answer for two things:

  • accuracy: is the answer right?
  • provenance: did the agent's SQL read the tables it should?

Alpha (v0.2.0): expect breaking changes between versions. Each release's notes in
CHANGELOG.md say what changed and what to do.

The honest-agent report: scores per eval over ten runs, and the evals below their threshold

What it does

  • Tests your real agent. It runs Claude or GPT with the tools your agent's MCP server
    offers (Snowflake, MotherDuck or your own), not a copy of them.
    Connect your agent
  • Grades every answer against your thresholds, by matching text, extracting a number, or
    an LLM judge. Writing evals
  • Keeps every run in a DuckDB file (local or in S3) or MotherDuck, with what each call sent and received.
    Storing results
  • Builds a report you can host for your team.
    The report
  • Alerts in Slack when an eval falls below its threshold.
    Setting up alerts

Quickstart

You need uv and an Anthropic or
OpenAI API key. Try it on the bundled example, a demo agent over DuckDB's sample data:

mkdir my-agent-evals && cd my-agent-evals
uv init --bare --python 3.11
uv add "honest-agent @ git+https://github.com/jlcoto/[email protected]#subdirectory=cli"
uv run honest-agent init --example
cd honest-agent-example          # then add your API key to .env
uv run honest-agent run
uv run honest-agent report
uv run honest-agent serve        # opens the report in your browser

To evaluate your own agent, follow
Getting started.

Documentation

Repository

  • cli/: the honest-agent Python CLI
  • ui/: the source of the web report
  • site/: the landing page and these docs
  • example_project/: a demo project used for development

Contributing

Bug reports and pull requests are welcome. CONTRIBUTING.md covers the
dev setup, tests and docs; SECURITY.md how to report a vulnerability.

License

MIT

Reviews (0)

No results found