Find the step that caused your AI agent's failure.

Your tracing tool shows what your agent did. Tracedown tells you which step broke it, shows the evidence, and replays the run with your fix. Open source, and it works on top of the traces you already have.

Works with LangGraph · CrewAI · OpenAI Agents · Claude Agent SDK · Google ADK · Pydantic AI · Vercel AI SDK · MCP · Langfuse · Arize Phoenix · OpenTelemetry

Try the live demo · Quick start · How it works · Integrations · Benchmarks · Comparison · Docs

A support agent tells a customer they will get $120 back; the real answer is $80. Every step of the run succeeded. Tracedown flags the $120 claim as coming from a stale source, walks back to step 2 (an outdated refund policy), says the agent was at fault, and replays the run from step 2 with the current policy, which gives $80.

_{A real run from tracedown_server/demo/agents.py. Every verdict shown is Tracedown's actual output. Watch in HD, or click through it yourself (the free host can take a minute to wake up).}

Why Tracedown

AI agents rarely crash. They fail quietly: a model writes "Friday" where a date belongs, a retriever returns last year's policy, a loop burns forty tool calls. Tracing tools record every step, and then leave you to work out which one broke the run.

Tracedown is the flight recorder that also reads the recording. It walks back from the symptom to the step that caused it, and shows the evidence.

Works with what you have. Import failed runs from Langfuse, Arize Phoenix or any OpenTelemetry exporter with no code changes, or add one line to 13 frameworks.
Explains the failure. The root cause, the evidence, and a verdict on who was at fault: the agent, or the user.
Replays with your fix. Earlier steps come from the recording; from the broken step on, the agent runs live with your change. Needs a run recorded with the SDK.
Runs without an LLM. Diagnosis is rules first: instant, free and private. An inexpensive LLM judge is optional.
Measured. Accuracy on a public, human-labelled benchmark is published, losses included.
Free and self-hosted. Apache-2.0, every feature, including SSO, roles, audit log and scale-out storage.

Quick start

See it work. One command starts the server and the web app, loads eight agents that fail in realistic ways (from a stale document to a prompt injection), and opens your browser. No API key, Node or Docker needed.

pip install agent-tracedown-server
tracedown up                                      # http://127.0.0.1:7070

If 7070 is in use, tracedown up tries 7071 through 7079 and prints and opens the
chosen URL. Use that port for your SDK endpoint too. Set --port or TDN_PORT to
require a particular port; an occupied explicit port still stops startup.

Needs Python 3.10+. A Mac's built-in Python is 3.9; with uv instead: uvx --python 3.12 --from agent-tracedown-server tracedown up.

Prefer the terminal? tracedown demo prints every diagnosis instead.

Diagnose the runs you already have. No code changes: paste an export into Diagnose a trace in the web app, or import from the command line. A 5-minute tutorial walks through it.

export LANGFUSE_PUBLIC_KEY=pk-lf-... LANGFUSE_SECRET_KEY=sk-lf-...
tracedown-import langfuse --errors-since 24h      # failed runs from the last day
tracedown-import otlp traces.json --failed        # or any OpenTelemetry export

Imported runs get the full diagnosis. Replay needs a run recorded with one of the SDKs below, because it runs your agent's own code.

Instrument your own agent. The SDKs have no dependencies.

import agent_tracedown as tdn                     # pip install agent-tracedown
tdn.init(project="support-bot", endpoint="http://localhost:7070")

with tdn.run("CalendarAssistant", input=message) as run:
    reply = tdn.llm_call(call_model, model="gpt-4o-mini", prompt=messages)
    tdn.tool_call("create_event", args, create_event, schema=SCHEMA)
    run.output(reply)

TypeScript

import * as tdn from "agent-tracedown";           // npm install agent-tracedown
tdn.init({ project: "support-bot", endpoint: "http://localhost:7070" });

await tdn.run("CalendarAssistant", { input: message }, async run => {
  const reply = await tdn.llmCall(callModel, { model: "gpt-4o-mini", prompt: messages });
  await tdn.toolCall("create_event", args, createEvent, { schema: SCHEMA });
  run.output(reply);
});

The TypeScript and Python SDKs record the same data, and the same run gets the same diagnosis from either.

Run it for a team. tracedown up is one process on SQLite. For Postgres, several workers, Docker Compose or Kubernetes, see Deployment:

git clone https://github.com/nbn07/agent-tracedown
cd agent-tracedown/deploy && docker compose up -d   # web app on :3000, API on :7070

How it works

1. Record	2. Diagnose	3. Replay
Every step of a run: model calls, tools, retrievals, memory and hand-offs. Or imported from your existing traces.	Walk back from the symptom to the root cause, with evidence, and decide who was at fault.	Re-run from the broken step with your fix. Earlier steps come from the recording, so nothing happens twice.

How Tracedown decides walks through each stage, with its limits. Every failed run gets an AI stack trace you can paste into an issue:

AgentFailureException: Invalid tool arguments
  at        step 5  PlannerAgent → execute_tool calendar_create_event
              start must be ISO-8601 date-time, got 'Friday'
  caused by step 4  PlannerAgent → chat gpt-4o-mini
              PlannerAgent produced 'Friday' at step 4, which calendar_create_event() could not use
  root cause: Bad LLM output (top suspect; check the evidence)
  evidence:   step 4 output 'Friday' is used verbatim in step 5 arguments
  next:       replay from step 4 with your fix: tdn.replay(bundle, mode="from-step", from_step=4)

Beyond single runs, Tracedown checks each claim in an answer against its sources, compares runs across prompt and model versions, groups failures into incidents with Slack, PagerDuty or GitHub alerts, and shows cost per run, step, model and agent.

Run detail in the web app: the AI stack trace, who is at fault, and the step graph with the cause highlighted

Integrations

Framework	One line
LangChain, LangGraph	`graph.invoke(x, config={"callbacks": [tdn_lc.callback("MyGraph")]})`
OpenAI Agents SDK	`tdn_oa.instrument()`
Claude Agent SDK	`async for m in tdn_claude.trace_stream(query(prompt=p, options=o), name="Agent")`
Google ADK	`App(name="support", root_agent=agent, plugins=[tdn_adk.TracedownPlugin()])`
Pydantic AI	`result = await tdn_pai.run(agent, prompt)`
MCP clients	`tdn_mcp.instrument(client)`
CrewAI	`tdn_crew.instrument()`
AutoGen AgentChat	`async for m in tdn_ag.trace_stream(team.run_stream(task=t), name="Team")`
Semantic Kernel	`tdn_sk.instrument(kernel)`
LlamaIndex	`tdn_li.instrument()`
OpenAI and Anthropic clients	`tdn.plugins.instrument("openai", client)`
Vercel AI SDK (TypeScript)	`generateText({ model: traceModel(model), tools: traceTools(tools) })`
OpenAI Agents SDK (TypeScript)	`await instrument()` from `agent-tracedown/openai-agents`
Existing traces	Langfuse, Arize Phoenix, OpenInference, OpenLLMetry, any OTLP exporter

Tested versions

Each adapter is tested in CI against the real library: LangGraph 1.2, openai-agents 0.24, claude-agent-sdk 0.2, google-adk 2.13, pydantic-ai 2.64, mcp 1.30 and 2.3, CrewAI 1.15, AutoGen 0.7, Semantic Kernel 1.46, LlamaIndex 0.14, Vercel AI SDK 5 to 7, and @openai/agents 0.21. Python adapters live in agent_tracedown.integrations.

Tracedown also exports: OTLP to Jaeger, Tempo, Datadog or any collector, and Prometheus metrics at /metrics.

Benchmarks

Who&When (ICML 2025) asks which step caused each of 184 failed multi-agent runs, labelled by people. Exact match, one run, no tuning on this data:

Right step	First guess _{algorithm · hand-crafted}	In top three _{algorithm · hand-crafted}
Tracedown with a Claude Sonnet 5.5 judge	42.9% · 24.1%	58.7% · 36.2%
Tracedown with a Claude Haiku 5.5 judge (cheaper)	36.5% · 20.7%	50.8% · 32.8%
Tracedown, rules only	21.4% · 19.0%	33.3% · 31.0%
Baseline: first failure-looking output	23.0% · 15.5%	–
Baseline: random step	12.0% · 4.2%	36.0% · 12.5%
Best GPT-4o method in the paper	16.6% · 8.8%	–

The first guess is wrong more often than it is right, so treat it as the first place to look and check the evidence. On the algorithm-generated set, rules alone do no better than a random pick within the top three; the judge is what lifts it. How Tracedown decides explains the method and where it fails.

On τ-bench failures, Tracedown's first guess ties a simple rule ("blame the first database change") at 74%, and its top three contain the right step 81% of the time. It used to lose 68% to 74%; the fix was found on the development split and scored once on the test split. The method and every change after the first scoring are in LABELING.md. The judge's answers are saved in the repository, so re-scoring needs no API key.

Comparison

Tracedown is a root-cause layer that works alongside your tracing tool, not a replacement for it.

	Open source	Explains why a run failed	Replays from a step with a fix	Published accuracy	Works without an LLM
Tracedown	Apache-2.0, every feature	Yes	Yes	Yes	Yes
AgentDebugX	MIT	Yes, with a proposed fix	Yes, after approval	Yes	Partly
Comet Opik	Apache-2.0 core	Yes, with a suggested fix	Reruns failing inputs	No	No
Laminar	Yes	Pattern search in plain English	Yes	No	Search uses an LLM
Arize Phoenix	Elastic License 2.0	Through an AI assistant	Model calls only	No	No
TraceRoot	Apache-2.0 core	Yes, against your source code	No	No	No
Langfuse	MIT core	No	No	–	–
LangSmith, Arize AX, Galileo	Commercial	Yes	Not documented	No	No

AgentDebugX is the closest project. Scored the same way on Who&When (agent and step both right, all 184 runs), Tracedown reaches 36.4% with a Claude Sonnet 5.5 judge and 31.5% with Claude Haiku 5.5; AgentDebugX with Qwen 3.5 9B reaches 28.8%. The models differ, so this compares setups, not methods.

_{Based on each project's documentation, the AgentDebugX paper and Arize's 2026 comparison, October 2026. Something out of date? Open an issue.}

Architecture

flowchart TB
    subgraph sources["Your agents and traces"]
        direction LR
        py["Python SDK"] ~~~ ts["TypeScript SDK"] ~~~ imp["tracedown-import<br/>Langfuse · Phoenix · OTLP"]
    end
    subgraph server["Tracedown server"]
        direction LR
        api["Ingest API"] --> q[("Kafka<br/>optional")] --> pipe["Analysis<br/>root cause · fault · claims<br/>drift · incidents · cost"]
    end
    subgraph outputs["Results"]
        direction LR
        store[("SQLite or Postgres<br/>ClickHouse at scale")] --> web["Web app"]
        alerts["Slack · PagerDuty · GitHub<br/>OTLP · Prometheus"]
    end
    sources --> api
    pipe --> store
    pipe --> alerts

It starts as one process on SQLite and scales out with Postgres, Redis, Kafka and ClickHouse. A Helm chart is included.

Deployment layouts

Layout	Metadata	Spans	Ingest	For
One process	SQLite	SQLite	In-process	A laptop, a demo, one team
Standard (Compose or Helm)	Postgres	Postgres	In-process, Redis cache	Most teams
Scale (Compose or Helm)	Postgres	ClickHouse, routed by project	Kafka and workers	High volume, data residency

helm install tracedown deploy/helm/tracedown

Screenshots



Runs, with root cause, fault and cost	Agent network with hand-offs and error rates

Incidents grouped by root cause	Analytics: success, latency, cost, tool reliability

Replay: the original run and the fix	Claims checked against their sources

Documentation: Wiki · How it decides · Deployment · Configuration · Design · Langfuse tutorial · Python SDK · TypeScript SDK · Telemetry spec

Roadmap

Proven fixes (in progress for v0.5): suggest a fix, replay the run with it, and show that it now passes. Done so far: the check that decides whether a replay fixed the run, and tdn.confirm_cause()
MCP server, to ask "why did run X fail?" from Claude Code, Cursor or any MCP client
GitHub Action that runs your agent's test conversations on every pull request
Write-back to Langfuse and Phoenix, so the verdict appears in the tool you already use

Ideas are welcome in Discussions; requests with a real failing run behind them get priority.

Using Tracedown?

If Tracedown found a bug for you, or you run it at work, say so in Show and tell. With your permission your team's name goes here. It is the most useful thing you can do for the project after reporting a wrong diagnosis.

Contributing

The most valuable contribution is a run Tracedown diagnoses wrongly: report a wrong diagnosis and it becomes a benchmark case that every later version is tested against. Detectors, framework adapters and documentation are welcome too. Start with CONTRIBUTING.md or an issue labelled good first issue: each one names the files to change and how to test. Questions go to Discussions. The wiki explains how the code fits together, with step-by-step guides for a first pull request, a new detector or a new framework integration.

Nothing to install: open a ready-made development environment in your browser, with every dependency set up and the tests passing.

pip install -e ./sdk/python -e "./server[dev]" && pytest

Contributors

Thank you to everyone who has made Tracedown better:

_{HarshRajSinghania}
_{Langfuse tutorial} _{AndreIsraelAO}
_{TypeScript drift demo} _Vamsi-2581
_{@tdn.traced_run} _Sonike
_{free port for tracedown up} _Jiritsuteki
_{FAQ page}