ATLAS

A failure-mode taxonomy layer for agents: reflect at meaningful boundaries, catch recurring mistakes, and learn from traces.

Paper: Adaptive Failure Taxonomies as Feedback for LLM-Agent Improvement Procedures

Documentation: multi-agent-systems-failure-taxonomy.github.io/ATLAS · Concepts · Getting started

Procedures that improve LLM agents act on the failures in execution traces: a
best-of-N selector picks the best of several attempts, a program-search loop
rewrites the agent after failed runs, and a runtime monitor reflects before an
action commits. All three need feedback that names why a trajectory failed, in
a form that aggregates across runs. Scalar rewards discard the reason. Free-form
reflections are unstructured and per-trace. Hand-authored taxonomies such as
MAST, from "Why Do Multi-Agent LLM Systems Fail?" (Cemri et al., 2025),
fix the vocabulary before observing the agent, its roles, or the target domain.

ATLAS induces the vocabulary instead. It reads a target system's own traces and
generates a compact set of evidence-grounded failure codes, with no
hand-authored codes and no per-trace annotation, then passes those codes back to
the improvement procedure alongside each trace. This repository packages that
idea as a runtime skill: it supervises an agent against a taxonomy at meaningful
boundaries, records the failures that actually occur, and generates or refines a
taxonomy specialized to your own traces. When no taxonomy is configured it starts
from a built-in 14-code adaptation of MAST.

ATLAS is not a task solver. It is a diagnostic feedback layer that gives an agent
a structured way to ask, "what mistake am I about to repeat?"

Adaptive failure taxonomies

An ATLAS taxonomy is a set of 15 to 30 failure codes induced from a system's own
traces, organized along three fixed axes. The axes follow MAST's empirical
clustering of failure modes; the concrete codes, role labels, descriptions, and
evidence patterns are induced per system:

The same induced vocabulary is a reusable feedback interface for more than one
improvement procedure. The paper,
Adaptive Failure Taxonomies as Feedback for LLM-Agent Improvement Procedures,
evaluates three:

1. Best-of-N trajectory selection. As judges on Terminal-Bench 2.0,
   ATLAS-Judge reaches 89.9% accuracy (+15 points over Pass@1) and beats judges
   that use a fixed taxonomy or none.
2. Evolutionary agent-system optimization. As mutation feedback,
   taxonomy-coded diagnoses beat free-form reflection across competitive
   programming, math, STEM QA, and discrete reasoning (OlympiadBench 87.9% to
   91.9% on a 655-problem held-out set).
3. Runtime feedback. For SWE-agent on SWE-bench Verified Mini, codes improve
   over free-form reflection in both in-prompt and external-judge use. This
   runtime setting is what the skill in this repository implements.

On TRAIL (117 expert-annotated GAIA traces), induced codes align with the human
gold at Cohen's kappa 0.725, more faithfully than TRAIL's hand-crafted
vocabulary.

How it works

1. A task starts. ATLAS selects the active taxonomy: an inherited stored
   taxonomy, or built-in MAST when none is configured.
2. At configured boundaries (checkpoints, tool failures, subagent stops), the
   agent reflects against the taxonomy and repairs when evidence demands it.
3. A final submission gate blocks completion until the reflection passes or
   retries are exhausted honestly.
4. One canonical trace is recorded at session end.
5. After enough traces, ATLAS generates a task-specific taxonomy (or refines
   the active one). Accepted taxonomies become inheritable records for future
   runs.

New to the terminology? Start with docs/CONCEPTS.md.

What it looks like

At a checkpoint, the agent reflects on its recent trajectory against the
active taxonomy in a fixed shape:

Observe:   The last two Bash runs failed with the same ImportError; no
           dependency check ran between attempts.
Correlate: Retrying an identical command without new information.
Map:       MAST-3 (Step repetition); evidence supports the match.
Decide:    One focused repair: verify the installed package version
           before the next run.

Mapping no codes ("none apply") is a valid outcome. Before the final answer is
released, a blocking gate requires the same reflection and allows a bounded
number of repairs. Everything the gates record is browsable live in the
dashboard.

A full walkthrough with dashboard screenshots is in
docs/EXAMPLE_RUN.md. Try the dashboard yourself with
python -m examples.dashboard_demo.

Install

Requirements: Python 3.10 or newer.

python -m pip install "git+https://github.com/multi-agent-systems-failure-taxonomy/ATLAS.git"

From a local checkout: python -m pip install .

ATLAS's own learning calls support Anthropic, OpenAI(-compatible), Gemini, and
AWS Bedrock model IDs. Optional extras ([anthropic], [bedrock]) and
credential setup live in docs/INSTALLATION.md.

Quick start

Create atlas.json in the project that will run the agent:

{
  "version": 1,
  "trace_output": "./atlas-program",
  "atlas_model": "gpt-5"
}

Every field, its default, and when to change it:
docs/CONFIGURATION.md.

Then choose the integration that matches your pipeline:

Check the setup:

atlas-doctor --config atlas.json

Results

Full evaluation artifacts (per-question result rows, the exact taxonomies used,
and replication steps) live in runs/:

Documentation

Main commands

Contributing

Development setup, verification commands, and package maps are in
CONTRIBUTING.md.

Related

The original ATLAS taxonomy-induction pipeline (the research code this skill
builds on) lives on the
paper-pipeline
branch and is vendored unchanged under vendor/atlas.

License

Apache-2.0. See LICENSE.