athena
Health Warn
- License — License: Apache-2.0
- Description — Repository has a description
- Active repo — Last push 0 days ago
- Low visibility — Only 5 GitHub stars
Code Pass
- Code scan — Scanned 12 files during light audit, no dangerous patterns found
Permissions Pass
- Permissions — No dangerous permissions requested
No AI report is available for this listing yet.
Deterministic Git & AST change intelligence engine and Model Context Protocol (MCP) server. Detects breaking changes, maps caller blast radius, identifies doc drift, and drafts evidence-grounded release notes with optional local LLM acceleration.
Athena (athena-mcp)
Local-First, Evidence-Backed Git & AST Release-Readiness and Change-Intelligence MCP Server
Athena is an open-source, local-first Model Context Protocol (MCP) server and developer CLI that delivers verifiable, ground-truth code-change intelligence. By combining deterministic Git diff analysis, Python Abstract Syntax Tree (ast) and TypeScript/JavaScript structural signature comparison, downstream caller blast-radius tracing, and documentation-drift detection, Athena eliminates the hallucination and superficiality of conventional AI release-note generators.
Key Differentiators
- Deterministic-First Core (100% LLM-Independent): Athena works out-of-the box completely offline without Ollama, without an external AI API, and without any network connection. All Git inspections, AST symbol deltas, caller blast-radius graphs, doc-drift warnings, and release notes execute deterministically with 0ms LLM wait time.
- Ground-Truth Evidence Citations: Every reported change story and potential breaking change is bound to verified
EvidenceRecorditems containing the 40-character commit SHA, commit subject, file path, exact base/target line spans, unified diff hunk header/excerpt, and confidence rating. - AST Contract Differencing & Recursive Function Defect Analysis: Athena ignores misleading Conventional Commit prefixes (e.g.
fix:commit messages that actually introduce new public methods or break signatures). It analyzes parameter defaults, positional-parameter index shifts (BUG-030),None-fallback body guards (griffe#469), return-type annotations,(str, Enum)<->StrEnummigrations (griffe#501), Enum members, public constants, newly raised exceptions (griffe#498), and recursive function bugs (broken recursive self-call arity, unconditional infinite recursion without a base case, unqualified class-method self-recursionNameErrors, and self-recursive caller blast radius). - Public API Exposure Differentiation: Automatically distinguishes symbols exported via
__all__, package__init__.pyre-exports, or TypeScriptexportdeclarations (is_public_api_export=True) from internal_privatehelper modules (is_public_api_export=False) so teams know when external package consumers are at risk even if internal callers look clean. - Unresolved Dynamic Dispatch Detection & Coverage Confidence Score: Scans repository ASTs for reflection and dynamic invocation patterns (
getattr,hasattr,setattr,globals(),locals(),eval,exec,importlib.import_module, and TypeScript bracket dispatchobj[key]()) and emits an explicitAnalysisCoverageReport(static_parse_coverage_pct,unresolved_dynamic_calls_count,overall_blast_radius_confidence) so CI and PR reviewers never get a false sense of security. - Three Resilient Operating Modes (Supporting
gemma:latest/ Gemma 7B &qwen2.5-coder:7b):- Mode A (Deterministic-only, Default): High-speed, 100% offline rule-based engine.
- Mode B (Optional Local AI Enhancement): When requested (
--llm,use_llm=True, orATHENA_ENABLE_LLM=1), local Ollama (gemma:latest/gemma:7borqwen2.5-coder:7b) enhances narrative explanations while strictly verifying evidence citations against deterministic ground truth. Includes automatic VRAM swap eviction for 6 GB GPUs. - Mode C (Fault-Tolerant Fallback): If Ollama times out, returns malformed JSON, or is offline, Athena gracefully degrades to deterministic output with clear diagnostic warnings.
- Drop-In GitHub Actions PR Audit Bot (
.github/workflows/athena-audit.yml): Ships with a ready-to-use GitHub Actions workflow that runs the multi-Python test matrix and posts a structured Breaking-Change, Blast-Radius & Release Notes summary table directly to$GITHUB_STEP_SUMMARYon every Pull Request. - Private & Read-Only: Athena never modifies your repository, never pushes code, and runs 100% locally on consumer hardware (verified on an NVIDIA RTX 4050 Laptop GPU with 6 GB VRAM and 16 GB RAM).
Architecture
flowchart TD
subgraph Client["MCP Clients & CLI"]
CLI["Athena CLI\n(athena / python -m athena.cli)"]
IDE["MCP Clients\n(Claude Desktop, Cursor, VS Code)"]
end
subgraph Server["Athena MCP Server (stdio)"]
MCP["athena.mcp_server\n(Tools & Schema Validation)"]
SEC["Security & Redaction Gate\n- Path Jail & Symlink Defense\n- CLI Flag Injection Defense\n- Bearer / Key Redaction"]
end
subgraph Pipeline["Athena Deterministic Analysis Pipeline (Mode A)"]
GIT["Git Inspector\n(Bounded Diff, Partial Clone Batching, Commits)"]
AST["AST & Structural Analyzer\n- Python ast Contract & Recursion Differ\n- TypeScript/JS Structural Scanner\n- Public API & None-Guard Analysis"]
BLAST["Blast-Radius & Caller Engine\n(Self-Recursion, Static Import, Arity & Dynamic Call Tracing)"]
DRIFT["Doc-Drift Scanner\n(Markdown Stale Symbol & Call-Arity Extraction)"]
VERIFY["Evidence & Coverage Verifier\n(SHA, Line Span, Parse Coverage & Confidence)"]
end
subgraph OptionalAI["Optional Local AI Enhancement (Mode B / Mode C)"]
OLLAMA["Local Ollama Service\n(gemma:latest 7B / qwen2.5-coder:7b)"]
FALLBACK["Graceful Degrader\n(Timeout / Malformed JSON / Offline)"]
end
subgraph Output["Evidence-Backed Results"]
REPORT["Audience-Tailored Release Notes\n(Developers, Maintainers, Clients)"]
BREAK["Potential Breaking Changes\n(Recursion Bugs, Public Exposure, Callers)"]
end
CLI --> SEC
IDE --> MCP
MCP --> SEC
SEC --> GIT
GIT --> AST
AST --> BLAST
BLAST --> DRIFT
DRIFT --> VERIFY
VERIFY -->|Default: Mode A| Output
VERIFY -->|Optional --llm: Mode B| OLLAMA
OLLAMA -->|Enhanced Output| Output
OLLAMA -.->|Failure / Timeout| FALLBACK
FALLBACK -->|Degraded Mode C| Output
Installation & Setup
Prerequisites
- Python:
>= 3.12(verified on Python3.12,3.13, and3.14) - Git:
>= 2.30(verified ongit 2.53) - (Optional) Ollama: Local Ollama server at
http://127.0.0.1:11434withgemma:latest(Gemma 7B,a72c7f4d0a15, ~5.0 GB) and/orqwen2.5-coder:7b(dae161e27b0e, ~4.7 GB) if Mode B is desired.
Portable Installation
Clone the repository and install Athena in a clean virtual environment:
# 1. Clone repository
git clone https://github.com/aaryanrwt/athena.git
cd athena
# 2. Create and activate virtual environment
python -m venv .venv
# On Linux / macOS:
source .venv/bin/activate
# On Windows (PowerShell):
.venv\Scripts\Activate.ps1
# 3. Install in editable development mode
pip install -e .
MCP Server Configuration
Athena communicates via standard JSON-RPC 2.0 over standard I/O (stdio).
Claude Desktop (claude_desktop_config.json)
Add Athena to your Claude Desktop configuration file:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json
{
"mcpServers": {
"athena": {
"command": "athena",
"args": ["mcp"],
"env": {
"ATHENA_ENABLE_LLM": "0"
}
}
}
}
Note: To enable optional local Ollama enhancement, set "ATHENA_ENABLE_LLM": "1".
Cursor / VS Code / Other MCP Clients
For generic MCP clients using executable commands:
{
"athena": {
"command": "python",
"args": ["-m", "athena.cli", "mcp"]
}
}
Registered MCP Tools
Athena exposes 4 read-only, schema-validated MCP tools:
| MCP Tool Name | Primary Parameters | Description |
|---|---|---|
inspect_repository |
repo_path (str), max_recent_commits (int, default 10) |
Safely inspects branch status, tags, HEAD commit, clean status, and recent commits. |
analyze_git_history |
repo_path (str), base_ref (str, default "AUTO"), target_ref (str, default "HEAD"), max_commits (int, default 50), use_llm (bool, default False) |
Extracts bounded revision range, commit history, unified diffs, and AST symbol deltas into grounded change stories. |
detect_potential_breaking_changes |
repo_path (str), base_ref (str, default "AUTO"), target_ref (str, default "HEAD"), include_low_confidence (bool, default False), use_llm (bool, default False) |
Detects signature mutations, removed symbols, newly raised exceptions, base class changes, affected downstream callers, and documentation drift. |
generate_release_notes |
repo_path (str), base_ref (str, default "AUTO"), target_ref (str, default "HEAD"), release_version (str, default "Unreleased"), audience (str: end_users | developers | maintainers | clients), output_file (str, optional), overwrite (bool, default False), use_llm (bool, default False) |
Synthesizes audience-tailored Markdown release notes with evidence citations, upgrade considerations, and potential breaking changes. |
CLI Usage & Verified Execution Output
The Athena CLI provides instant, zero-LLM deterministic analysis directly from your terminal. All examples below are executed directly against examples/demo_repo, a realistic Git repository with tags v0.1.0 and v0.2.0 (generated via python examples/create_demo_repo.py).
To run from source:
# 1. Generate the local demo Git repository (for testing):
python examples/create_demo_repo.py
# 2. Run Athena CLI (if installed via pip / editable):
athena <subcommand> [options]
# Or directly with Python:
python -m athena.cli <subcommand> [options]
1. Inspect Local Git Repository State
Command:
athena inspect examples/demo_repo
Verified Terminal Output:
{
"status": "ok",
"repo_path": "/path/to/athena/examples/demo_repo",
"is_valid_git_repo": true,
"is_empty_repo": false,
"is_shallow_repo": false,
"current_branch": "master",
"head_commit": {
"commit_hash": "48e9afc3a98e8912c5a4f8ffd530f55957fbf533",
"short_hash": "48e9afc",
"author_name": "Athena Demo",
"author_email": "[email protected]",
"authored_date": "2026-10-17T00:02:28+05:30",
"subject": "fix(orders): validate non-empty user_id in checkout_order",
"changed_files": [
"poetry.lock",
"src/orders.py"
],
"insertions": 185,
"deletions": 3,
"conventional_type": "fix",
"conventional_scope": "orders",
"declares_breaking_in_message": false
},
"branches": ["master"],
"tags": ["v0.1.0", "v0.2.0"],
"working_tree_clean": true,
"staged_files": [],
"unstaged_files": [],
"untracked_files": [],
"recent_commits": [
{
"commit_hash": "48e9afc3a98e8912c5a4f8ffd530f55957fbf533",
"short_hash": "48e9afc",
"subject": "fix(orders): validate non-empty user_id in checkout_order"
},
{
"commit_hash": "0ebf335ef6880c0f93f692df5b8efd783a6a0039",
"short_hash": "0ebf335",
"subject": "fix: minor cleanup in billing"
},
{
"commit_hash": "570fc967a42f552b003500886c7f1b7b293403f5",
"short_hash": "570fc96",
"subject": "feat: initial billing and order processing baseline"
}
]
}
2. Analyze Git History & Deterministic AST Symbol Changes
Command:
athena analyze examples/demo_repo --base v0.1.0 --target v0.2.0
Verified Terminal Output (Excerpt):
{
"status": "ok",
"base_ref": "v0.1.0",
"target_ref": "v0.2.0",
"resolved_base_sha": "570fc967a42f552b003500886c7f1b7b293403f5",
"resolved_target_sha": "48e9afc3a98e8912c5a4f8ffd530f55957fbf533",
"llm_backend_status": "deterministic_engine",
"total_commits": 2,
"total_changed_files": 3,
"total_diff_hunks": 2,
"stories": [
{
"story_id": "story_001",
"title": "Remove `cancel_legacy_invoice` from `src/billing.py`",
"category": "breaking_change",
"confidence": "high",
"observed_facts": [
"In 'src/billing.py', symbol 'cancel_legacy_invoice' (function) underwent 'removed'.",
"Public function 'cancel_legacy_invoice' was removed from src/billing.py.",
"Base signature: `def cancel_legacy_invoice(invoice_id: str) -> str`",
"Detected 1 downstream call site(s) in repository (e.g., src/orders.py:8).",
"Detected 1 potentially stale documentation reference(s) (e.g., README.md:5)."
],
"affected_files": ["src/billing.py"],
"evidence_ids": ["ev_004", "ev_002"]
},
{
"story_id": "story_002",
"title": "Add `issue_refund` in `src/billing.py`",
"category": "feature",
"confidence": "high",
"observed_facts": [
"Introduces new public capability (`issue_refund`) in `src/billing.py`.",
"File `src/billing.py` (modified): +12 insertions, -8 deletions.",
"AST symbol addition confirmed in `src/billing.py` at lines 18-19."
],
"affected_files": ["src/billing.py"],
"evidence_ids": ["ev_005", "ev_002"]
}
]
}
3. Detect Potential Breaking Changes & Caller Blast Radius
Command:
athena breaking-changes examples/demo_repo --base v0.1.0 --target v0.2.0
Verified Terminal Output:
{
"status": "ok",
"total_potential_breaking_changes": 6,
"potential_breaking_changes": [
{
"finding_id": "brk_001",
"title": "Potential Breaking Change in `cancel_legacy_invoice` (removed) — Requires Human Review",
"severity": "high",
"confidence": "high",
"affected_symbol": "cancel_legacy_invoice",
"file_path": "src/billing.py",
"change_kind": "removed",
"old_signature": "def cancel_legacy_invoice(invoice_id: str) -> str",
"new_signature": null,
"observed_facts": [
"In 'src/billing.py', symbol 'cancel_legacy_invoice' (function) underwent 'removed'.",
"Public function 'cancel_legacy_invoice' was removed from src/billing.py.",
"Base signature: `def cancel_legacy_invoice(invoice_id: str) -> str`",
"Detected 1 downstream call site(s) in repository (e.g., src/orders.py:8).",
"Detected 1 potentially stale documentation reference(s) (e.g., README.md:5)."
],
"affected_callers": [
{
"caller_file": "src/orders.py",
"caller_symbol": "cancel_legacy_invoice",
"line_number": 8,
"call_snippet": "cancel_legacy_invoice(\"inv_001\")",
"resolution_confidence": "exact_ast"
}
],
"doc_drift": [
{
"doc_file": "README.md",
"line_number": 5,
"referenced_symbol": "cancel_legacy_invoice",
"snippet": "`cancel_legacy_invoice(invoice_id)`",
"drift_reason": "Documentation in 'README.md:5' references 'cancel_legacy_invoice', whose signature/availability changed (removed: Public function 'cancel_legacy_invoice' was removed from src/billing.py.).",
"evidence_ids": ["ev_004"]
}
],
"evidence_ids": ["ev_004", "ev_002"],
"requires_human_review": true
},
{
"finding_id": "brk_003",
"title": "Potential Breaking Change in `process_payment` (signature_changed) — Requires Human Review",
"severity": "high",
"confidence": "high",
"affected_symbol": "process_payment",
"file_path": "src/billing.py",
"change_kind": "signature_changed",
"old_signature": "def process_payment(user_id: str, amount_cents: int, currency: str = 'USD') -> bool",
"new_signature": "def process_payment(user_id: str, amount_cents: int, idempotency_key: str, currency: str = 'USD') -> dict",
"observed_facts": [
"Added required parameter(s) ['idempotency_key'] without default values to 'process_payment'.",
"Detected 1 downstream call site(s) in repository (e.g., src/orders.py:6)."
],
"affected_callers": [
{
"caller_file": "src/orders.py",
"caller_symbol": "process_payment",
"line_number": 6,
"call_snippet": "res = process_payment(user_id, cents, \"USD\")",
"resolution_confidence": "exact_ast"
}
],
"evidence_ids": ["ev_006", "ev_002"],
"requires_human_review": true
}
]
}
4. Generate Evidence-Grounded Release Notes
Command:
athena release-notes examples/demo_repo --base v0.1.0 --target v0.2.0 --audience developers --markdown-only
Verified Terminal Output (Actual Markdown Rendered):
# Release Notes — Unreleased
- **Revision Range**: `v0.1.0` (`570fc967`) -> `v0.2.0` (`48e9afc3`)
- **Target Audience**: `developers`
- **Analysis Engine**: `deterministic_engine`
## Release Summary
**Developer Technical Summary**: Release `Unreleased` (`v0.1.0` -> `v0.2.0`) spans 2 commit(s) across 3 changed file(s), synthesized into 7 evidence-backed change stories and 6 potential breaking change(s) requiring human review.
## Potential Breaking Changes (Requires Review)
- **[Requires Human Review | severity=high | confidence=high] Potential Breaking Change in `cancel_legacy_invoice` (removed) — Requires Human Review** (`def cancel_legacy_invoice(invoice_id: str) -> str` -> `removed`): Deterministic AST analysis flagged `cancel_legacy_invoice` in `src/billing.py` as a potential breaking change (Public function 'cancel_legacy_invoice' was removed from src/billing.py.). Review callers and migration notes before release. | Affected callers (1): `src/orders.py:8` | Stale docs: `README.md:5` — `0ebf335e` `src/billing.py:11-12` [ev_004], `0ebf335e` `src/billing.py:1-19` [ev_002]
- **[Requires Human Review | severity=medium | confidence=medium] Potential Breaking Change in `InvoiceStatus` (base_class_changed) — Requires Human Review** (`class InvoiceStatus(str, Enum)` -> `class InvoiceStatus(Enum)`): Deterministic AST analysis flagged `InvoiceStatus` in `src/billing.py` as a potential breaking change (Class 'InvoiceStatus' base classes changed from ['str', 'Enum'] to ['Enum'] (potential isinstance/inheritance break).). Review callers and migration notes before release. — `0ebf335e` `src/billing.py:3-5` [ev_007], `0ebf335e` `src/billing.py:1-19` [ev_002]
- **[Requires Human Review | severity=high | confidence=high] Potential Breaking Change in `process_payment` (signature_changed) — Requires Human Review** (`def process_payment(user_id: str, amount_cents: int, currency: str = 'USD') -> bool` -> `def process_payment(user_id: str, amount_cents: int, idempotency_key: str, currency: str = 'USD') -> dict`): Deterministic AST analysis flagged `process_payment` in `src/billing.py` as a potential breaking change (Added required parameter(s) ['idempotency_key'] without default values to 'process_payment'.). Review callers and migration notes before release. | Affected callers (1): `src/orders.py:6` | Stale docs: `README.md:4` — `0ebf335e` `src/billing.py:7-11` [ev_008], `0ebf335e` `src/billing.py:1-19` [ev_002]
- **[Requires Human Review | severity=medium | confidence=medium] Potential Breaking Change in `process_payment` (return_type_changed) — Requires Human Review** (`def process_payment(user_id: str, amount_cents: int, currency: str = 'USD') -> bool` -> `def process_payment(user_id: str, amount_cents: int, idempotency_key: str, currency: str = 'USD') -> dict`): Deterministic AST analysis flagged `process_payment` in `src/billing.py` as a potential breaking change (Return type annotation of 'process_payment' changed from 'bool' to 'dict'.). Review callers and migration notes before release. | Affected callers (1): `src/orders.py:6` | Stale docs: `README.md:4` — `0ebf335e` `src/billing.py:7-11` [ev_009], `0ebf335e` `src/billing.py:1-19` [ev_002]
## New Features
- **Add `issue_refund` in `src/billing.py`** (`confidence=high`): Introduces new public capability (`issue_refund`) in `src/billing.py`. Although commit message used prefix 'fix' ('fix: minor cleanup in billing'), AST diff confirms new public symbol(s) `issue_refund` were added. *(Observed: File `src/billing.py` (modified): +12 insertions, -8 deletions.)* — `0ebf335e` `src/billing.py:18-19` [ev_005], `0ebf335e` `src/billing.py:1-19` [ev_002]
## Upgrade Considerations
- **`cancel_legacy_invoice`**: Requires human review: External callers or plugins invoking the previous signature may fail at runtime with TypeError, AttributeError, or unexpected behavior.
- **`process_payment`**: Requires human review: Parameter `idempotency_key` is now required without a default.
- **Documentation Drift (`README.md:5`)**: Documentation in 'README.md:5' references 'cancel_legacy_invoice', whose signature/availability changed (removed: Public function 'cancel_legacy_invoice' was removed from src/billing.py.).
- **Documentation Drift (`README.md:4`)**: Documentation in 'README.md:4' references 'process_payment', whose signature/availability changed (signature_changed: Added required parameter(s) ['idempotency_key'] without default values to 'process_payment'.).
Public Repository Evaluation & Benchmark Results
Athena includes an automated, reproducible evaluation harness (tests/evaluation/run_evaluation.py) cataloging 176 public repositories across technology leaders (Meta, Microsoft, Hugging Face, Vercel, Google, Ollama, Apple, NVIDIA, AWS), financial institutions (Goldman Sachs, FINOS), and top open-source ecosystems totaling 5,367,030 (5,000k+) cumulative GitHub stars (including 101 repositories with >= 10,000+ stars and active daily maintainers).
Multi-Commit (50+ Commits) & Full MCP Tool Suite Validation
Following the resolution of single-commit and partial-clone evaluation defects (BUG-001, BUG-028, BUG-029), Athena's evaluation harness enforces deep historical reachability and full 4-tool MCP verification:
- Clones repositories into
tests/evaluation/cachewith depth >= 75 (--clone-depth 75,--filter=blob:none). - Verifies commit reachability with
git rev-list --count HEADand selectsHEAD~50..HEAD. - Runs all 4 MCP server tools (
inspect_repository,analyze_git_history,detect_potential_breaking_changes, andgenerate_release_notes), plus Python recursive function AST scanning and live local Ollamagemma:latest(Gemma 7B) /qwen2.5-coder:7bsynthesis.
Summary Metrics Across 85 Cloned & Evaluated Repositories (4,398,500+ Passed Stars)
- Manifest Corpus:
176repositories (5,367,030= 5,367k+ cumulative GitHub stars,101with>= 10,000stars) - Total Repositories Cloned & Evaluated via MCP:
85(82with>= 10,000+GitHub stars) - Passed Deep-History (50+ Commits) & 4/4 MCP Tool Suite Evaluation:
81 / 85(95.3%overall;98.8%of reachable 50+ commit repos) - Cumulative GitHub Stars of Passed Repositories:
4,398,500stars - Total Commits Inspected Across Corpus:
4,050commits - Total Diff Hunks Processed:
13,817diff hunks - Total Recursive Python Functions Analyzed:
44recursive functions
Representative Sample from the 81 Passed Repositories
| Organization / Repository | Primary Language | Stars | Range Commits | Inspected Commits | Files Changed | Diff Hunks | Structural Files | Rec Fns | Potential Breaks | MCP Tools | Duration |
|---|---|---|---|---|---|---|---|---|---|---|---|
facebook/react |
JavaScript | 230,000 | 50 | 50 | 403 | 51 | 299 | 0 | 0 | 4/4 OK | 90.06s |
microsoft/vscode |
TypeScript | 165,000 | 50 | 50 | 245 | 45 | 226 | 0 | 32 | 4/4 OK | 26.75s |
huggingface/transformers |
Python | 133,000 | 50 | 50 | 295 | 22 | 254 | 2 | 0 | 4/4 OK | 38.81s |
vercel/next.js |
JavaScript | 126,000 | 50 | 50 | 476 | 48 | 377 | 0 | 5 | 4/4 OK | 41.35s |
golang/go |
Go | 124,000 | 50 | 50 | 148 | 117 | 0 | 0 | 0 | 4/4 OK (ollama:gemma:latest) |
62.43s |
ollama/ollama |
Go | 115,000 | 119 | 50 | 206 | 60 | 4 | 0 | 3 | 4/4 OK | 65.96s |
nodejs/node |
JavaScript | 108,000 | 50 | 50 | 1,741 | 32 | 472 | 0 | 0 | 4/4 OK | 21.55s |
pytorch/pytorch |
Python | 86,000 | 50 | 50 | 147 | 59 | 95 | 0 | 149 | 4/4 OK | 20.98s |
django/django |
Python | 80,000 | 50 | 50 | 90 | 42 | 48 | 5 | 0 | 4/4 OK | 86.32s |
tiangolo/fastapi |
Python | 78,000 | 50 | 50 | 57 | 50 | 14 | 2 | 0 | 4/4 OK | 69.82s |
pallets/flask |
Python | 68,000 | 120 | 50 | 81 | 258 | 30 | 0 | 30 | 4/4 OK | 75.12s |
ansible/ansible |
Python | 62,000 | 50 | 50 | 414 | 62 | 146 | 3 | 2 | 4/4 OK | 79.95s |
scikit-learn/scikit-learn |
Python | 59,000 | 50 | 50 | 327 | 50 | 235 | 0 | 9 | 4/4 OK | 108.24s |
psf/requests |
Python | 52,000 | 50 | 50 | 34 | 88 | 15 | 0 | 2 | 4/4 OK | 34.12s |
pandas-dev/pandas |
Python | 43,000 | 50 | 50 | 122 | 53 | 98 | 2 | 0 | 4/4 OK | 120.56s |
vllm-project/vllm |
Python | 38,000 | 50 | 50 | 354 | 78 | 121 | 0 | 107 | 4/4 OK | 20.81s |
google/jax |
Python | 31,000 | 59 | 50 | 98 | 113 | 73 | 5 | 5 | 4/4 OK | 164.50s |
celery/celery |
Python | 24,500 | 50 | 50 | 87 | 86 | 62 | 3 | 2 | 4/4 OK | 81.88s |
pallets/click |
Python | 15,600 | 244 | 50 | 93 | 391 | 58 | 1 | 61 | 4/4 OK | 73.21s |
encode/httpx |
Python | 13,500 | 50 | 50 | 49 | 156 | 30 | 1 | 48 | 4/4 OK | 64.10s |
goldmansachs/gs-quant |
Python | 2,900 | 50 | 50 | 502 | 1,798 | 314 | 5 | 143 | 4/4 OK | 193.14s |
NVIDIA/cuda-python |
Python | 1,400 | 50 | 50 | 436 | 1,113 | 178 | 0 | 61 | 4/4 OK | 132.84s |
Complete 85-repository table is preserved in tests/evaluation/results/EVALUATION_REPORT.md and machine-readable data in tests/evaluation/results/evaluation_results.json.
Ground-Truth Semantic Quality Benchmark (19 Scenarios — 100.0% Pass Rate)
Athena was rigorously evaluated against a dedicated 19-scenario ground-truth contract & recursion mutation benchmark (tests/evaluation/run_quality_benchmark.py). 19 / 19 scenarios passed (100.0% accuracy) across Mode A (Deterministic) and Mode B (both ollama:qwen2.5-coder:7b and ollama:gemma:latest):
| Scenario ID | Test Scenario | Ground Truth Contract Change | Mode A & B Detection | Pass / Fail | Evidence / Observed Facts |
|---|---|---|---|---|---|
SCENARIO-01 |
Function Rename | calculate_tax renamed to compute_tax |
Flagged as removed public symbol | PASS | calculate_tax flagged (severity=high) |
SCENARIO-02 |
Added Required Parameter | Added timeout: int without default |
Flagged as breaking signature mutation | PASS | Exact parameter diff tracked |
SCENARIO-03 |
Parameter Removed | Removed timeout: int without kwargs |
Flagged as breaking signature mutation | PASS | Parameter removal identified |
SCENARIO-04 |
Added Optional Parameter | Added limit: int = 100 with default |
Exempted as backward-compatible | PASS | is_breaking_candidate=False |
SCENARIO-05 |
Unguarded default=None |
Default changed to None without guard |
Flagged as breaking (TypeError risk) |
PASS | Body AST verified lack of guard |
SCENARIO-06 |
Guarded default=None |
Default changed to None with if val is None: |
Exempted as non-breaking (griffe#469) |
PASS | Body fallback guard detected |
SCENARIO-07 |
Return Type Changed | Return annotation int -> str |
Flagged as return type mutation | PASS | Type annotation diff verified |
SCENARIO-08 |
Newly Raised Exception | Function now raises ValueError |
Flagged as new exception path (griffe#498) |
PASS | AST raise node detected |
SCENARIO-09 |
Base Class Mutation | Base class added required method parameter | Flagged as inheritance contract break | PASS | Subclass override risk flagged |
SCENARIO-10 |
Harmless Refactor | Local body rewrite with identical signature | Zero breaking changes reported | PASS | 0 false positives |
SCENARIO-11 |
Caller Blast Radius | Cross-file function signature mutation | Call sites mapped in caller_a.py & caller_b.py |
PASS | Downstream file & line spans traced |
SCENARIO-12 |
Lockfile & Doc Drift | Bumps poetry.lock + stale :param: docstring |
Lockfile ignored; phantom parameter flagged | PASS | Doc drift flagged, 0 false API breaks |
SCENARIO-13 |
Public API Exposure Flag | __all__ / __init__.py export vs _private.py |
is_public_api_export=True vs False |
PASS | External consumer risk differentiated |
SCENARIO-14 |
Dynamic Call & Coverage | getattr, importlib, globals() dispatch |
3 unresolved dynamic calls flagged; confidence=medium |
PASS | Coverage 100.0%, dynamic blindspots surfaced |
SCENARIO-15 |
StrEnum & Enum Members |
(str, Enum) -> StrEnum (griffe#501) + removed member |
Both base semantic break & removed member flagged | PASS | Enum member & formatting contract verified |
SCENARIO-16 |
.gitconfig & qwen2.5-coder:7b |
diff.noprefix=true + stale README call + qwen2.5-coder:7b |
Mode A & Mode B (ollama:qwen2.5-coder:7b) both pass |
PASS | .gitconfig spoofing immune + doc arity drift caught |
SCENARIO-17 |
Recursive Function Arity Break | Recursive function adds required param, omits in self-call | Flagged in breaking findings & affected_callers (BUG-021) |
PASS | Self-recursive call arity mismatch caught |
SCENARIO-18 |
Infinite Recursion & Method Call | Missing base case & unqualified step() in class method |
Flagged RecursionError & NameError defects (BUG-022) |
PASS | AST control-flow & method scope verified |
SCENARIO-19 |
Ollama gemma:latest (Gemma 7B) |
Hybrid recursive diff synthesis with gemma:7b alias |
ollama:gemma:latest verified with 0 warnings (BUG-025) |
PASS | Verified against ev_001 & src/recursion.py |
Complete benchmark report is documented in tests/evaluation/results/QUALITY_BENCHMARK_REPORT.md.
Formal Defect Reports & Research Audit
- Bug Reports (BUG-001 through BUG-029): Full root-cause analysis and resolution records for 29 isolated engineering defects (including evaluation depth, Windows process tree termination, packfile permission locks, metric conflation, unguarded parameter fallbacks, SemVer
+build metadata, NTFS Alternate Data Stream injection, Windows reserved device names, positional-only parameter markers,@staticmethodargument preservation,@propertygetter/setter coexistence, TypeScript generic functions, aliased imports,.gitconfigdiff.noprefixspoofing immunity, historical Git blob caller lookup, Markdown call-arity drift in modified docs,StrEnummigration & Enum member diffing, body-only function mutation test-gap tracking, pre-filter coverage calculation on multi-language repos, self-recursive caller blindspot & broken recursive arity (BUG-021), unconditional infinite recursion & unqualified method self-calls (BUG-022), deep ASTRecursionErrorhardening (BUG-023), multi-class method short-name collision (BUG-024), local Ollamagemma:latest7B VRAM swap eviction & JSON normalization (BUG-025), MCP tool error boundaries &bool-as-intvalidation (BUG-026),pytestcache recursion isolation (BUG-027), partial-clone--numstat/-Mlazy network fetch timeouts (BUG-028), and redundantHEADblob subprocess storms (BUG-029)) are cataloged intests/evaluation/results/BUG_REPORTS.md. - Implementation & Research Audit Matrix: Full traceability from the research specification to implemented capabilities is available in
tests/evaluation/results/IMPLEMENTATION_AUDIT.md.
Tested Security Controls & Boundaries
- Path Canonicalization & Jail: All filesystem paths are resolved and validated with
is_relative_to(repo_root). Path traversal attempts (e.g.../../etc/passwdor symlink escapes) are blocked withINVALID_REPO_PATH. - Windows Device Name & ADS Guards: Output file paths are verified against Windows reserved device names (
CON,PRN,AUX,NUL,COM1-9,LPT1-9) and NTFS Alternate Data Stream colons (:) to prevent device locks and stream injection. - CLI Flag Injection &
.gitconfigSpoofing Neutralization: Revisions and arguments passed to Git subprocesses are validated to prevent flag injection attacks (blocking leading-,--upload-pack,-o,--exec), and all Git commands enforce-c diff.noprefix=false -c diff.mnemonicPrefix=false -c color.ui=never. - Safe File Writing: Release-note writing requires explicit
overwrite=Trueto prevent destructive overwriting of existing changelogs. - Pre-LLM Secret Redaction: High-entropy strings, AWS tokens (
AKIA...), GitHub personal access tokens (ghp_...), Bearer headers, and private keys are redacted prior to LLM or markdown exposure. - Prompt Injection Isolation: Repository content forwarded to optional LLMs is strictly fenced inside
<untrusted_repo_content>XML tags with explicit system instructions to ignore embedded directives.
Current Scope & Limitations
- Language AST Support: Complete AST contract and recursion defect analysis (parameter defaults, body guards, return types, exception raises, base classes, Enum members, public constants, recursive self-call arity, and infinite recursion base-case detection) is implemented for Python. TypeScript/JavaScript uses structural symbol, generic, enum, and export scanning. Other languages (C++, Java, Go, Rust) use unified diff heuristics and commit metadata.
- Inter-file Caller Resolution & Dynamic Dispatch: Static caller analysis resolves callers across local modules within the same repository using import hierarchy tracking (including aliased imports and self-recursive calls) and verifies call-site argument arity. When reflection (
getattr), dynamic imports (importlib.import_module), or bracket dispatch (handlers[action]()) are present, Athena explicitly flags them inunresolved_dynamic_callsand adjustsanalysis_coverage.overall_blast_radius_confidenceso reviewers know the exact boundary of static analysis. - Read-Only Operation: Athena does not write tags, create branches, or commit code to external repositories.
Running the Automated Test Suite
Athena features a comprehensive automated test suite covering all functional capabilities, security boundaries, MCP stdio protocol roundtrips, recursive function edge cases, and live gemma:latest (Gemma 7B) & qwen2.5-coder:7b integration (49 tests):
# Run complete test suite (test_athena_mvp.py, test_mcp_protocol.py, test_real_world_edge_cases.py)
python -m pytest -v
# Run 19-scenario ground-truth semantic quality & Gemma 7B / Qwen 7B benchmark
python tests/evaluation/run_quality_benchmark.py
# Run large-scale multi-commit & MCP repository evaluation
python tests/evaluation/run_evaluation.py --sort-by-stars --limit 85 --llm --model gemma:7b
License
Athena is licensed under the Apache License, Version 2.0.
Reviews (0)
Sign in to leave a review.
Leave a reviewNo results found