athena

mcp
Guvenlik Denetimi
Uyari
Health Uyari
  • License — License: Apache-2.0
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Low visibility — Only 5 GitHub stars
Code Gecti
  • Code scan — Scanned 12 files during light audit, no dangerous patterns found
Permissions Gecti
  • Permissions — No dangerous permissions requested

Bu listing icin henuz AI raporu yok.

SUMMARY

Deterministic Git & AST change intelligence engine and Model Context Protocol (MCP) server. Detects breaking changes, maps caller blast radius, identifies doc drift, and drafts evidence-grounded release notes with optional local LLM acceleration.

README.md

Athena — Local-First Git & AST Intelligence


Athena (athena-mcp)

M8ven Score

Local-First, Evidence-Backed Git & AST Release-Readiness and Change-Intelligence MCP Server

Athena is an open-source, local-first Model Context Protocol (MCP) server and developer CLI that delivers verifiable, ground-truth code-change intelligence. By combining deterministic Git diff analysis, Python Abstract Syntax Tree (ast) and TypeScript/JavaScript structural signature comparison, downstream caller blast-radius tracing, and documentation-drift detection, Athena eliminates the hallucination and superficiality of conventional AI release-note generators.

Athena in Action — AST & Git Change Intelligence Showcase


Key Differentiators

  1. Deterministic-First Core (100% LLM-Independent): Athena works out-of-the box completely offline without Ollama, without an external AI API, and without any network connection. All Git inspections, AST symbol deltas, caller blast-radius graphs, doc-drift warnings, and release notes execute deterministically with 0ms LLM wait time.
  2. Ground-Truth Evidence Citations: Every reported change story and potential breaking change is bound to verified EvidenceRecord items containing the 40-character commit SHA, commit subject, file path, exact base/target line spans, unified diff hunk header/excerpt, and confidence rating.
  3. AST Contract Differencing & Recursive Function Defect Analysis: Athena ignores misleading Conventional Commit prefixes (e.g. fix: commit messages that actually introduce new public methods or break signatures). It analyzes parameter defaults, positional-parameter index shifts (BUG-030), None-fallback body guards (griffe#469), return-type annotations, (str, Enum) <-> StrEnum migrations (griffe#501), Enum members, public constants, newly raised exceptions (griffe#498), and recursive function bugs (broken recursive self-call arity, unconditional infinite recursion without a base case, unqualified class-method self-recursion NameErrors, and self-recursive caller blast radius).
  4. Public API Exposure Differentiation: Automatically distinguishes symbols exported via __all__, package __init__.py re-exports, or TypeScript export declarations (is_public_api_export=True) from internal _private helper modules (is_public_api_export=False) so teams know when external package consumers are at risk even if internal callers look clean.
  5. Unresolved Dynamic Dispatch Detection & Coverage Confidence Score: Scans repository ASTs for reflection and dynamic invocation patterns (getattr, hasattr, setattr, globals(), locals(), eval, exec, importlib.import_module, and TypeScript bracket dispatch obj[key]()) and emits an explicit AnalysisCoverageReport (static_parse_coverage_pct, unresolved_dynamic_calls_count, overall_blast_radius_confidence) so CI and PR reviewers never get a false sense of security.
  6. Three Resilient Operating Modes (Supporting gemma:latest / Gemma 7B & qwen2.5-coder:7b):
    • Mode A (Deterministic-only, Default): High-speed, 100% offline rule-based engine.
    • Mode B (Optional Local AI Enhancement): When requested (--llm, use_llm=True, or ATHENA_ENABLE_LLM=1), local Ollama (gemma:latest / gemma:7b or qwen2.5-coder:7b) enhances narrative explanations while strictly verifying evidence citations against deterministic ground truth. Includes automatic VRAM swap eviction for 6 GB GPUs.
    • Mode C (Fault-Tolerant Fallback): If Ollama times out, returns malformed JSON, or is offline, Athena gracefully degrades to deterministic output with clear diagnostic warnings.
  7. Drop-In GitHub Actions PR Audit Bot (.github/workflows/athena-audit.yml): Ships with a ready-to-use GitHub Actions workflow that runs the multi-Python test matrix and posts a structured Breaking-Change, Blast-Radius & Release Notes summary table directly to $GITHUB_STEP_SUMMARY on every Pull Request.
  8. Private & Read-Only: Athena never modifies your repository, never pushes code, and runs 100% locally on consumer hardware (verified on an NVIDIA RTX 4050 Laptop GPU with 6 GB VRAM and 16 GB RAM).

Architecture

flowchart TD
    subgraph Client["MCP Clients & CLI"]
        CLI["Athena CLI\n(athena / python -m athena.cli)"]
        IDE["MCP Clients\n(Claude Desktop, Cursor, VS Code)"]
    end

    subgraph Server["Athena MCP Server (stdio)"]
        MCP["athena.mcp_server\n(Tools & Schema Validation)"]
        SEC["Security & Redaction Gate\n- Path Jail & Symlink Defense\n- CLI Flag Injection Defense\n- Bearer / Key Redaction"]
    end

    subgraph Pipeline["Athena Deterministic Analysis Pipeline (Mode A)"]
        GIT["Git Inspector\n(Bounded Diff, Partial Clone Batching, Commits)"]
        AST["AST & Structural Analyzer\n- Python ast Contract & Recursion Differ\n- TypeScript/JS Structural Scanner\n- Public API & None-Guard Analysis"]
        BLAST["Blast-Radius & Caller Engine\n(Self-Recursion, Static Import, Arity & Dynamic Call Tracing)"]
        DRIFT["Doc-Drift Scanner\n(Markdown Stale Symbol & Call-Arity Extraction)"]
        VERIFY["Evidence & Coverage Verifier\n(SHA, Line Span, Parse Coverage & Confidence)"]
    end

    subgraph OptionalAI["Optional Local AI Enhancement (Mode B / Mode C)"]
        OLLAMA["Local Ollama Service\n(gemma:latest 7B / qwen2.5-coder:7b)"]
        FALLBACK["Graceful Degrader\n(Timeout / Malformed JSON / Offline)"]
    end

    subgraph Output["Evidence-Backed Results"]
        REPORT["Audience-Tailored Release Notes\n(Developers, Maintainers, Clients)"]
        BREAK["Potential Breaking Changes\n(Recursion Bugs, Public Exposure, Callers)"]
    end

    CLI --> SEC
    IDE --> MCP
    MCP --> SEC
    SEC --> GIT
    GIT --> AST
    AST --> BLAST
    BLAST --> DRIFT
    DRIFT --> VERIFY

    VERIFY -->|Default: Mode A| Output
    VERIFY -->|Optional --llm: Mode B| OLLAMA
    OLLAMA -->|Enhanced Output| Output
    OLLAMA -.->|Failure / Timeout| FALLBACK
    FALLBACK -->|Degraded Mode C| Output

Installation & Setup

Prerequisites

  • Python: >= 3.12 (verified on Python 3.12, 3.13, and 3.14)
  • Git: >= 2.30 (verified on git 2.53)
  • (Optional) Ollama: Local Ollama server at http://127.0.0.1:11434 with gemma:latest (Gemma 7B, a72c7f4d0a15, ~5.0 GB) and/or qwen2.5-coder:7b (dae161e27b0e, ~4.7 GB) if Mode B is desired.

Portable Installation

Clone the repository and install Athena in a clean virtual environment:

# 1. Clone repository
git clone https://github.com/aaryanrwt/athena.git
cd athena

# 2. Create and activate virtual environment
python -m venv .venv

# On Linux / macOS:
source .venv/bin/activate

# On Windows (PowerShell):
.venv\Scripts\Activate.ps1

# 3. Install in editable development mode
pip install -e .

MCP Server Configuration

Athena communicates via standard JSON-RPC 2.0 over standard I/O (stdio).

Claude Desktop (claude_desktop_config.json)

Add Athena to your Claude Desktop configuration file:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
  • Windows: %APPDATA%\Claude\claude_desktop_config.json
{
  "mcpServers": {
    "athena": {
      "command": "athena",
      "args": ["mcp"],
      "env": {
        "ATHENA_ENABLE_LLM": "0"
      }
    }
  }
}

Note: To enable optional local Ollama enhancement, set "ATHENA_ENABLE_LLM": "1".

Cursor / VS Code / Other MCP Clients

For generic MCP clients using executable commands:

{
  "athena": {
    "command": "python",
    "args": ["-m", "athena.cli", "mcp"]
  }
}

Registered MCP Tools

Athena exposes 4 read-only, schema-validated MCP tools:

MCP Tool Name Primary Parameters Description
inspect_repository repo_path (str), max_recent_commits (int, default 10) Safely inspects branch status, tags, HEAD commit, clean status, and recent commits.
analyze_git_history repo_path (str), base_ref (str, default "AUTO"), target_ref (str, default "HEAD"), max_commits (int, default 50), use_llm (bool, default False) Extracts bounded revision range, commit history, unified diffs, and AST symbol deltas into grounded change stories.
detect_potential_breaking_changes repo_path (str), base_ref (str, default "AUTO"), target_ref (str, default "HEAD"), include_low_confidence (bool, default False), use_llm (bool, default False) Detects signature mutations, removed symbols, newly raised exceptions, base class changes, affected downstream callers, and documentation drift.
generate_release_notes repo_path (str), base_ref (str, default "AUTO"), target_ref (str, default "HEAD"), release_version (str, default "Unreleased"), audience (str: end_users | developers | maintainers | clients), output_file (str, optional), overwrite (bool, default False), use_llm (bool, default False) Synthesizes audience-tailored Markdown release notes with evidence citations, upgrade considerations, and potential breaking changes.

CLI Usage & Verified Execution Output

The Athena CLI provides instant, zero-LLM deterministic analysis directly from your terminal. All examples below are executed directly against examples/demo_repo, a realistic Git repository with tags v0.1.0 and v0.2.0 (generated via python examples/create_demo_repo.py).

To run from source:

# 1. Generate the local demo Git repository (for testing):
python examples/create_demo_repo.py

# 2. Run Athena CLI (if installed via pip / editable):
athena <subcommand> [options]

# Or directly with Python:
python -m athena.cli <subcommand> [options]

1. Inspect Local Git Repository State

Command:

athena inspect examples/demo_repo

Verified Terminal Output:

{
  "status": "ok",
  "repo_path": "/path/to/athena/examples/demo_repo",
  "is_valid_git_repo": true,
  "is_empty_repo": false,
  "is_shallow_repo": false,
  "current_branch": "master",
  "head_commit": {
    "commit_hash": "48e9afc3a98e8912c5a4f8ffd530f55957fbf533",
    "short_hash": "48e9afc",
    "author_name": "Athena Demo",
    "author_email": "[email protected]",
    "authored_date": "2026-10-17T00:02:28+05:30",
    "subject": "fix(orders): validate non-empty user_id in checkout_order",
    "changed_files": [
      "poetry.lock",
      "src/orders.py"
    ],
    "insertions": 185,
    "deletions": 3,
    "conventional_type": "fix",
    "conventional_scope": "orders",
    "declares_breaking_in_message": false
  },
  "branches": ["master"],
  "tags": ["v0.1.0", "v0.2.0"],
  "working_tree_clean": true,
  "staged_files": [],
  "unstaged_files": [],
  "untracked_files": [],
  "recent_commits": [
    {
      "commit_hash": "48e9afc3a98e8912c5a4f8ffd530f55957fbf533",
      "short_hash": "48e9afc",
      "subject": "fix(orders): validate non-empty user_id in checkout_order"
    },
    {
      "commit_hash": "0ebf335ef6880c0f93f692df5b8efd783a6a0039",
      "short_hash": "0ebf335",
      "subject": "fix: minor cleanup in billing"
    },
    {
      "commit_hash": "570fc967a42f552b003500886c7f1b7b293403f5",
      "short_hash": "570fc96",
      "subject": "feat: initial billing and order processing baseline"
    }
  ]
}

2. Analyze Git History & Deterministic AST Symbol Changes

Command:

athena analyze examples/demo_repo --base v0.1.0 --target v0.2.0

Verified Terminal Output (Excerpt):

{
  "status": "ok",
  "base_ref": "v0.1.0",
  "target_ref": "v0.2.0",
  "resolved_base_sha": "570fc967a42f552b003500886c7f1b7b293403f5",
  "resolved_target_sha": "48e9afc3a98e8912c5a4f8ffd530f55957fbf533",
  "llm_backend_status": "deterministic_engine",
  "total_commits": 2,
  "total_changed_files": 3,
  "total_diff_hunks": 2,
  "stories": [
    {
      "story_id": "story_001",
      "title": "Remove `cancel_legacy_invoice` from `src/billing.py`",
      "category": "breaking_change",
      "confidence": "high",
      "observed_facts": [
        "In 'src/billing.py', symbol 'cancel_legacy_invoice' (function) underwent 'removed'.",
        "Public function 'cancel_legacy_invoice' was removed from src/billing.py.",
        "Base signature: `def cancel_legacy_invoice(invoice_id: str) -> str`",
        "Detected 1 downstream call site(s) in repository (e.g., src/orders.py:8).",
        "Detected 1 potentially stale documentation reference(s) (e.g., README.md:5)."
      ],
      "affected_files": ["src/billing.py"],
      "evidence_ids": ["ev_004", "ev_002"]
    },
    {
      "story_id": "story_002",
      "title": "Add `issue_refund` in `src/billing.py`",
      "category": "feature",
      "confidence": "high",
      "observed_facts": [
        "Introduces new public capability (`issue_refund`) in `src/billing.py`.",
        "File `src/billing.py` (modified): +12 insertions, -8 deletions.",
        "AST symbol addition confirmed in `src/billing.py` at lines 18-19."
      ],
      "affected_files": ["src/billing.py"],
      "evidence_ids": ["ev_005", "ev_002"]
    }
  ]
}

3. Detect Potential Breaking Changes & Caller Blast Radius

Command:

athena breaking-changes examples/demo_repo --base v0.1.0 --target v0.2.0

Verified Terminal Output:

{
  "status": "ok",
  "total_potential_breaking_changes": 6,
  "potential_breaking_changes": [
    {
      "finding_id": "brk_001",
      "title": "Potential Breaking Change in `cancel_legacy_invoice` (removed) — Requires Human Review",
      "severity": "high",
      "confidence": "high",
      "affected_symbol": "cancel_legacy_invoice",
      "file_path": "src/billing.py",
      "change_kind": "removed",
      "old_signature": "def cancel_legacy_invoice(invoice_id: str) -> str",
      "new_signature": null,
      "observed_facts": [
        "In 'src/billing.py', symbol 'cancel_legacy_invoice' (function) underwent 'removed'.",
        "Public function 'cancel_legacy_invoice' was removed from src/billing.py.",
        "Base signature: `def cancel_legacy_invoice(invoice_id: str) -> str`",
        "Detected 1 downstream call site(s) in repository (e.g., src/orders.py:8).",
        "Detected 1 potentially stale documentation reference(s) (e.g., README.md:5)."
      ],
      "affected_callers": [
        {
          "caller_file": "src/orders.py",
          "caller_symbol": "cancel_legacy_invoice",
          "line_number": 8,
          "call_snippet": "cancel_legacy_invoice(\"inv_001\")",
          "resolution_confidence": "exact_ast"
        }
      ],
      "doc_drift": [
        {
          "doc_file": "README.md",
          "line_number": 5,
          "referenced_symbol": "cancel_legacy_invoice",
          "snippet": "`cancel_legacy_invoice(invoice_id)`",
          "drift_reason": "Documentation in 'README.md:5' references 'cancel_legacy_invoice', whose signature/availability changed (removed: Public function 'cancel_legacy_invoice' was removed from src/billing.py.).",
          "evidence_ids": ["ev_004"]
        }
      ],
      "evidence_ids": ["ev_004", "ev_002"],
      "requires_human_review": true
    },
    {
      "finding_id": "brk_003",
      "title": "Potential Breaking Change in `process_payment` (signature_changed) — Requires Human Review",
      "severity": "high",
      "confidence": "high",
      "affected_symbol": "process_payment",
      "file_path": "src/billing.py",
      "change_kind": "signature_changed",
      "old_signature": "def process_payment(user_id: str, amount_cents: int, currency: str = 'USD') -> bool",
      "new_signature": "def process_payment(user_id: str, amount_cents: int, idempotency_key: str, currency: str = 'USD') -> dict",
      "observed_facts": [
        "Added required parameter(s) ['idempotency_key'] without default values to 'process_payment'.",
        "Detected 1 downstream call site(s) in repository (e.g., src/orders.py:6)."
      ],
      "affected_callers": [
        {
          "caller_file": "src/orders.py",
          "caller_symbol": "process_payment",
          "line_number": 6,
          "call_snippet": "res = process_payment(user_id, cents, \"USD\")",
          "resolution_confidence": "exact_ast"
        }
      ],
      "evidence_ids": ["ev_006", "ev_002"],
      "requires_human_review": true
    }
  ]
}

4. Generate Evidence-Grounded Release Notes

Command:

athena release-notes examples/demo_repo --base v0.1.0 --target v0.2.0 --audience developers --markdown-only

Verified Terminal Output (Actual Markdown Rendered):

# Release Notes — Unreleased

- **Revision Range**: `v0.1.0` (`570fc967`) -> `v0.2.0` (`48e9afc3`)
- **Target Audience**: `developers`
- **Analysis Engine**: `deterministic_engine`

## Release Summary

**Developer Technical Summary**: Release `Unreleased` (`v0.1.0` -> `v0.2.0`) spans 2 commit(s) across 3 changed file(s), synthesized into 7 evidence-backed change stories and 6 potential breaking change(s) requiring human review.

## Potential Breaking Changes (Requires Review)

- **[Requires Human Review | severity=high | confidence=high] Potential Breaking Change in `cancel_legacy_invoice` (removed) — Requires Human Review** (`def cancel_legacy_invoice(invoice_id: str) -> str` -> `removed`): Deterministic AST analysis flagged `cancel_legacy_invoice` in `src/billing.py` as a potential breaking change (Public function 'cancel_legacy_invoice' was removed from src/billing.py.). Review callers and migration notes before release. | Affected callers (1): `src/orders.py:8` | Stale docs: `README.md:5` — `0ebf335e` `src/billing.py:11-12` [ev_004], `0ebf335e` `src/billing.py:1-19` [ev_002]
- **[Requires Human Review | severity=medium | confidence=medium] Potential Breaking Change in `InvoiceStatus` (base_class_changed) — Requires Human Review** (`class InvoiceStatus(str, Enum)` -> `class InvoiceStatus(Enum)`): Deterministic AST analysis flagged `InvoiceStatus` in `src/billing.py` as a potential breaking change (Class 'InvoiceStatus' base classes changed from ['str', 'Enum'] to ['Enum'] (potential isinstance/inheritance break).). Review callers and migration notes before release. — `0ebf335e` `src/billing.py:3-5` [ev_007], `0ebf335e` `src/billing.py:1-19` [ev_002]
- **[Requires Human Review | severity=high | confidence=high] Potential Breaking Change in `process_payment` (signature_changed) — Requires Human Review** (`def process_payment(user_id: str, amount_cents: int, currency: str = 'USD') -> bool` -> `def process_payment(user_id: str, amount_cents: int, idempotency_key: str, currency: str = 'USD') -> dict`): Deterministic AST analysis flagged `process_payment` in `src/billing.py` as a potential breaking change (Added required parameter(s) ['idempotency_key'] without default values to 'process_payment'.). Review callers and migration notes before release. | Affected callers (1): `src/orders.py:6` | Stale docs: `README.md:4` — `0ebf335e` `src/billing.py:7-11` [ev_008], `0ebf335e` `src/billing.py:1-19` [ev_002]
- **[Requires Human Review | severity=medium | confidence=medium] Potential Breaking Change in `process_payment` (return_type_changed) — Requires Human Review** (`def process_payment(user_id: str, amount_cents: int, currency: str = 'USD') -> bool` -> `def process_payment(user_id: str, amount_cents: int, idempotency_key: str, currency: str = 'USD') -> dict`): Deterministic AST analysis flagged `process_payment` in `src/billing.py` as a potential breaking change (Return type annotation of 'process_payment' changed from 'bool' to 'dict'.). Review callers and migration notes before release. | Affected callers (1): `src/orders.py:6` | Stale docs: `README.md:4` — `0ebf335e` `src/billing.py:7-11` [ev_009], `0ebf335e` `src/billing.py:1-19` [ev_002]

## New Features

- **Add `issue_refund` in `src/billing.py`** (`confidence=high`): Introduces new public capability (`issue_refund`) in `src/billing.py`. Although commit message used prefix 'fix' ('fix: minor cleanup in billing'), AST diff confirms new public symbol(s) `issue_refund` were added. *(Observed: File `src/billing.py` (modified): +12 insertions, -8 deletions.)* — `0ebf335e` `src/billing.py:18-19` [ev_005], `0ebf335e` `src/billing.py:1-19` [ev_002]

## Upgrade Considerations

- **`cancel_legacy_invoice`**: Requires human review: External callers or plugins invoking the previous signature may fail at runtime with TypeError, AttributeError, or unexpected behavior.
- **`process_payment`**: Requires human review: Parameter `idempotency_key` is now required without a default.
- **Documentation Drift (`README.md:5`)**: Documentation in 'README.md:5' references 'cancel_legacy_invoice', whose signature/availability changed (removed: Public function 'cancel_legacy_invoice' was removed from src/billing.py.).
- **Documentation Drift (`README.md:4`)**: Documentation in 'README.md:4' references 'process_payment', whose signature/availability changed (signature_changed: Added required parameter(s) ['idempotency_key'] without default values to 'process_payment'.).

Public Repository Evaluation & Benchmark Results

Athena includes an automated, reproducible evaluation harness (tests/evaluation/run_evaluation.py) cataloging 176 public repositories across technology leaders (Meta, Microsoft, Hugging Face, Vercel, Google, Ollama, Apple, NVIDIA, AWS), financial institutions (Goldman Sachs, FINOS), and top open-source ecosystems totaling 5,367,030 (5,000k+) cumulative GitHub stars (including 101 repositories with >= 10,000+ stars and active daily maintainers).

Multi-Commit (50+ Commits) & Full MCP Tool Suite Validation

Following the resolution of single-commit and partial-clone evaluation defects (BUG-001, BUG-028, BUG-029), Athena's evaluation harness enforces deep historical reachability and full 4-tool MCP verification:

  1. Clones repositories into tests/evaluation/cache with depth >= 75 (--clone-depth 75, --filter=blob:none).
  2. Verifies commit reachability with git rev-list --count HEAD and selects HEAD~50..HEAD.
  3. Runs all 4 MCP server tools (inspect_repository, analyze_git_history, detect_potential_breaking_changes, and generate_release_notes), plus Python recursive function AST scanning and live local Ollama gemma:latest (Gemma 7B) / qwen2.5-coder:7b synthesis.

Summary Metrics Across 85 Cloned & Evaluated Repositories (4,398,500+ Passed Stars)

  • Manifest Corpus: 176 repositories (5,367,030 = 5,367k+ cumulative GitHub stars, 101 with >= 10,000 stars)
  • Total Repositories Cloned & Evaluated via MCP: 85 (82 with >= 10,000+ GitHub stars)
  • Passed Deep-History (50+ Commits) & 4/4 MCP Tool Suite Evaluation: 81 / 85 (95.3% overall; 98.8% of reachable 50+ commit repos)
  • Cumulative GitHub Stars of Passed Repositories: 4,398,500 stars
  • Total Commits Inspected Across Corpus: 4,050 commits
  • Total Diff Hunks Processed: 13,817 diff hunks
  • Total Recursive Python Functions Analyzed: 44 recursive functions

Representative Sample from the 81 Passed Repositories

Organization / Repository Primary Language Stars Range Commits Inspected Commits Files Changed Diff Hunks Structural Files Rec Fns Potential Breaks MCP Tools Duration
facebook/react JavaScript 230,000 50 50 403 51 299 0 0 4/4 OK 90.06s
microsoft/vscode TypeScript 165,000 50 50 245 45 226 0 32 4/4 OK 26.75s
huggingface/transformers Python 133,000 50 50 295 22 254 2 0 4/4 OK 38.81s
vercel/next.js JavaScript 126,000 50 50 476 48 377 0 5 4/4 OK 41.35s
golang/go Go 124,000 50 50 148 117 0 0 0 4/4 OK (ollama:gemma:latest) 62.43s
ollama/ollama Go 115,000 119 50 206 60 4 0 3 4/4 OK 65.96s
nodejs/node JavaScript 108,000 50 50 1,741 32 472 0 0 4/4 OK 21.55s
pytorch/pytorch Python 86,000 50 50 147 59 95 0 149 4/4 OK 20.98s
django/django Python 80,000 50 50 90 42 48 5 0 4/4 OK 86.32s
tiangolo/fastapi Python 78,000 50 50 57 50 14 2 0 4/4 OK 69.82s
pallets/flask Python 68,000 120 50 81 258 30 0 30 4/4 OK 75.12s
ansible/ansible Python 62,000 50 50 414 62 146 3 2 4/4 OK 79.95s
scikit-learn/scikit-learn Python 59,000 50 50 327 50 235 0 9 4/4 OK 108.24s
psf/requests Python 52,000 50 50 34 88 15 0 2 4/4 OK 34.12s
pandas-dev/pandas Python 43,000 50 50 122 53 98 2 0 4/4 OK 120.56s
vllm-project/vllm Python 38,000 50 50 354 78 121 0 107 4/4 OK 20.81s
google/jax Python 31,000 59 50 98 113 73 5 5 4/4 OK 164.50s
celery/celery Python 24,500 50 50 87 86 62 3 2 4/4 OK 81.88s
pallets/click Python 15,600 244 50 93 391 58 1 61 4/4 OK 73.21s
encode/httpx Python 13,500 50 50 49 156 30 1 48 4/4 OK 64.10s
goldmansachs/gs-quant Python 2,900 50 50 502 1,798 314 5 143 4/4 OK 193.14s
NVIDIA/cuda-python Python 1,400 50 50 436 1,113 178 0 61 4/4 OK 132.84s

Complete 85-repository table is preserved in tests/evaluation/results/EVALUATION_REPORT.md and machine-readable data in tests/evaluation/results/evaluation_results.json.


Ground-Truth Semantic Quality Benchmark (19 Scenarios — 100.0% Pass Rate)

Athena was rigorously evaluated against a dedicated 19-scenario ground-truth contract & recursion mutation benchmark (tests/evaluation/run_quality_benchmark.py). 19 / 19 scenarios passed (100.0% accuracy) across Mode A (Deterministic) and Mode B (both ollama:qwen2.5-coder:7b and ollama:gemma:latest):

Scenario ID Test Scenario Ground Truth Contract Change Mode A & B Detection Pass / Fail Evidence / Observed Facts
SCENARIO-01 Function Rename calculate_tax renamed to compute_tax Flagged as removed public symbol PASS calculate_tax flagged (severity=high)
SCENARIO-02 Added Required Parameter Added timeout: int without default Flagged as breaking signature mutation PASS Exact parameter diff tracked
SCENARIO-03 Parameter Removed Removed timeout: int without kwargs Flagged as breaking signature mutation PASS Parameter removal identified
SCENARIO-04 Added Optional Parameter Added limit: int = 100 with default Exempted as backward-compatible PASS is_breaking_candidate=False
SCENARIO-05 Unguarded default=None Default changed to None without guard Flagged as breaking (TypeError risk) PASS Body AST verified lack of guard
SCENARIO-06 Guarded default=None Default changed to None with if val is None: Exempted as non-breaking (griffe#469) PASS Body fallback guard detected
SCENARIO-07 Return Type Changed Return annotation int -> str Flagged as return type mutation PASS Type annotation diff verified
SCENARIO-08 Newly Raised Exception Function now raises ValueError Flagged as new exception path (griffe#498) PASS AST raise node detected
SCENARIO-09 Base Class Mutation Base class added required method parameter Flagged as inheritance contract break PASS Subclass override risk flagged
SCENARIO-10 Harmless Refactor Local body rewrite with identical signature Zero breaking changes reported PASS 0 false positives
SCENARIO-11 Caller Blast Radius Cross-file function signature mutation Call sites mapped in caller_a.py & caller_b.py PASS Downstream file & line spans traced
SCENARIO-12 Lockfile & Doc Drift Bumps poetry.lock + stale :param: docstring Lockfile ignored; phantom parameter flagged PASS Doc drift flagged, 0 false API breaks
SCENARIO-13 Public API Exposure Flag __all__ / __init__.py export vs _private.py is_public_api_export=True vs False PASS External consumer risk differentiated
SCENARIO-14 Dynamic Call & Coverage getattr, importlib, globals() dispatch 3 unresolved dynamic calls flagged; confidence=medium PASS Coverage 100.0%, dynamic blindspots surfaced
SCENARIO-15 StrEnum & Enum Members (str, Enum) -> StrEnum (griffe#501) + removed member Both base semantic break & removed member flagged PASS Enum member & formatting contract verified
SCENARIO-16 .gitconfig & qwen2.5-coder:7b diff.noprefix=true + stale README call + qwen2.5-coder:7b Mode A & Mode B (ollama:qwen2.5-coder:7b) both pass PASS .gitconfig spoofing immune + doc arity drift caught
SCENARIO-17 Recursive Function Arity Break Recursive function adds required param, omits in self-call Flagged in breaking findings & affected_callers (BUG-021) PASS Self-recursive call arity mismatch caught
SCENARIO-18 Infinite Recursion & Method Call Missing base case & unqualified step() in class method Flagged RecursionError & NameError defects (BUG-022) PASS AST control-flow & method scope verified
SCENARIO-19 Ollama gemma:latest (Gemma 7B) Hybrid recursive diff synthesis with gemma:7b alias ollama:gemma:latest verified with 0 warnings (BUG-025) PASS Verified against ev_001 & src/recursion.py

Complete benchmark report is documented in tests/evaluation/results/QUALITY_BENCHMARK_REPORT.md.


Formal Defect Reports & Research Audit

  • Bug Reports (BUG-001 through BUG-029): Full root-cause analysis and resolution records for 29 isolated engineering defects (including evaluation depth, Windows process tree termination, packfile permission locks, metric conflation, unguarded parameter fallbacks, SemVer + build metadata, NTFS Alternate Data Stream injection, Windows reserved device names, positional-only parameter markers, @staticmethod argument preservation, @property getter/setter coexistence, TypeScript generic functions, aliased imports, .gitconfig diff.noprefix spoofing immunity, historical Git blob caller lookup, Markdown call-arity drift in modified docs, StrEnum migration & Enum member diffing, body-only function mutation test-gap tracking, pre-filter coverage calculation on multi-language repos, self-recursive caller blindspot & broken recursive arity (BUG-021), unconditional infinite recursion & unqualified method self-calls (BUG-022), deep AST RecursionError hardening (BUG-023), multi-class method short-name collision (BUG-024), local Ollama gemma:latest 7B VRAM swap eviction & JSON normalization (BUG-025), MCP tool error boundaries & bool-as-int validation (BUG-026), pytest cache recursion isolation (BUG-027), partial-clone --numstat / -M lazy network fetch timeouts (BUG-028), and redundant HEAD blob subprocess storms (BUG-029)) are cataloged in tests/evaluation/results/BUG_REPORTS.md.
  • Implementation & Research Audit Matrix: Full traceability from the research specification to implemented capabilities is available in tests/evaluation/results/IMPLEMENTATION_AUDIT.md.

Tested Security Controls & Boundaries

  • Path Canonicalization & Jail: All filesystem paths are resolved and validated with is_relative_to(repo_root). Path traversal attempts (e.g. ../../etc/passwd or symlink escapes) are blocked with INVALID_REPO_PATH.
  • Windows Device Name & ADS Guards: Output file paths are verified against Windows reserved device names (CON, PRN, AUX, NUL, COM1-9, LPT1-9) and NTFS Alternate Data Stream colons (:) to prevent device locks and stream injection.
  • CLI Flag Injection & .gitconfig Spoofing Neutralization: Revisions and arguments passed to Git subprocesses are validated to prevent flag injection attacks (blocking leading -, --upload-pack, -o, --exec), and all Git commands enforce -c diff.noprefix=false -c diff.mnemonicPrefix=false -c color.ui=never.
  • Safe File Writing: Release-note writing requires explicit overwrite=True to prevent destructive overwriting of existing changelogs.
  • Pre-LLM Secret Redaction: High-entropy strings, AWS tokens (AKIA...), GitHub personal access tokens (ghp_...), Bearer headers, and private keys are redacted prior to LLM or markdown exposure.
  • Prompt Injection Isolation: Repository content forwarded to optional LLMs is strictly fenced inside <untrusted_repo_content> XML tags with explicit system instructions to ignore embedded directives.

Current Scope & Limitations

  • Language AST Support: Complete AST contract and recursion defect analysis (parameter defaults, body guards, return types, exception raises, base classes, Enum members, public constants, recursive self-call arity, and infinite recursion base-case detection) is implemented for Python. TypeScript/JavaScript uses structural symbol, generic, enum, and export scanning. Other languages (C++, Java, Go, Rust) use unified diff heuristics and commit metadata.
  • Inter-file Caller Resolution & Dynamic Dispatch: Static caller analysis resolves callers across local modules within the same repository using import hierarchy tracking (including aliased imports and self-recursive calls) and verifies call-site argument arity. When reflection (getattr), dynamic imports (importlib.import_module), or bracket dispatch (handlers[action]()) are present, Athena explicitly flags them in unresolved_dynamic_calls and adjusts analysis_coverage.overall_blast_radius_confidence so reviewers know the exact boundary of static analysis.
  • Read-Only Operation: Athena does not write tags, create branches, or commit code to external repositories.

Running the Automated Test Suite

Athena features a comprehensive automated test suite covering all functional capabilities, security boundaries, MCP stdio protocol roundtrips, recursive function edge cases, and live gemma:latest (Gemma 7B) & qwen2.5-coder:7b integration (49 tests):

# Run complete test suite (test_athena_mvp.py, test_mcp_protocol.py, test_real_world_edge_cases.py)
python -m pytest -v

# Run 19-scenario ground-truth semantic quality & Gemma 7B / Qwen 7B benchmark
python tests/evaluation/run_quality_benchmark.py

# Run large-scale multi-commit & MCP repository evaluation
python tests/evaluation/run_evaluation.py --sort-by-stars --limit 85 --llm --model gemma:7b

License

Athena is licensed under the Apache License, Version 2.0.

Yorumlar (0)

Sonuc bulunamadi