Black Sparrow

High-performance, local-first website crawler, 120-rule technical SEO audit engine, and AI-native auditor written in Rust.

Overview

Black Sparrow is a lightweight, blazing-fast website crawler and technical SEO auditor designed as a modern, local-first alternative to heavy legacy tools like Screaming Frog and expensive cloud crawlers.

Built from the ground up in Rust, Black Sparrow streams HTML responses through Cloudflare's zero-copy lol_html parser, dynamically adjusts crawl speed using network congestion algorithms (AIMD), persists audit history in an embedded SQLite database, and exposes a native Model Context Protocol (MCP) server for autonomous AI coding agents.

Key Features

• ⚡ Blazingly Fast: Crawls ~500–800 pages/second on raw HTTP using under 50 MB of RAM.
• 🔒 Local-First & Private: All data stays on your machine in an embedded SQLite WAL database. No accounts, no cloud dependencies, zero telemetry.
• 🧠 AI-Native (MCP): Native stdio Model Context Protocol server. AI agents in Claude Desktop, Cursor, and Windsurf can launch audits and inspect findings autonomously.
• 🛡️ AIMD Congestion Politeness: Additive-Increase / Multiplicative-Decrease rate controller actively protects target servers from overload.
• 🔍 120 Comprehensive Rules: Validates HTTP transport, metadata, headings, directives, canonicalization, links, security headers, CLS image dimensions, hreflang reciprocity, and Google Rich Results.
• 🤖 GEO & AI Search Readiness: Audits /llms.txt, probes AI crawler permissions (GPTBot, ClaudeBot), and flags bot-challenge firewall screens.
• 🕸️ Topology & Internal PageRank: Ingests link graphs into petgraph to detect orphan pages, canonical loops, and distribute internal link equity.
• 📊 Multi-Format Reporting: Generates interactive standalone TUI-style HTML reports, Screaming Frog-compatible CSV suites, Markdown for LLMs, and structured JSON.

Quickstart

Quick Install (Standalone Binary)

Install blacksparrow (with sparrow alias) directly without needing Rust or build tools:

macOS & Linux:

curl -fsSL https://raw.githubusercontent.com/Shantodotdev/blacksparrow/main/install.sh | bash

Windows (PowerShell):

powershell -ExecutionPolicy Bypass -c "irm https://github.com/Shantodotdev/blacksparrow/releases/latest/download/blacksparrow-installer.ps1 | iex"

Windows users can also download the standalone .msi installer from GitHub Releases.

Build from Source

Prerequisites: Requires Rust 1.80+ (rustup.rs). On Linux, ensure build-essential, pkg-config, and libssl-dev are installed (see CONTRIBUTING.md for OS-specific details).

1. Clone the Repository

git clone https://github.com/Shantodotdev/blacksparrow.git
cd blacksparrow

2. Choose How to Run

You have three flexible options depending on your workflow:

Option A: Run Directly with Cargo (Best for quick testing)

Run audits immediately without installing anything to your system PATH:

# Cargo compiles and runs blacksparrow on the fly
cargo run -- audit https://example.com --max-pages 100

Option B: Build the Standalone Binary (Best for production & scripts)

Build a self-contained, optimized release binary into ./target/release/blacksparrow:

# Compile optimized release binary
cargo build --release

# Run the binary directly
./target/release/blacksparrow audit https://example.com --max-pages 500

Option C: Install Globally as a System Command (Best for daily use)

cargo install --path . compiles the release binary and copies it to your global Cargo binary directory (~/.cargo/bin). This makes blacksparrow available from any folder in your terminal like any standard Unix tool:

# Install to ~/.cargo/bin/blacksparrow
cargo install --path .

# Now available globally from any terminal folder (or use 'sparrow' alias via install.sh)
blacksparrow audit https://example.com --max-pages 500

Basic Usage

You can use either blacksparrow or its alias sparrow:

1. Audit a Website

# Run a 500-page crawl with concurrency limit of 10
sparrow audit https://example.com --max-pages 500 -c 10

2. Quick Single-Page Inspection

# Inspect a single URL instantly without crawling the whole site
sparrow inspect https://example.com/blog/post-1

3. Inspect Issues by Severity

# List all critical and alert issues from the latest session
sparrow issues <SESSION_ID> --severity critical,alert

4. Export Reports

# Export an interactive HTML dashboard and Screaming Frog-compatible CSVs
sparrow report <SESSION_ID> -f html,csv -o ./reports

5. Check AI Search Readiness

# Check /llms.txt and AI bot permissions in robots.txt
sparrow check-ai https://example.com

Use with AI Agents (MCP)

Black Sparrow includes a built-in Model Context Protocol (MCP) server running pure-Rust JSON-RPC 2.0 over stdio.

Quick Setup

Tell your AI coding agent (Claude, Cursor, Windsurf) to register Black Sparrow:

"Add an MCP server named blacksparrow with the command blacksparrow and args ['mcp']."

What Agents Can Do

• seo_start_audit: Spawns non-blocking background crawls (returns a session token in $<1$s).
• seo_audit_status: Polls live crawl progress, page counts, and real-time health score.
• seo_get_markdown_report: Generates executive Markdown summaries tailored for LLM context windows.
• seo_query_issues: Filters issues by category, severity, and rule code.
• seo_quick_page_check: Instantly audits single URLs during local development.

(See Model Context Protocol Guide for full tool schemas and configurations).

Documentation

Explore the complete technical documentation in docs/:

Contributing

We welcome contributions! Please review CONTRIBUTING.md for development setup, testing guidelines, and instructions on how to add new SEO rules.

License

Black Sparrow is dual-licensed under either:

• MIT License (LICENSE-MIT)
• Apache License, Version 2.0 (LICENSE-APACHE)

at your option.