AgenticCrawler

  █████╗  ██████╗██████╗  █████╗ ██╗    ██╗██╗     
 ██╔══██╗██╔════╝██╔══██╗██╔══██╗██║    ██║██║     
 ███████║██║     ██████╔╝███████║██║ █╗ ██║██║     
 ██╔══██║██║     ██╔══██╗██╔══██║██║███╗██║██║     
 ██║  ██║╚██████╗██║  ██║██║  ██║╚███╔███╔╝███████╗
 ╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝ ╚══╝╚══╝ ╚══════╝

LLM-powered web crawler. Describe what you want in plain English — get structured data back.

Single binary. No Python runtime. 21 tools. 25 LLM providers. MCP server built-in.

Why acrawl?

Most web scraping still means writing code: XPath selectors, pagination logic, retry handling, anti-bot workarounds. LLMs can read pages like humans do, but wiring one up to a browser is a project in itself.

acrawl is that wiring, packaged as a single Rust binary. You describe a goal; the agent figures out which pages to visit, what to click, what to extract, and when it's done.

• No code required. Describe the goal in English. The agent plans and executes.
• One binary, zero runtimes. cargo build --release produces a self-contained executable. No Python, no Node runtime — just Rust and a Chromium download for browser automation.
• Smart fetching. Static pages are served over HTTP (fast). When JavaScript or interaction is needed, acrawl detects JS framework markers (__next_data__, __nuxt, __vue, ng-app, React roots), auth redirects, and short <noscript> bodies — then transparently escalates to a headless browser.
• 21 tools, not a chatbot. The agent has real tools — navigate, click, fill forms, run JS, take screenshots, manage tabs — plus a fork/join layer to spawn parallel sub-agents across multiple browser tabs.
• 25 LLM providers. Anthropic, OpenAI, Google Gemini, DeepSeek, AWS Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, Groq, Mistral, xAI, Cohere, Alibaba DashScope, OpenRouter, and more. Or bring your own via any OpenAI-compatible endpoint.
• MCP client. Extend the agent with custom tools via Model Context Protocol servers (stdio, SSE, HTTP, WebSocket).
• MCP server. acrawl mcp exposes all 17 browser tools plus an autonomous run_goal agent to any MCP-compatible client — Claude Code, Cursor, Windsurf, VS Code, Zed, JetBrains, TRAE, Gemini CLI, and more. Install with acrawl mcp install.

How does it compare?

Quick Start

Install

Linux / macOS (x64 / ARM64):

curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash

Windows (x64, PowerShell):

irm https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.ps1 | iex

This downloads the latest binary, verifies its SHA256 checksum, and sets up CloakBrowser for stealth browser automation. Requires Node.js 20+ for browser features.

acrawl checks for updates on startup and shows a notification when a new version is available.

git clone https://github.com/Mingye-Lu/AgenticCrawler.git
cd AgenticCrawler
cargo build --release

# Install CloakBrowser (required for browser automation — binary auto-downloads on first use)
npm install

Browser Extension (optional)

The acrawl Bridge extension lets acrawl control your real browser (with your sessions, cookies, and existing extensions) instead of a headless CloakBrowser instance. Download acrawl-extension.zip from the latest release, unzip it, then load it into your browser:

Enable Developer mode, click Load unpacked, and select the unzipped folder. Then run /extension in the acrawl REPL to connect. See extension/README.md for full setup details.

Configure

# Set up your LLM provider (interactive prompt)
./target/release/acrawl auth anthropic   # or: openai, other

Credentials are stored in ~/.acrawl/credentials.json. Override the config directory with ACRAWL_CONFIG_HOME.

Run

# Interactive REPL
./target/release/acrawl

# One-shot mode
./target/release/acrawl prompt "scrape all book titles and prices from books.toscrape.com"

# Resume a saved session
./target/release/acrawl --resume session.json /status /compact

Examples

Scrape a product catalog:

acrawl > scrape all book titles, prices, and ratings from books.toscrape.com

The agent navigates to the site, reads the page, extracts the data, paginates through all 50 pages, and returns structured JSON.

Fill and submit a form:

acrawl > go to example.com/contact, fill in name "Jane Doe", email "[email protected]",
         message "Hello", and submit the form

The agent locates form fields, fills them in, clicks submit, and confirms the result.

Monitor a price:

acrawl > check the current price of "Rust in Action" on books.toscrape.com

Single-page extraction — the agent fetches, reads, and returns the price without unnecessary navigation.

Extract from JS-rendered pages:

acrawl > get all repository names and star counts from github.com/trending

Static HTTP won't work here. acrawl detects React/Next.js markers and automatically escalates to a headless browser to render the JavaScript.

Parallel multi-page crawl:

acrawl > scrape the title, author, and price of every book across all 50 pages on books.toscrape.com.
         Fork a sub-agent for each page to speed this up.

The agent spawns up to 5 concurrent sub-agents, each on its own browser tab, to crawl pages in parallel. Results are merged when all sub-agents finish.

Features

21-Tool Toolbox

Navigation

Interaction

Content Extraction

Agent Control

Sub-Agent Parallelism

The agent can fork child agents to crawl multiple pages concurrently. Each child gets its own browser tab, step budget, and independent state.

Smart Fetch Routing

Every navigate call goes through a two-tier fetch router:

1. HTTP first — fast reqwest-based fetch (30s timeout, follows up to 10 redirects).
2. Auto-escalation — if any of the following are detected, the request is transparently replayed in a headless browser:
   • HTTP 403, 429, or 503 responses
   • JS framework markers: __next_data__, __nuxt, __vue, ng-app, _react, data-reactroot
   • Auth redirects: URLs containing /login, /signin, /auth, /oauth, accounts.google.com
   • Short response body (< 500 chars) with a <noscript> tag

When --no-headless / --headed is set, all fetches go directly through the browser.

25 LLM Providers

Models use the provider/model-id format: anthropic/claude-sonnet-4-6, openai/gpt-4o, amazon-bedrock/anthropic.claude-sonnet-4-6-20250514-v1:0, etc.

Interactive TUI

The default interface is a full terminal UI with:

• Markdown rendering with syntax highlighting and streaming output
• Slash command overlay — type / to see all commands with Tab completion
• Model picker — /model opens a searchable list grouped by provider category
• Auth modal — /auth walks through provider setup interactively
• Session header — shows current model, session ID, cost, and context usage in real time
• Debug mode — /debug toggles raw tool call input/output in the transcript
• Reasoning effort — Ctrl+T cycles through high/medium/low for reasoning models (o3, o4-mini)

Keybindings:

Running acrawl without a TTY on stdout (e.g. piped or redirected) exits with an error pointing at acrawl prompt for one-shot use and acrawl --resume for session maintenance.

Session Management

• Auto-save — sessions are saved automatically on exit.
• Resume — --resume session.json reloads a conversation. Resume-safe slash commands (/status, /compact, /cost, /config, /version, /export, /help, /clear) can be appended to the command line.
• Export — /export [file] writes a human-readable markdown transcript.
• Auto-compaction — when context exceeds the token threshold (default 200K), acrawl summarizes older messages while preserving the most recent turns, unique tools used, and pending work items.
• Multiple sessions — /session list to browse, /session switch <id> to switch.

Tool Allowlist

Use --allowedTools to restrict which tools the agent can invoke (comma-separated, flag is repeatable):

acrawl prompt "scrape titles" --allowedTools navigate,read_content,screenshot

Omit --allowedTools to allow all 21 tools. Useful for locking down a crawl to read-only tools or excluding fork/wait_for_subagents when sub-agent parallelism is not desired.

MCP Extensibility

acrawl supports Model Context Protocol servers as a client, allowing you to extend the agent with custom tools. MCP tools are namespaced as server_name__tool_name and available alongside the built-in 20.

Supported transports: stdio, SSE, HTTP, WebSocket.

MCP Server (expose acrawl as a tool)

acrawl mcp starts a built-in MCP server that exposes acrawl's browser automation capabilities to external agents like Claude Code, Cursor, VS Code, Zed, JetBrains, TRAE, Gemini CLI, or any MCP-compatible client.

The server provides 18 tools in two modes:

Direct browser tools (17) — fine-grained control for clients that orchestrate themselves:
navigate, click, click_at, fill_form, page_map, read_content, screenshot, go_back, scroll, wait, select_option, execute_js, hover, press_key, switch_tab, list_resources, save_file

Autonomous agent (1) — delegate a full crawl task:

• run_goal — Execute a high-level crawl goal autonomously. The agent plans, navigates, and extracts data using its own LLM loop. Requires ~/.acrawl/credentials.json configured with a model.

Transport: stdio only (no SSE / HTTP / WebSocket in this release).

Quick install

acrawl mcp install

Interactive installer that auto-detects your IDEs, lets you toggle which to configure (Space to select, Enter to confirm), and writes the correct config for each. Supports global (user-level) and project-level scopes.

Supported clients: Claude Code, Claude Desktop, Cursor, Windsurf, VS Code (Copilot), OpenCode, Zed, TRAE, JetBrains IDEs, Gemini CLI, Qwen Code, Codex CLI, Hermes, OpenClaw, Goose, Crush, Aider.

Manual configuration

If you prefer to configure manually, add this to your IDE's MCP config file:

{
  "mcpServers": {
    "acrawl": {
      "command": "acrawl",
      "args": ["mcp"]
    }
  }
}

{
  "servers": {
    "acrawl": {
      "command": "acrawl",
      "args": ["mcp"]
    }
  }
}

{
  "mcp": {
    "acrawl": {
      "type": "local",
      "command": ["acrawl", "mcp"]
    }
  }
}

{
  "context_servers": {
    "acrawl": {
      "command": {
        "path": "acrawl",
        "args": ["mcp"],
        "env": {}
      },
      "settings": {}
    }
  }
}

Or via the Claude Code CLI directly:

claude mcp add acrawl -- acrawl mcp

The browser tools share a persistent session across calls. run_goal creates its own isolated agent and browser.

Requirements: The 17 browser tools work without any configuration. run_goal requires ~/.acrawl/credentials.json (via acrawl auth) for its internal LLM.

Usage

acrawl [OPTIONS] [COMMAND]

Commands:
  prompt <text>      Run a single goal non-interactively
  mcp                Start MCP server (stdio transport)
  mcp install        Install MCP config into your IDEs interactively
  auth [provider]    Configure provider credentials
  system-prompt      Print the system prompt (for debugging)

Options:
  --model MODEL            Model in provider/id format (e.g. anthropic/claude-sonnet-4-6)
  --output-format FORMAT   text | json
  --resume FILE            Resume a saved session (with optional /commands)
  --compact                Compact history on resume
  --headless[=BOOL]        Force browser headless on/off
  --no-headless, --headed  Launch browser in visible mode
  --allowedTools TOOLS     Restrict available tools (comma-separated, repeatable)
  -p TEXT                  Shorthand for prompt mode
  -V, --version            Print version

Slash Commands

Configuration

All config lives in ~/.acrawl/ (override with ACRAWL_CONFIG_HOME).

credentials.json

Managed via acrawl auth. Stores per-provider:

Azure additionally requires resource_name and deployment_name. Bedrock requires aws_access_key_id, aws_secret_access_key, and region. Vertex requires gcp_project_id and gcp_region.

settings.json

Created with defaults on first run.

Environment Variables

Provider-specific env vars (see provider table above) are read as fallbacks when no credentials.json entry exists.

How It Works

flowchart LR
    Goal([Goal\nnatural language]) --> Plan
    Plan --> Navigate --> Observe --> Act --> Extract
    Extract -->|repeat until done| Plan
    Extract --> Output([Output\nJSON / CSV])

1. The agent receives a goal and builds a multi-step plan via a 7-section system prompt covering identity, operating procedure, data integrity, constraints, error recovery, completion protocol, and parallel exploration guidance.
2. Each turn, it picks from its 21 tools based on what it observes on the page.
3. navigate hits the FetchRouter, which tries HTTP first and auto-escalates to a headless Chromium browser when JavaScript, auth redirects, or framework markers are detected.
4. The browser is driven by an embedded Node.js subprocess (the PlaywrightBridge) speaking newline-delimited JSON over stdio — uses CloakBrowser for stealth browsing, not stock Playwright. Alternatively, acrawl can drive the user's real browser via a Chrome extension (/extension command) using CDP over a local WebSocket bridge.
5. For multi-page tasks, the agent can fork child agents onto separate browser tabs, each with independent state and step budgets. wait_for_subagents or done merges results.
6. When context grows large, auto-compaction summarizes older messages while preserving recent turns, tool usage, and pending work items.
7. The agent calls done when the goal is met, or stops when the step limit is reached.

Architecture

crates/
  core/         Shared types, traits, error hierarchy (acrawl-core)
  api/          25 provider clients (Anthropic, OpenAI, Gemini, DeepSeek, Bedrock, Azure, ...), SSE streaming
  browser/      PlaywrightBridge, ExtensionBridge, FetchRouter, BrowserContext, WsBridgeServer
  agent/        21 tools, agent loop, sub-agent fork/join, CrawlState
  runtime/      ConversationRuntime, config, sessions, MCP client stack, OAuth PKCE
  render/       Markdown rendering, tool output formatting, OutputSink
  mcp-server/   Built-in MCP server (JSON-RPC over stdio), IDE installer
  tui/          Ratatui terminal UI (acrawl-tui)
  cli/          Thin binary entry point, LiveCli orchestration, session management
  commands/     17 slash commands with resume-safety annotations
  crawler/      Transitional re-export shim (will be removed)

11 crates, ~38K lines of Rust, 770 tests.

Development

cargo build --release                                     # build
cargo test --workspace                                    # run all tests
cargo clippy --workspace --all-targets -- -D warnings     # lint (pedantic)
cargo fmt --check                                         # format check

See CONTRIBUTING.md for the full development guide.

Changelog

See CHANGELOG.md.

Security

See SECURITY.md for the security policy and how to report vulnerabilities.

License

MIT