lookacrawler

mcp
Guvenlik Denetimi
Basarisiz
Health Uyari
  • License — License: MIT
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Low visibility — Only 5 GitHub stars
Code Basarisiz
  • exec() — Shell command execution in anti-slop-linter.ts
  • process.env — Environment variable access in audit-fixes.test.ts
  • network request — Outbound network request in benchmark-live.ts
  • network request — Outbound network request in benchmark.ts
  • process.env — Environment variable access in browser-manager.ts
Permissions Gecti
  • Permissions — No dangerous permissions requested

Bu listing icin henuz AI raporu yok.

SUMMARY

Free, open-source, token-efficient local alternative to Firecrawl with native MCP Server for LLMs (>73% token reduction).

README.md

🕷️ LookaCrawler

Free, open-source, token-efficient local alternative to Firecrawl with native Model Context Protocol (MCP) Server for LLMs.

License: MIT
Bun
TypeScript
MCP Ready
GitHub Stars
Local-First


📌 Why LookaCrawler?

Web crawling for Large Language Models (LLMs) is broken by default: modern web pages contain massive HTML bloat (scripts, tracking pixels, nested divs, navigation headers, stylesheets), costing thousands of wasted tokens per page.

LookaCrawler is an open-source, token-optimized local crawler that cuts a median 90% of tokens on real pages (21–98% across 8 sites, see benchmark-live.ts), extracts clean Markdown, renders JavaScript-heavy pages with a real browser, and exposes a native Model Context Protocol (MCP) Server ready for Claude Desktop, Cursor, and Antigravity.


🥊 Comparison: LookaCrawler vs Alternatives

Feature 🕷️ LookaCrawler 🔥 Firecrawl (Cloud) ⚡ Jina Reader
Pricing / Cost $0.00 (100% Free Open Source) $16 to $99+/month Rate-limited API
Token Reduction Median 90% fewer tokens (real pages) + Footnotes Standard Markdown Basic Markdown
Autonomous Crawling Native map & crawl (BFS + URL patterns) Cloud Crawler Single-page only
Pre-Crawl Actions Native Playwright (click, scroll, fill) Paid Addon None
Link Formatting Inline, References Footnotes, Strip Inline only Inline only
Data Privacy Local-first (Zero Telemetry) Cloud Provider Cloud API
MCP Integration Native Tools + Resources + Prompts Community Wrapper None
JS Rendering Real Chrome (Playwright) Cloud Proxies Basic Headers
Local SQLite Cache Built-in (24h TTL cache) Redis / Paid Addon None
JS SPA Support Playwright + Chrome Pool Cloud Headless Headless

🚀 Key Features

  • Token Economy First: Automatically prunes scripts, styles, inline SVGs, tracking tags, navigations, footers, redundant forms, and boilerplate containers with high link density (>80%).
  • Advanced Link & Image Formatting:
    • link_format: Choose between inline (standard markdown), references (footnote citations [1], saving ~25% tokens on repetitive URLs), or strip (pure text).
    • image_mode: Choose between ignore (zero tokens), alt_only (preserves semantic context without URL bloat), or markdown (full ![alt](url)).
  • Autonomous Mapping & Recursive Crawling:
    • map_website: Inspects /robots.txt, sitemaps, and root anchors to discover all pages in a domain.
    • crawl_website: Breadth-first autonomous crawling with max depth, max pages, URL pattern filters (substring, * wildcard), and real-time token accounting.
  • Pre-Crawl Browser Actions: Automate clicks, scrolls, typing, and waits in Playwright before extracting content (dismiss cookie banners, scroll for infinite loading, expand accordions).
  • Dual Hybrid Crawling Engine:
    • fast: Ultra-fast native HTTP GET with backoff. Escalates to deep when a page needs JavaScript to render.
    • deep: Headless Playwright engine launching real Google Chrome for pages that need JavaScript. Challenge pages (CAPTCHA, bot checks) are detected and reported instead of retried.
  • Native MCP Ecosystem:
    • Tools: extract_web_content, crawl_website, map_website, batch_extract_web_content, extract_structured_data, extract_contacts, search_places, discover_local_leads, deep_enrich_lead, lint_anti_slop, lookup_social_profile, search_social_intent, audit_company_website, analyze_competitors, generate_growth_plan, gtm_pipeline_run.
    • Resources: Live telemetry at crawler://metrics and cache analytics at crawler://cache/stats.
    • Prompts: Pre-engineered templates crawl-and-summarize and compare-pages.
  • Entity Extraction: contacts pulls emails, phone numbers, WhatsApp links and social profiles; places returns structured Google Maps business records with formula-injection-safe CSV output.
  • B2B Lead Discovery: maps (Google Maps engine) plus enrich, which crawls a company's own site for verified emails, socials and decision makers.
  • GTM Toolkit (no paid APIs): audit (SEO + conversion checklist), intel (audit + Maps competitor review + deterministic growth plan), gtm (Maps discovery, enrichment, outreach draft written to JSONL/CSV; never sends anything), antislop (AI-slop copy linter), intent (pt-BR/EN buying-intent scorer), social-lookup and intent-search (public profile meta tags, Reddit .json), daemon (Maps scraper web-mode service).
  • Local SQLite Caching: Stores extracted Markdown in crawler_cache.sqlite to eliminate duplicate network calls.
  • Structured JSON & Metadata Extraction: Extracts Open Graph tags (og:title, og:description), publication dates, canonical URLs, and custom CSS selectors.

🔌 1-Click MCP Setup (Claude Desktop & Cursor)

Add LookaCrawler to your claude_desktop_config.json or Cursor MCP settings:

{
  "mcpServers": {
    "lookacrawler": {
      "command": "bun",
      "args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
    }
  }
}

Now you can prompt Claude or Cursor:

"Crawl https://example.com/docs and extract the API documentation using LookaCrawler."


📦 Quick Start & CLI Usage

1. Installation

Requires Bun 1.1+ (high-performance runtime with native SQLite):

# Clone the repository
git clone https://github.com/lucasmartins-ai/lookacrawler.git
cd lookacrawler

# Install dependencies
bun install

2. CLI Commands

# Single URL fast Markdown extraction
bun run cli.ts extract https://news.ycombinator.com --mode fast

# Single URL fast Markdown extraction with reference footnotes
bun run cli.ts extract https://example.com --link-format references --output page.md

# Headless Playwright deep extraction with CSS selector target
bun run cli.ts extract https://example.com --mode deep --selector "main" --json

# Discover all website URLs and sitemaps
bun run cli.ts map https://example.com --max-urls 500

# Recursively crawl documentation with URL pattern filtering and token accounting
bun run cli.ts crawl https://example.com/docs --max-depth 2 --max-pages 15 --include /docs/ --link-format references

# Batch concurrent multi-URL crawling
bun run cli.ts batch https://site1.com https://site2.com --concurrency 4

# Structured JSON schema extraction
bun run cli.ts structured https://example.com --schema '{"title":"h1","links":"a"}'

# Extract emails, phones, WhatsApp links and social profiles from a site
bun run cli.ts contacts https://example.com

# Search local businesses on Google Maps (browser-driven)
bun run cli.ts places "dentists in Bristol" --limit 5 --csv -o places.csv

# B2B lead discovery via the bundled google-maps-scraper engine
# (build it first: bun run build:maps)
bun run cli.ts maps "dental clinic" "Bristol" --limit 20

# Enrich a company website: emails, socials, decision makers, short summary
bun run cli.ts enrich https://example.com --title "Example Ltd" --mode fast

# Enrich many leads at once from a JSON array of {title, website}
bun run cli.ts enrich-batch leads.json --concurrency 3 -o enriched.json

# Audit a site: SEO checklist, conversion channels, services, digital maturity
bun run cli.ts audit https://example.com

# Audit + Maps competitor review + deterministic growth plan (--locale en for English)
bun run cli.ts intel --site https://example.com --niche "dental clinic" --loc Bristol --limit 3

# Discover -> enrich -> draft outreach copy -> JSONL (or .csv). Draft only, nothing is sent
bun run cli.ts gtm --niche "dental clinic" --loc Bristol --limit 5 --out leads.jsonl

# Lint copy for AI-slop (argument, --file, or stdin)
bun run cli.ts antislop "No cenário atual, uma solução revolucionária"

# Score buying intent of a post or comment (generic lexicon; --vertical car for vehicles)
bun run cli.ts intent "Procuro um sofá até R$ 2.500"

# Public social lookups, no keys: profile meta tags and Reddit search
bun run cli.ts social-lookup https://www.instagram.com/example/
bun run cli.ts intent-search "recommend a dentist Goiania" --subreddit goiania

# Run the Maps scraper as a local service (127.0.0.1:8088); `maps`, `gtm` and `intel` use it when up
bun run cli.ts daemon start
bun run cli.ts daemon status
bun run cli.ts daemon stop

# Start MCP Server via SSE on port 3000
bun run cli.ts serve --transport sse --port 3000

Behind a TLS-intercepting proxy? Export LOOKACRAWLER_INSECURE_TLS=true to
accept the proxy's forged certificate. Off by default — never enable it for
untrusted networks.

Rotating proxies? Export LOOKACRAWLER_PROXIES with a comma or
whitespace separated list of URLs. Every request without an explicit
--proxy pulls the next entry, so one blocked IP is not fatal for the run.

Local proxy (Tor, mitmproxy, a sidecar)? A proxy passed per request
must be a public host. List trusted local proxies in
LOOKACRAWLER_ALLOWED_PROXIES (comma separated, exact URLs); entries in
LOOKACRAWLER_PROXIES are trusted too.

SSE server security

Variable Effect
LOOKACRAWLER_SSE_TOKEN Bearer token required on every request. If the server is bound to a non-loopback address (e.g. --host 0.0.0.0) and this is unset, a random token is generated and printed at startup.
LOOKACRAWLER_SSE_ALLOWED_ORIGINS Comma-separated web origins allowed to call the server from a browser (CORS). Requests with any other Origin header are refused. CLI/desktop MCP clients are unaffected.
LOOKACRAWLER_SSE_ALLOW_UNAUTHENTICATED true disables the token requirement on non-loopback binds. Only for isolated networks.
LOOKACRAWLER_SSE_MAX_SESSIONS Maximum concurrent SSE clients (each gets its own MCP session). Default 16; further connections get 503 until one disconnects.
LOOKACRAWLER_MAX_BROWSERS Maximum concurrent Chromium processes (one per distinct proxy). Default 4; idle ones are closed after LOOKACRAWLER_BROWSER_IDLE_MS (default 120000).

Every outbound connection goes through a built-in loopback egress guard that
resolves each destination once, refuses private/link-local/loopback addresses
and connects to exactly the IP it checked. This covers redirects and requests a
page makes on its own (fetch, XHR, WebSocket) in deep mode, and prevents DNS
rebinding. With a proxy (--proxy, LOOKACRAWLER_PROXIES) the guard sits in
front of it: it validates the destination, then asks the proxy to connect to
that IP (HTTP CONNECT, SOCKS5 with optional user/password, or SOCKS4), so a
proxy — even a trusted local one — can never be steered at an internal host.
Side effect: DNS is resolved on this machine, not at the proxy, so geo-DNS/CDN
edges follow your location rather than the proxy's, and SOCKS4 cannot reach
IPv6-only sites. LOOKACRAWLER_ALLOW_LOCAL=true turns all of this off and talks
to the proxy directly (for local development only).

3. Google Maps engine (optional)

The maps command, daemon, gtm, intel and the discover_local_leads MCP
tool drive a Google Maps scraper written in Go. Its source lives in
google-maps-scraper/ (MIT, originally by
Georgios Komninos (gosom), via
coelhobugado/google-maps-scraper,
with local fixes for hangs and leaked browsers).

Responsible use. Google's Maps Platform terms restrict scraping Google Maps
content. Use this engine for low-volume research you review yourself, respect
each site's terms, and prefer the official Places API for commercial volume. Build it once (needs Go 1.26+;
install-browser also needs Node.js 18+; the Docker image ships it built):

bun run build:maps
cd google-maps-scraper && ./maps-leads install-browser   # first run only

The binary is looked up next to cli.ts, in ../google-maps-scraper/, or at
$MAPS_LEADS_BIN (an explicit path that does not exist is an error, not a
fallback).

Without it, maps fails fast with a clear message; extract, crawl,
contacts, places and enrich are unaffected. Pass --email to let the
scraper extract public emails during discovery.

TikTok (public data, no login)

bun run cli.ts tiktok profile @khaby.lame --limit 10
bun run cli.ts tiktok trends --region BR --period 7 --kind hashtag

Same as the MCP tools tiktok_profile and tiktok_trends. Output envelope:
{ "ok": true, "source": "yt-dlp"|"browser"|"creative-center", "items": [...] } or
{ "ok": false, "status": "blocked"|"error", "reason": "..." } (exit code 1).
blocked means TikTok answered 403/429 or a CAPTCHA: final, do not retry.
Profiles need yt-dlp with curl_cffi on PATH (LOOKACRAWLER_YTDLP overrides
the binary). Nothing here logs in, signs requests or solves CAPTCHAs; from a
headless browser TikTok often answers with a CAPTCHA (profiles) or 403 (Creative
Center). Measurements and limits: docs/specs/tiktok.md.

4. Docker Deployment

# Build and run Docker container
docker build -t lookacrawler .
docker run -p 3000:3000 -e LOOKACRAWLER_SSE_TOKEN="$(openssl rand -hex 24)" lookacrawler

The container listens on 0.0.0.0, so a token is always required. Without
LOOKACRAWLER_SSE_TOKEN a random one is generated on each start and printed in
the container logs.


🧪 Architecture & Testing

Incoming URL ──► [Local SQLite Cache Check] ──(Hit)──► Return Cached Markdown
                       │ (Miss)
                       ▼
            [Fast HTTP GET Request] ──(Blocked?)──► [Escalate to Browser Rendering]
                       │                                      │
                       ▼                                      ▼
            [HTML DOM Tree Parser] ◄──────────────────────────┘
                       │
                       ▼
       [Aggressive Token Noise Pruner]
       (Strips SVG, Nav, Ads, Tracking, CSS, JS)
                       │
                       ▼
         [Mozilla Readability Engine]
                       │
                       ▼
         [Turndown Markdown Converter] ──► Return Clean LLM Markdown

Run test suite:

bun test

⭐ Star & Support

If LookaCrawler saves you API fees and token costs:

  • ⭐ Star this repository to help other developers find it!
  • 💡 Open an Issue / PR for rendering improvements or crawler features.

Built by LookADev

lookacrawler is built and maintained by LookADev, an engineering studio specializing in AI agents, web architecture, and token optimization.

Start a project → lookadev.com · Email: [email protected]

📄 License

Open-source software licensed under the MIT License.

Yorumlar (0)

Sonuc bulunamadi