Crawlit

Give your AI agent the web: Google search, scraping and crawling on your own machine.
Clean Markdown for LLMs. Firecrawl-compatible API. One docker compose up. $0 per page, forever.

Website · Quickstart · AI agent skill · API · Benchmarks · llms.txt

# Search Google and get every result back as clean Markdown — one request
curl -X POST localhost:3000/v1/search -H 'Content-Type: application/json' \
  -d '{"query": "fastify rate limit plugin", "limit": 3, "scrapeOptions": {"formats": ["markdown"]}}'

{
  "success": true,
  "data": {
    "engine": "google",
    "results": [
      {
        "position": 1,
        "title": "fastify/fastify-rate-limit",
        "url": "https://github.com/fastify/fastify-rate-limit",
        "snippet": "A low overhead rate limiter for your routes...",
        "markdown": "# @fastify/rate-limit\n\nA low overhead rate limiter for your routes..."
      }
      // ...
    ]
  }
}

Why Crawlit

AI agents are only as good as what they can read. The hosted options charge per page, cap your usage, or send your browsing through someone else's servers. Crawlit is the same capability as a single container you own:

• Everything an agent needs in one API. Search the web, read a page, read 100 pages at once, map a site, or crawl it. You get Markdown back, with no HTML soup.
• Fast by default. Plain HTTP when that's enough and a stealth browser only when a page needs JavaScript. A worker-thread extraction pool, keep-alive connections, and a Redis cache.
• Drop-in for Firecrawl. /v1/scrape, /v1/crawl and /v1/map take the same requests as Firecrawl, so existing clients work by changing the base URL.
• Built for agents. The crawlit-skill pack teaches Claude Code, Codex, OpenCode and others to use it instead of their built-in web tools.
• Free and private. MIT license, no API key, no per-page billing, and your queries never leave your machine except to the sites you fetch.

Benchmarks

v1.1 against v1.0, same machine, same network, same requests. Apple M1, 8 cores, Docker Desktop.

Where the speed comes from:

• linkedom instead of JSDOM for Readability: 3–4× faster with identical text.
• One HTML parse per page instead of three.
• Extraction runs in worker threads, so a huge page doesn't block other requests.
• The crawl queue was throttled to 2 jobs/s globally and had racy counters. Both are fixed.

Reproduce it yourself against any running instance:

scripts/benchmark.sh

Quickstart (30 seconds)

You only need Docker.

git clone https://github.com/arufian/Crawlit.git
cd Crawlit
docker compose up -d --build     # first build downloads Chromium (~90 MB)
curl localhost:3000/health       # {"status":"ok","version":"1.1.0",...}

Try it:

# Any page → Markdown
curl -X POST localhost:3000/v1/scrape -H 'Content-Type: application/json' \
  -d '{"url": "https://en.wikipedia.org/wiki/Web_crawler"}'

# Search Google
curl -X POST localhost:3000/v1/search -H 'Content-Type: application/json' \
  -d '{"query": "best open source web crawler", "limit": 5}'

docker compose down      # stop (cache persists)
docker compose down -v   # stop + wipe Redis data

Use it from your AI agent

Install the crawlit-skill and your agent uses Crawlit for all web work: searching, reading docs, and crawling sites.

# Claude Code
claude plugin marketplace add arufian/crawlit-skill
claude plugin install crawlit@crawlit-skills

# Codex
codex plugin marketplace add arufian/crawlit-skill
codex plugin add crawlit@crawlit-skills

# OpenCode, Cursor, and other agents (GitHub CLI)
gh skill install arufian/crawlit-skill skills/crawlit --agent opencode --scope user

Then just ask:

• "Search the web for the latest Playwright Docker best practices and summarize them"
• "Read https://docs.example.com/api and write a client for it"
• "Crawl the docs under https://docs.example.com/guide and save them as markdown"
• "What pages does https://example.com have about pricing?"

Building your own agent? Any HTTP client works. The API is small, JSON in and JSON out, and documented below and in llms.txt.

What you can do

Highlights:

• Auto mode. Plain HTTP first, then the stealth browser only for JS apps and bot walls.
• Clean output. GFM tables, absolute links, and no base64 image blobs.
• Any page encoding. Shift_JIS, EUC-JP and GBK pages decode correctly, and old frameset sites list their frames as links.
• Speed and reliability. Redirects are followed, identical concurrent requests are merged, and results are cached in Redis.
• Optional extras. Schema-guided LLM extraction (OpenAI or Anthropic), proxy support per request or globally, and Markdown files saved to disk with YAML frontmatter.

API

All endpoints take and return JSON. Base URL: http://localhost:3000.

Scrape a single page

curl -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links"]
  }'

The x-crawlit-source response header says which path produced the content: http, browser, or cache. Redirects are followed and metadata.sourceURL is the final URL. Plain-text, Markdown and JSON URLs are returned as-is.

curl -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "actions": [
      { "type": "click", "selector": ".load-more" },
      { "type": "wait", "delay": 1000 },
      { "type": "scroll" }
    ],
    "formats": ["markdown"],
    "save": true
  }'

curl -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "formats": ["markdown"],
    "extract": {
      "schema": {
        "type": "object",
        "properties": {
          "topStory": { "type": "string" },
          "points": { "type": "number" }
        },
        "required": ["topStory"]
      },
      "prompt": "Extract the top story title and its points"
    }
  }'

Requires OPENAI_API_KEY or ANTHROPIC_API_KEY.

Batch scrape

Scrape many URLs in parallel in one request. Accepts every scrape option above, plus urls and concurrency.

curl -X POST http://localhost:3000/v1/batch/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com", "https://docs.example.com/start"],
    "formats": ["markdown"],
    "concurrency": 8
  }'

# { "success": true, "total": 2, "completed": 2, "failed": 0,
#   "data": [{ "url": "...", "success": true, "markdown": "...", "metadata": {...} }, ...] }

Failed URLs appear in data with success: false and an error; the request itself still succeeds. Unlike Firecrawl's async batch API, this endpoint responds when all pages are done.

Search the web

curl -X POST http://localhost:3000/v1/search \
  -H "Content-Type: application/json" \
  -d '{ "query": "crawlit firecrawl alternative", "limit": 5 }'

# { "success": true, "data": { "query": "...", "engine": "google",
#   "results": [{ "title": "...", "url": "...", "snippet": "...", "position": 1 }], "total": 5 } }

How Google is queried:

1. Plain HTTP to Google's lightweight results page (about 0.5s, direct result URLs).
2. If that is throttled, the stealth browser.
3. If Google is blocked both ways, DuckDuckGo. The response then contains fallbackFrom with the reason.

A blocked path is skipped for 10 minutes. GET /health shows each path's state. If every engine is blocked, the response has blocked: true and empty results. Heavy automated searching from one IP gets throttled by Google; set PROXY_URL to spread the load.

Map URLs (no content fetch)

curl -X POST http://localhost:3000/v1/map \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://docs.example.com", "search": "/guide/" }'

# { "success": true, "links": [...], "total": 42 }

Fetches the common sitemap locations, every Sitemap: listed in robots.txt, and the seed page's links in parallel, then merges them.

Crawl multiple pages

# Start
curl -X POST http://localhost:3000/v1/crawl \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://docs.example.com", "maxDepth": 2, "limit": 100, "includePaths": ["^/docs/"] }'
# { "success": true, "id": "<crawl-id>", "url": "/v1/crawl/<crawl-id>" }

# Poll (paged; follow "next" until it's absent)
curl http://localhost:3000/v1/crawl/<crawl-id>
# { "status": "running", "completed": 12, "failed": 0, "total": 47, "next": "...", "data": [...] }

# Cancel
curl -X DELETE http://localhost:3000/v1/crawl/<crawl-id>

Crawls respect robots.txt and skip binary assets (images, archives, PDFs).

Saved files

With save: true, files land in ./output/ on your host (Docker volume), each with YAML frontmatter:

---
title: "Getting Started"
url: "https://docs.example.com/getting-started"
scraped_at: "2026-04-25T10:00:00.000Z"
description: "..."
language: "en"
---

## Getting Started
...

Comparison

Crawlit is the only option that combines all four:

• Free, MIT-licensed, and self-hosted.
• A Firecrawl-compatible API.
• Built-in web search.
• A ready-made agent skill pack.

Jina Reader pricing estimated at $0.05–0.10 per 1K characters. Firecrawl assumes annual billing. Apify pricing based on Starter plan with additional usage.

Configuration

Everything works with defaults. To change something, cp .env.example .env and edit.

FAQ

Is it really free?
Yes. MIT license, no account, and no API key. It costs only the machine it runs on.

Do I need to know Node.js?
No. Docker is the only requirement.

Does Google block it?
Light use works fine. Heavy automated searching from one IP gets throttled. Crawlit then falls back to DuckDuckGo automatically, tells you so in fallbackFrom, and retries Google after 10 minutes. For heavy use, set PROXY_URL.

Can I use my existing Firecrawl client?
Mostly. /v1/scrape, /v1/crawl and /v1/map take Firecrawl-style requests: point the base URL at http://localhost:3000. Two differences: /v1/search returns results under data.results instead of a bare data array, and /v1/batch/scrape is synchronous while Firecrawl's is async.

What about sites behind Cloudflare or Akamai?
The stealth browser handles basic challenges. Enterprise bot protection also needs a residential proxy. Set PROXY_URL, or pass proxy per request.

Does it respect robots.txt?
Crawls do. Single scrapes fetch exactly the URL you ask for.

Development

npm install
npm run dev          # hot reload (needs Redis: docker run -d -p 6379:6379 redis:7-alpine)
npm run typecheck
npm run test:unit    # no server needed
npm run test:api     # integration tests against a running server
npm run test:load    # 1000-concurrent load test
npm test             # all of the above

Architecture

POST /v1/scrape ──► cache / in-flight dedupe
(+ batch/scrape)     └─ auto: HTTP (undici, keep-alive) ──► JS shell or bot wall? ──► browser (Playwright pool)
                          └─ worker thread: cheerio (metadata + links) + linkedom/Readability + Turndown/GFM

POST /v1/crawl  ──► BullMQ queue (Redis)
                     └─ Worker: robots.txt → scrape → recordPage → acceptLinks (atomic limit) → addBulk children

POST /v1/map    ──► sitemaps + robots.txt + seed page links (parallel) → normalize/filter

POST /v1/search ──► Google basic HTML (HTTP) → Google (browser) → DuckDuckGo  ──► cache

Built with Fastify, Playwright + playwright-extra (stealth), Mozilla Readability on linkedom, Turndown, Piscina, BullMQ, Redis, the Vercel AI SDK, and Prometheus metrics at /metrics.

See CHANGELOG.md for what's new and CONTRIBUTING.md to get involved.

License

MIT. Use it, fork it, ship it.

_{If Crawlit saves you a Firecrawl bill, a ⭐ helps other people find it.}