ByteCrawl

Léelo en español.

Give your AI agent focused web crawling. ByteCrawl is an MCP server (and a
small Python library) that doesn't just scrape a page — it crawls a whole site
and returns the pages most relevant to your topic first, using Shark-Search
and OPIC in pure Python.

• Webpage: https://bytecrawl.vercel.app/
• Hosted MCP endpoint: https://bytecrawl.vercel.app/mcp
• Agent skill: https://bytecrawl.vercel.app/agent-onboarding/SKILL.md
• Latest release: 1.1.2 — six MCP tools, strategy comparison,
  absolute Page.links() (changelog)

Point an agent at it (no install, no key)

If you work through an AI agent, the fastest route is neither of the sections
below — hand it the skill and let it choose:

Read and follow https://bytecrawl.vercel.app/agent-onboarding/SKILL.md

One Markdown file, readable by Claude Code, Cursor, or anything that can fetch a
URL. It routes to the right path for the job (hosted API, MCP, Python library,
or a real browser for JS-rendered pages), documents how to select a crawl
strategy, and includes measured numbers for what that choice buys you.
Read it yourself.

Quick start (MCP — nothing to install)

Point any MCP-capable agent (Claude Code, Claude Desktop, Cursor...) at the
hosted endpoint:

claude mcp add --transport http bytecrawl https://bytecrawl.vercel.app/mcp

Now the agent has six tools — the same six things the playground lets a human do:

Then just ask

You don't call the tools yourself. Once the server is connected you ask in plain
language and the agent picks one:

Use bytecrawl to find everything on python.org about the packaging ecosystem

Read https://example.com/pricing with bytecrawl and give me the plans as a table

Which crawl strategy does best on wikipedia.org for "san francisco"? Compare them

Saying the name is worth the two syllables: most agents ship their own
single-page fetcher and will reach for that by default. Naming bytecrawl is
what gets you a focused crawl instead of one page read in isolation.

The hosted server is static-only, rate-limited per IP, caps crawls at 10 pages,
and refuses non-public URLs (SSRF guard). For heavy use or JS-rendered sites,
run it locally:

pip install bytecrawl[mcp]
claude mcp add bytecrawl -- bytecrawl-mcp                 # full power, on your machine
pip install bytecrawl[browser] && playwright install chromium   # + JS rendering

Why focused crawling?

Most crawlers visit pages in whatever order they find them. With a limited
request budget, order is everything — Shark-Search chases the branches that
smell like your query and lets the rest decay, so 100 requests get you the 100
most useful pages, not the 100 closest to the seed.

from bytecrawl import SharkSearch

result = SharkSearch(query="vector databases").crawl(
    "https://example.com", max_pages=100)

for page in result.top(10):
    print(f'{page["relevance"]:.3f}  {page["url"]}')

• BFS — level by level, closest to the seed first.
• Shark-Search (Hersovici et al., 1998) — topical best-first; links inherit
  their parent's relevance with decay.
• OPIC (Abiteboul et al., 2003) — live PageRank via "cash" flow, no full
  graph needed (a pagerank() implementation is included to compare against).

All three share one loop — pop, fetch, score links, push — so the same page
budget across strategies is a fair comparison. Every surface takes the same
three names, shark (default) / opic / bfs:

An unknown name is rejected rather than silently defaulted: the API returns 400,
the MCP tool raises ValueError. SharkSearch requires query — without a
topic it has nothing to rank. BFS and OPIC accept query too, but only to
score the pages in the result; it does not change their traversal order.

From en.wikipedia.org/wiki/Silicon_Valley with query san francisco and 20
pages each, Shark returns 20 pages above 0.1 relevance against 4 for OPIC and 1
for BFS, and its best page scores 0.7774 against BFS's 0.1068 — 7.3x on the same
request budget. Reproduce it with result.relevant(0.1). A lower threshold will
not close the gap: 20 pages leaves thousands of URLs still queued, so coverage
never converges the way it does on a site small enough to exhaust.

Shark-Search exposes both parameters from the paper — delta (0.5) sets how
fast a branch with no signal fades (decay is δⁿ with depth) and gamma (0.8)
splits a link's score between its parent and its own anchor text,
score = γ·inherited + (1−γ)·local.

Versus the alternatives: Scrapy is a framework you wire up yourself, Firecrawl
is a paid SaaS — ByteCrawl is a plain library with a 3-package core and these
frontier strategies built in.

Library API

from bytecrawl import Scraper

bot = Scraper()
page = bot.fetch("https://books.toscrape.com")   # auto: static, browser fallback
books = page.extract("article.product_pod",
                     {"title": "h3 a::attr(title)", "price": "p.price_color::text"})
page.markdown()   # clean Markdown for LLMs   ·   page.tokens()   # token estimate

bot.static(url)                                   # plain HTML
bot.api(url, params={...})                        # hidden JSON API
bot.browser(url, wait="div.results")              # JS via Playwright
bot.crawl(url, item="article", fields={...},
          next_page="li.next a::attr(href)")      # pagination
bot.session().login(url, data, csrf_field="csrf_token")   # authenticated

Install

pip install bytecrawl            # slim core (requests + beautifulsoup4 + lxml)
pip install bytecrawl[llm]       # + Markdown for LLMs
pip install bytecrawl[browser]   # + Playwright (then: playwright install chromium)
pip install bytecrawl[mcp]       # + local MCP server (Python 3.10+)
pip install bytecrawl[all]

A missing extra never fails silently — each one raises an ImportError naming the
exact command to run.

HTTP API (no install)

The same engine behind one hosted GET endpoint. Everything except url is
optional; no key, no account.

curl "https://bytecrawl.vercel.app/api?url=books.toscrape.com"
curl "https://bytecrawl.vercel.app/api?url=en.wikipedia.org/wiki/Silicon_Valley&method=crawl&query=san+francisco&strategy=shark"

method is one of markdown (default), text, html, links, json,
extract, crawl. Static HTML only, crawls capped at 10 pages, rate-limited,
responses cached 10 minutes, private and loopback addresses refused. Full
reference: https://bytecrawl.vercel.app/docs

Learn each scraping technique

A guided walkthrough (in Spanish) with a runnable example against a practice
site:
static HTML ·
dynamic JS ·
hidden APIs ·
pagination ·
login ·
graph crawling ·
Markdown for LLMs ·
ethics

Contributing

pip install -e ".[llm,dev,mcp]"
pytest              # 120 tests, no network required
pytest -m live      # + live browser tests (needs the browser extra)
ruff check bytecrawl tests

Scrape responsibly: respect robots.txt, terms of service and rate limits.
ByteCrawl ships with a configurable delay between requests.

License

MIT — see LICENSE.