Awesome Scrapers 
A curated list of scrapers, crawlers, browser automation tools, extraction libraries, and managed services.
⚠️ = known maintenance concern; verify repository status before adoption.
How to Choose
🤖 AI-Powered Scraping
LLMs understand page structure, extract via natural language, and output LLM-ready formats.
| Tool |
Stars |
Language |
Description |
| Firecrawl |
168k |
TypeScript |
Websites → LLM-ready markdown or structured data via API. |
| browser-use |
109k |
Python |
AI agents that control a browser to complete tasks autonomously. |
| Crawl4AI |
78k |
Python |
LLM-friendly web crawler with structured extraction. |
| Docling |
65k |
Python |
IBM — parse PDFs, DOCX into AI-ready output. |
| ScrapeGraphAI |
30k |
Python |
Graph pipelines + LLMs to extract data via plain English. |
| Stagehand |
24k |
TypeScript |
Browser automation combining natural language with code precision. |
| Skyvern |
23k |
Python |
Browser workflows with computer vision + LLMs, no selectors needed. |
| Jina Reader |
12k |
TypeScript |
Any URL → LLM-friendly markdown with vision model support. |
| llm-scraper |
7k |
TypeScript |
Structured data from any webpage using LLMs with Zod schemas. |
| Spider |
3k |
Rust |
Async, configurable web crawler with streaming and browser support. |
(⬆ back to top)
🥷 Stealth & Anti-Detection
The cat-and-mouse game of modern scraping.
| Tool |
Stars |
Language |
Description |
| Scrapling |
75k |
Python |
Adaptive scraping with built-in anti-detection and auto-matching. |
| SeleniumBase |
13k |
Python |
Browser automation with UC (Undetected Chrome) mode. |
| Camoufox |
11k |
Python |
Firefox-based browser automation project with fingerprint-management features. |
| curl_cffi |
6k |
Python |
HTTP client with browser TLS/JA3/HTTP2 fingerprint impersonation. |
| Nodriver |
5k |
Python |
Successor to undetected-chromedriver — direct CDP, no WebDriver. |
| Botasaurus |
6k |
Python |
Scraping framework with anti-detection, parallelism, and caching. |
| Patchright |
4k |
JavaScript |
Undetected Playwright fork that passes bot detection. |
(⬆ back to top)
🔌 MCP Servers (Model Context Protocol)
Connect LLM agents (Claude, GPT, etc.) directly to scraping tools.
| Server |
Stars |
Description |
| Playwright MCP |
36k |
Browser automation via accessibility snapshots (by Microsoft). |
| Firecrawl MCP |
7k |
Web scraping and search in Claude/Cursor via Firecrawl API. |
| Browserbase MCP |
3k |
Cloud browser control with Stagehand AI. |
| Bright Data MCP |
3k |
Web access with geo-unblocking and bot evasion. |
(⬆ back to top)
🌐 Browser Automation
The foundation for dynamic/JS-heavy scraping.
| Tool |
Stars |
Language |
Description |
| Puppeteer |
95k |
JavaScript |
Google's Chrome/Firefox control via DevTools Protocol. |
| Playwright |
95k |
Multi |
Cross-browser automation (Chromium, Firefox, WebKit) by Microsoft. |
| Selenium |
34k |
Multi |
The OG browser automation (W3C WebDriver standard). |
| Crawlee |
25k |
TypeScript |
Scraping/automation library with proxy rotation by Apify. |
(⬆ back to top)
🕷️ Web Scraping Frameworks
Python
| Tool |
Stars |
Description |
| Scrapy |
64k |
Python scraping framework — middleware, pipelines, extensions. |
| MechanicalSoup |
5k |
Stateful browser-like interaction for simple scraping. |
| scrapy-playwright |
1k |
Playwright integration for Scrapy — JS rendering with full pipeline. |
Go
| Tool |
Stars |
Description |
| Colly |
25k |
Fast scraping framework for Go. |
| Katana |
17k |
Crawling and spidering framework by ProjectDiscovery. |
| Ferret |
6k |
Declarative scraping with FQL query language. |
Ruby
| Tool |
Stars |
Description |
| Nokogiri |
6k |
Standard HTML/XML parser for Ruby. |
(⬆ back to top)
📡 HTTP Clients
The network layer — making requests that look human.
| Tool |
Stars |
Language |
Description |
| aiohttp |
17k |
Python |
Async HTTP client/server for high-concurrency scraping. |
| httpx |
15k |
Python |
Async/sync HTTP client with HTTP/2 support. |
| curl_cffi |
6k |
Python |
HTTP client impersonating browser TLS fingerprints (also in Stealth). |
| got-scraping |
768 |
Node.js |
HTTP client with header/TLS mimicry by Apify. |
(⬆ back to top)
🧩 HTML & XML Parsing
| Tool |
Stars |
Language |
Description |
| Cheerio |
30k |
JavaScript |
jQuery-like HTML manipulation for Node.js. |
| goquery |
15k |
Go |
jQuery-like HTML selector for Go. |
| jsoup |
11k |
Java |
HTML parser with CSS selectors and XSS sanitization. |
| AngleSharp |
6k |
C# |
W3C-compliant HTML5 parser for .NET. |
| Beautiful Soup |
- |
Python |
Most popular Python HTML/XML parser. |
| lxml |
3k |
Python |
Fast XML/HTML parser with XPath and XSLT. |
| html5ever |
3k |
Rust |
Browser-grade HTML5 parser from Mozilla Servo. |
| selectolax |
2k |
Python |
5-30x faster than Beautiful Soup using Lexbor engine. |
| parsel |
1k |
Python |
CSS/XPath selectors for HTML+JSON (powers Scrapy). |
(⬆ back to top)
📝 Content & Text Extraction
Pull clean text out of messy HTML — essential for LLM/RAG pipelines.
| Tool |
Stars |
Language |
Description |
| Readability.js |
11k |
JavaScript |
Mozilla's article extractor (powers Firefox Reader View). |
| Trafilatura |
7k |
Python |
Web text extraction with metadata and language detection. |
| html2text |
2k |
Python |
HTML → clean Markdown. |
| Markdownify |
2k |
Python |
Flexible HTML-to-Markdown with customizable options. |
| newspaper4k |
1k |
Python |
News article extraction with NLP and multilingual support. |
(⬆ back to top)
📱 Social Media Scrapers
Platforms frequently change APIs and block scrapers. Check issue trackers for current status.
| Tool |
Stars |
Platform |
Description |
| Instaloader |
13k |
Instagram |
Posts, stories, reels, highlights with metadata. |
| TikTok-Api |
7k |
TikTok |
Unofficial API wrapper for Python. |
| PRAW |
4k |
Reddit |
Official Python Reddit API Wrapper. |
(⬆ back to top)
🎬 Media Downloaders
| Tool |
Stars |
Description |
| yt-dlp |
185k |
YouTube and 1000+ sites (fork of youtube-dl). |
| lux |
32k |
Go video downloader — 40+ sites (formerly annie). |
| spotdl |
26k |
Spotify tracks/playlists with metadata and album art. |
| gallery-dl |
19k |
Image galleries from 100+ sites (Pixiv, Twitter, Reddit). |
(⬆ back to top)
📄 Document & PDF Extraction
| Tool |
Stars |
Language |
Description |
| Docling |
65k |
Python |
IBM — PDFs, DOCX, PPTX into AI-ready output. |
| Unstructured |
15k |
Python |
ETL pipeline for documents → structured data for LLMs. |
| pdfplumber |
11k |
Python |
Text, tables, and layout from PDFs with precision. |
| PyMuPDF |
10k |
Python |
Fast PDF/XPS/EPUB extraction and rendering. |
| Tabula |
7k |
Java |
Data tables from PDFs. ⚠️ |
| pdfminer.six |
7k |
Python |
PDF text extraction with layout analysis. |
| Camelot |
4k |
Python |
PDF table extraction — lattice and stream modes. |
| tabula-py |
2k |
Python |
Python wrapper for Tabula. ⚠️ |
(⬆ back to top)
👁️ OCR & Screen Scraping
| Tool |
Stars |
Language |
Description |
| Tesseract |
76k |
C++ |
Google's OCR engine — 100+ languages. |
| PaddleOCR |
88k |
Python |
Lightweight OCR — 100+ languages with LLM integration. |
| EasyOCR |
30k |
Python |
Ready-to-use OCR — 80+ languages, PyTorch. |
| pytesseract |
6k |
Python |
Python wrapper for Tesseract. |
(⬆ back to top)
⛓️ Blockchain & On-Chain
| Tool |
Type |
Description |
| The Graph |
Open Source (3k) |
Blockchain indexing via GraphQL subgraphs. |
| Subsquid |
Open Source |
Blockchain indexer — 50k+ blocks/sec. |
| Dune Analytics |
SaaS |
SQL-based blockchain analytics. |
| Etherscan APIs |
Freemium API |
REST APIs for Ethereum data. |
(⬆ back to top)
☁️ Managed Scraping APIs
Pay-per-request services that handle proxies, browsers, and anti-bot for you.
| Service |
Best For |
Key Feature |
| Apify |
Full-stack platform |
Marketplace of pre-built Actors plus Crawlee-based development and managed execution. |
| ScrapingBee |
Simple API access |
JavaScript rendering, screenshots, and search APIs. |
| ZenRows |
Anti-bot bypass |
Managed scraping API with browser rendering and proxy rotation. |
| ScrapFly |
Multi-API |
Scraping, screenshots, extraction, and crawler APIs. |
| Browserless |
Headless browsers |
Headless Chrome in Docker, BrowserQL, self-hostable. |
| Browserbase |
AI browser agents |
Cloud browsers for AI, session persistence, Stagehand integration. |
| Oxylabs |
Enterprise |
ML-driven proxy rotation, e-commerce specialized. |
| Bright Data |
Scale |
Web Unlocker with CAPTCHA solving, geo-routing, mobile UA. |
| SerpApi |
SERP data |
Structured results from Google, Bing, Yahoo. |
| ScraperAPI |
Getting started |
Proxy rotation, browser rendering, and structured scraping endpoints. |
(⬆ back to top)
🧪 CAPTCHA Solving
| Service |
Method |
Pricing |
Supports |
| 2Captcha |
Human workers |
Usage-based; see current pricing |
reCAPTCHA, Turnstile, FunCaptcha, GeeTest, image. |
| Anti-Captcha |
Human workers |
Usage-based; see current pricing |
reCAPTCHA, hCaptcha, FunCaptcha, Turnstile. |
| CapSolver |
Automated solving |
Usage-based; see current pricing |
reCAPTCHA, AWS WAF, Cloudflare, GeeTest. |
| CapMonster Cloud |
Automated solving |
Usage-based; see current pricing |
reCAPTCHA, hCaptcha, and Turnstile. |
(⬆ back to top)
🌍 Proxy Providers
| Provider |
Coverage |
Pricing |
Highlights |
| Bright Data |
Residential, ISP, mobile, and datacenter proxies |
Pay-as-you-go |
Broad proxy and web-data product portfolio with geographic targeting. |
| Oxylabs |
Residential, ISP, mobile, and datacenter proxies |
Contact sales |
Enterprise proxy services with country, city, and ZIP targeting options. |
| Decodo |
Residential, ISP, mobile, and datacenter proxies |
Published plans |
Geographic targeting and rotating or sticky sessions. |
| IPRoyal |
Residential, ISP, mobile, and datacenter proxies |
Published plans |
Proxy services with geographic targeting and pay-as-you-go options. |
| NetNut |
Residential, ISP, mobile, and datacenter proxies |
Contact sales |
ISP-sourced proxy connectivity and geographic targeting. |
| Webshare |
Residential and datacenter proxies |
Published plans |
Self-service proxy plans and a limited free tier. |
(⬆ back to top)
🆓 Try Free
(⬆ back to top)
🪦 Deprecated Tools Graveyard
| Dead Tool |
Why |
Use Instead |
| PhantomJS |
Archived 2018 |
Playwright, Puppeteer. |
| CasperJS |
Depended on PhantomJS |
Playwright, Puppeteer. |
| Nightmare |
Unmaintained since 2020 |
Playwright. |
| Zombie.js |
Unmaintained |
Playwright, Puppeteer. |
| SlimerJS |
Unmaintained, Gecko-based |
Playwright (Firefox). |
| Splash |
Scrapinghub, deprecated |
Scrapy-Playwright. |
| twint |
Archived Mar 2023, blocked by Twitter |
Official API. |
| Goutte (PHP) |
Deprecated by Symfony |
Symfony BrowserKit + DomCrawler. |
| snscrape |
Unmaintained since Nov 2023 |
Official APIs. |
| undetected-chromedriver |
Aging, last push Jul 2025 |
Nodriver, Camoufox. |
| puppeteer-extra-stealth |
Unmaintained since Jul 2024 |
Patchright, Camoufox. |
| playwright-stealth |
Unmaintained since Nov 2023 |
Patchright, Camoufox. |
| curl-impersonate |
Unmaintained since Jul 2024 |
curl_cffi. |
| GoogleScraper |
Unmaintained since Jul 2021 |
SerpApi. |
| pyautogui |
Unmaintained since Aug 2024 |
pytesseract + Playwright. |
| SikuliX |
Stale, niche |
Playwright, pytesseract. |
(⬆ back to top)
Disclosure
Some links in the Managed Scraping APIs, CAPTCHA Solving, Proxy Providers, and Try Free sections are affiliate/referral links. These help support the maintenance of this list. All tools are included based on merit — affiliate status does not influence placement or rankings.
(⬆ back to top)
🔗 Related Awesome Lists
| List |
Description |
| awesome-ai |
AI APIs, tools, frameworks, platforms, and learning resources. |
| awesome-robotics |
Robotics frameworks, simulators, and platforms. |
| awesome-web3-ai |
Web3 x AI tools, agent frameworks, and protocols. |
Contributing
Contributions welcome! Please read the contribution guidelines first.
- Add tools you've actually used or evaluated
- Include star count and language where applicable
- Note if a tool is unmaintained (last commit >1 year ago)
- Commercial tools/services are fine but must be clearly labeled
To the extent possible under law, Edward Tay has waived all copyright and related or neighboring rights to this work.