pagesift 0.50.0

High-performance, memory-bounded web scraping and extraction engine built for AI agents and LLM pipelines.
Turn any website, document, or archive into clean Markdown and structured data — one Rust binary, native SDKs, dual Firecrawl v1/v2 & Jina Reader APIs, at a fraction of the memory.

English · 简体中文 · 日本語 · Español

🏆 Benchmark Showdown (All 10 Domains + 100k Real-World Showdown)

All benchmarks run on AMD EPYC (32 physical cores, 128 GB RAM) and bare-metal Intel Core i7-10700 (8 cores, 16 threads, 15.5 GB RAM) under an active 12 GiB memory watchdog (MemoryMax=12288M). Verified raw datasets and reproduction scripts are archived in bench/results/final/.

⚔️ The 100,000 Live Public Internet Battle: PageSift vs. Spider 2.53.9

In a head-to-head live public internet showdown on a 32-core AMD EPYC server across 100,000 live Tranco global domains under 768 concurrency:

+-------------------------------------------------------------------------------+
| PageSift 0.50 vs Spider 2.53.9 (100,000 Live Public Domains)                  |
+-------------------------------------------------------------------------------+
| Metric                      | PageSift 0.50 (Rust)   | Spider 2.53.9 (Rust)   |
+-----------------------------+------------------------+------------------------+
| Total Domains Processed     | 105,660 URLs           | 100,000 URLs           |
| Genuine 200 OK Fetches      | 62,937 pages           | 72,209 pages (raw)     |
| Valid Markdown (>100B)      | 60,659 documents       | 59,803 documents       |
| Extraction Conversion Rate  | 96.38% (Rich Markdown) | 82.82% (17.2% Empty)   |
| Content Deduplicated        | 2,449 duplicate pages  | 3,484 duplicate pages  |
| Peak Resident Memory (RSS)  | 188.4 MB (FLAT LINE)   | 8,978.9 MB (~9.0 GB!)  |
| Memory Efficiency Ratio     | 1.0× (Baseline)        | 47.6× Memory Bloat     |
| Long-Term Stability (1M+)   | 100% OOM-Proof         | 💥 OOM Inevitable      |
+-------------------------------------------------------------------------------+

[!TIP]
Machine-verifiable datasets: Read the complete benchmark whitepaper at docs/releases/0.50.0.md and examine all raw JSON/CSV records in bench/results/final/.

What's New in 0.50.0

• 🧠 Dynamic Adaptive Concurrency & Timeout Engine (pagesift::adaptive):
  • Bottleneck-Constrained Model ($\min$ Principal): Concurrency is strictly bounded by the intersection of CPU capacity, physical memory safety (2MB/conn), and OS file descriptors (ulimit -n):
    $$\text{Concurrency} = \min\big(C_{cpu},; M_{mem},; FD_{safe}\big)$$
  • Auto-Configuring Hardware Profiles: Micro (1C1GB), Small (2C2GB~4C4GB), Mid Desktop (8C16GB / i7-10700), High Server (32C128GB / EPYC), and Extreme Supernode (64C+ 128GB+ FD>=65k).
  • Dual Operational Modes: --tuning-mode adaptive (saturates hardware safely) vs. --tuning-mode conservative (caps CPU < 20% for background workstation tasks).
  • RTT-Aware Safe Timeout: Calibrated 3.5s connect timeout and 10.0s total timeout, preventing drops of high-latency international websites.
• 🌐 Dual-Protocol Firecrawl v1 & v2 Architecture: Drop-in compatibility with official Firecrawl Python and Node.js SDKs covering /v1/scrape, /v1/crawl, /v1/batch/scrape, /v1/map and all /v2/* endpoints.
• ⚡ Jina Reader Protocol Adapter: Instant URL scraping via GET /r/{*url} or POST /r, with Server-Sent Events (Accept: text/event-stream) streaming for real-time LLM agent feeds.
• 🛡️ 12 GiB RAM Defense Under Chaos: Zero memory leaks across 50,000 chaos URLs while Spider crashed from 11.7 GB OOM.
• 🦀 100% Pure Safe Rust & Modular Architecture: #![forbid(unsafe_code)] preserved across all core crates. All modified and created .rs source files strictly <= 500 lines.

Quick Start

1. Docker / Self-Hosted Server

Run the standalone container (includes headless browser renderer):

docker run -d -p 3002:3002 -v pagesift-data:/data \
  -e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:0.50.0

Firecrawl SDK Drop-In Replacement

Point your existing Firecrawl application directly to PageSift — zero code refactoring:

from firecrawl import FirecrawlApp

# Drop-in local PageSift instance
app = FirecrawlApp(api_url="http://localhost:3002", api_key="sk-test")
res = app.scrape_url("https://example.com", params={"formats": ["markdown"]})
print(res["markdown"])

Jina Reader Drop-In Replacement

Point any LLM Agent or simple HTTP client to PageSift's /r/ endpoint:

# Direct Markdown output
curl -s http://localhost:3002/r/https://en.wikipedia.org/wiki/Rust_(programming_language)

# Streamed SSE output
curl -s -N -H "Accept: text/event-stream" http://localhost:3002/r/https://news.ycombinator.com

2. Python Package

pip install pagesift

import pagesift

# Single-page fast extraction (< 2ms)
page = pagesift.scrape("https://example.com", formats=["markdown", "metadata"])
print(page.markdown)
print(page.metadata.title, page.metadata.published_date)

# Batch scraping with adaptive bounded memory
results = pagesift.scrape_many(["https://site1.com", "https://site2.com"], concurrency=8)

3. Node.js / TypeScript Package

npm install @pagesift/node
# or
npm install pagesift

import { scrape, scrapeMany } from '@pagesift/node'

// Scrape page to clean LLM markdown
const page = await scrape('https://example.com', {
  formats: ['markdown', 'links']
})
console.log(page.markdown)

// Concurrent stream processing
const stream = scrapeMany(['https://a.com', 'https://b.com'], { concurrency: 4 })
for await (const page of stream) {
  console.log(`Scraped: ${page.url} in ${page.elapsedMs}ms`)
}

4. Command Line Interface (CLI)

# Install standalone binary via cargo
cargo install pagesift --locked

# Scrape single URL
pagesift scrape "https://news.ycombinator.com" --format markdown

# High-concurrency polite crawl with adaptive hardware tuning
pagesift crawl "https://docs.rs" --limit 5000 --tuning-mode adaptive --output ./data/

Why PageSift?

Architectural Highlights

• Segmented Bounded Frontier: Memory does not scale with crawl depth. Even with 100,000 URLs, backpressure and bounded queues prevent out-of-memory disasters.
• Dynamic Hardware Adaptive Tuning: Detects system resources and calculates the safe theoretical saturation point without crashing or exhausting sockets.
• Selective Rendering Pipeline: Automatically detects static vs dynamic pages; executes headless Chrome only when dynamic JavaScript hydration provides net content gains.
• SSRF & Sandbox Defense: Strict private IP filtering, DNS-rebinding guards, and perpetual browser sandboxing ensure zero local network exposure.
• Ethical Politeness: Per-host token buckets, automatic robots.txt compliance, and RFC-compliant User-Agent identification.

License & Commercial Options

• Open Source: Licensed under AGPL-3.0.
• Commercial Licensing: Need proprietary linking or private deployment without AGPL obligations? Check out our Commercial Licensing Guide or contact [email protected].