🛰️ AgentSpan

The Web Access Gateway for AI Agents — Multi-Platform, Multi-Tenant, Designed for Speed.

AgentSpan gives AI agents persistent, scalable, cached access to 52 internet
platforms through one REST API, an SSE event stream, a native MCP server (92
tools), 9 language SDKs, a CLI, and a React dashboard — all on an async Rust core.

The channels also heal themselves — a background monitor auto-switches failing
backends, reinstalls broken CLI tools, and alerts on sustained outages — and the
gateway learns from its own traffic, suggesting cache-TTL tweaks, faster
backends, and platforms worth adding next.

Languages: English · 中文 · 日本語 · 한국어

Why AgentSpan?

Agent Reach proved the "capability
layer" model (install → diagnose → route) for agent web access. AgentSpan takes that
model and turns it into a real gateway — it does the reading itself, behind one API.

What AgentSpan is not

• Not a browser-automation agent (can't click, log in, fill forms). Use
  Browser-Use or Playwright for that.
• Not a generic crawler competing with Firecrawl on arbitrary-site quality.
  AgentSpan is optimized for the 52 platforms it knows.
• Not a hosted SaaS. It's a self-hosted binary you run on your own infra.

Website

Two surfaces, one design system — built with React 18 + TypeScript + Vite + GSAP
(ScrollTrigger) + Lenis, pure CSS (no Tailwind/Bootstrap):

• Home (/) — the marketing page; the whole scroll is the landing experience.
• Status (/status) — a live, read-only gateway dashboard.

Home (/)

A section-based editorial / futuristic-HUD clone reverse-engineered from a
reference design and filled with real AgentSpan content:

Signature chrome: corner-frame brackets, a live mouse-coordinate HUD, a sound
toggle, dot-grid section texture, animated chevron pill CTAs, a 1s per-section
background cross-fade, and GSAP scroll-reveal on every section.

Status dashboard (/status)

A read-only at-a-glance view of the gateway, sharing the home page's chrome,
fonts, colors and animations: gateway status header (count-up stat cards),
a searchable/filterable grid of all 52 channels, performance charts
(request volume + latency, recharts), a health monitor table with fallback
status, quick actions, and a copy-to-clipboard install snippet.

Launch: cd web && npm install && npm run dev → / and /status.
(?static renders without the preloader/smooth-scroll, handy for screenshots.)

Architecture

flowchart TD
    subgraph Clients
      SDKs["9 SDKs (Py/JS/Rust/Go/Ruby/Java/PHP/C#/Swift)"]
      CLI["agentspan CLI"]
      Dash["React Dashboard"]
      MCP["MCP clients (Claude/Cursor/...)"]
    end
    subgraph Gateway["AgentSpan Core (Rust, async)"]
      API["REST API + SSE"]
      MCPS["MCP Server (92 tools)"]
      Router["Router: probe · select · retry · circuit-break"]
      Cache["3-tier Cache (L1/L2/L3)"]
      Auth["Auth: keys · tenants · RBAC · rate-limit · audit"]
    end
    Backends["Backends: HTTP APIs · CLI tools · Jina Reader"]
    Clients --> API & MCPS
    API --> Router --> Backends
    Router --> Cache
    API --> Auth

Quick start (3 commands)

cargo run --bin agentspan -- serve            # 1. start the gateway on :8080
curl "localhost:8080/api/v1/read?url=https://example.com"   # 2. read any page
curl "localhost:8080/api/v1/channels/hackernews/search?q=rust"  # 3. search

Or with Docker:

docker compose up --build      # API + UI + Redis + PostgreSQL + Prometheus + Grafana

Use with AI agents (Claude Code, Cursor, Windsurf)

# One command — writes the MCP config to the right file automatically:
agentspan mcp install --client claude-code    # or: cursor, windsurf, cline

# Or print the config and paste it yourself:
agentspan mcp print-config --client cursor

# See all 92 tools:
agentspan mcp tools

See the full guides: Claude Code ·
Cursor ·
Windsurf.

Use an SDK

# Once published to PyPI:
# pip install agentspan
# Until then, install from source:
pip install -e sdk/python
from agentspan import AgentSpanClient
client = AgentSpanClient(base_url="http://localhost:8080")
print((await client.read("https://example.com")).body)

See sdk/README.md for all 9 languages and
docs/api-reference.md for the full API.

Channels (52)

Every channel implements read and/or search, runs through a health-checked
backend router, and reduces tokens via format_for_llm. Run agentspan doctor
to see which backend is serving each channel right now.

CLI

agentspan serve | doctor | watch | format | benchmark | transcribe | tunnel
         | plugin | install | uninstall | setup | config | skill | mcp | completions | update

Highlights: benchmark (cache-hit vs backend p50/p99), tunnel (public URL via
cloudflared), plugin (community channels), transcribe (Whisper), format
(per-channel token-reduction rules), completions <shell> (bash/zsh/fish/
powershell/elvish), and config backup/config restore.

Observability & operations

• GET /metrics — Prometheus text exposition (request counts, errors, shed
  requests, latency, channel gauge). Public, so scrapers need no API key.
• Request tracing — every request/response carries an x-trace-id (an
  inbound x-trace-id/x-request-id is honoured); the id is attached to audit
  events for correlation.
• Resource limits — a 2 MiB request-body cap plus a global in-flight
  concurrency limit that sheds excess load with 503 instead of unbounded
  queueing.
• Graceful shutdown — agentspan serve drains in-flight requests on Ctrl-C
  and (on Unix) SIGTERM.

Smart routing & content

• Adaptive routing — opt in with BackendRouter::with_adaptive_routing().
  The router learns each backend's EWMA latency and success rate from live
  traffic and prefers the better performer within a health tier. Unknown
  backends stay optimistic so they still get tried.
• Content intelligence — fetched pages are classified (article / forum /
  code / docs) with key facts (URLs, dates, code blocks), reading stats
  (word/sentence count, reading time), and an extractive summary (the most
  salient sentences) pulled out and attached as metadata; smart_truncate trims
  on sentence boundaries, not mid-word. All dependency-free heuristics — no model
  calls.
• Conditional revalidation — the direct-HTTP path remembers ETag /
  Last-Modified and revalidates with If-None-Match / If-Modified-Since, so
  unchanged pages come back as a cheap 304 instead of a full re-download.
• Federated search — POST /api/v1/search/federated queries many channels
  at once, de-duplicates by URL (merging which channels found each result), and
  ranks cross-source hits first. One channel failing doesn't sink the query.
  Pass "rerank": true to instead order results by a dependency-free lexical
  relevance score (title-weighted TF with a phrase bonus), and "collapse": true
  to merge near-duplicate hits (the same story re-syndicated under different
  URLs) by title similarity.
• Request coalescing — with_request_coalescing() collapses a dogpile of
  concurrent identical reads into a single upstream fetch (single-flight), so
  five agents asking for the same URL in the same second hit upstream once.

Agent memory

A namespaced key/value scratchpad so agents can persist small state — cursors,
seen-sets, notes — across requests without standing up their own store:

curl -X PUT localhost:8080/api/v1/memory/agent1/cursor \
  -H 'content-type: application/json' -d '{"value":{"page":3},"ttl_secs":3600}'
curl localhost:8080/api/v1/memory/agent1/cursor      # -> {"value":{"page":3},...}
curl localhost:8080/api/v1/memory/agent1             # -> {"keys":["cursor"]}

In-memory and process-local (survives requests, not a restart); entries take an
optional TTL and are swept lazily. GET/PUT/DELETE on
/api/v1/memory/{namespace}/{key}, GET on /api/v1/memory/{namespace}.

AI-native content primitives

Dependency-free, deterministic building blocks in agentspan-channels that make
reads cheaper and more useful for agents — no model calls, no extra services:

• Token-budget compiler (budget::fit_to_budget) — "give me this page in
  ≤ 2 000 tokens." Fits any text to a hard token ceiling, degrading gracefully from
  extractive summary to boundary-aware truncation, and reports which strategy it
  used. The estimate is a safe upper bound, so the ceiling holds.
• Content fingerprinting (fingerprint) — an exact FNV-1a hash plus a 64-bit
  SimHash, so an agent can ask "did this page meaningfully change since I last
  read it, and by how much?" without diffing bodies. Complements ETag revalidation
  (which only says the bytes differ).
• Structured extraction (extract) — pull typed fields (title, links, dates,
  emails, prices, summary) into JSON and project just the keys you asked for, so a
  channel can answer in a fixed schema with zero LLM-extraction spend.

These ship as library primitives; REST/MCP surfacing is the next wave.

Project structure

crates/         Rust workspace (core, probe, router, cache, auth, channels, mcp, api, cli)
sdk/            9 SDKs (python, js, rust, go, ruby, java, php, csharp, swift)
web/            React marketing site + admin dashboard (Vite + GSAP + Lenis)
integrations/   MCP client configs, VS Code/JetBrains/Nvim plugins, GitHub Action
plugins/        Community channel registry
docs/           API reference, guides, mdbook site, i18n
tests/          Integration + load test suites

See KNOWN_ISSUES.md for current rough edges and build quirks.

Community

• Issues — bugs and feature requests
• Discussions — questions and show & tell
• CONTRIBUTING.md — how to add a channel or MCP tool

License

MIT — see LICENSE.

Security

• Binds to 127.0.0.1 by default; the Docker image sets 0.0.0.0.
• Single-user mode (default): admin routes return 403. Set
  auth.require_api_key=true to enable scoped API keys.
• API keys stored as SHA-256 hashes; ~/.agentspan/config.yaml is 0600 and
  secrets are masked in output/logs. See SECURITY.md.

License

MIT — see LICENSE.