searxng-mcp

An MCP server for private web search via a self-hosted SearXNG instance. Results are reranked by a local ML model, full-page content is fetched via Firecrawl, and an optional Ollama instance provides query expansion and LLM-synthesized summaries.

Designed for use with Claude Code and LibreChat agents that need web search without sending queries to a third-party search API.

Built with Claude Code using the multi-agent workflow from homelab-agent — the same platform that uses searxng-mcp in production for AI-assisted research.

Tools

Parameters

category — general (default), news, it, science

time_range — day, week, month, year — limits results by publication date. Omit for all-time results.

fetch_count — number of top reranked results to fetch full content for (default 1, max 3 for search_and_fetch; default 3, max 5 for search_and_summarize).

domain_profile — apply a named domain filter profile: homelab (surfaces self-hosted/Linux docs) or dev (surfaces Stack Overflow, MDN, npm). Omit for default filters.

expand — when true, rewrites the query via Ollama (OLLAMA_EXPAND_MODEL) before searching to improve recall. Requires OLLAMA_URL. Defaults to the EXPAND_QUERIES env var value.

language — BCP-47 language code (e.g. en, de) or all to restrict to a specific language. Omit to use the SearXNG instance default. Available on search, search_and_fetch, and search_and_summarize.

Architecture

MCP client (stdio)
      │
      ▼
  searxng-mcp ──────────────→ Valkey ($VALKEY_URL)        → result cache (search 1h, fetch 24h)
      │
      ├── expand (optional) →  Ollama ($OLLAMA_URL)        → rewritten query (qwen3:4b)
      ├── search ───────────→ SearXNG ($SEARXNG_URL)      → raw results
      ├── rerank ───────────→ Reranker ($RERANKER_URL)    → ranked results
      │                       (fallback: SearXNG order if reranker unavailable)
      ├── fetch content ────┬→ GitHub API (github.com)    → markdown
      │                     ├→ Kiwix ($KIWIX_URL)         → ZIM content (Wikipedia/SO/Arch Wiki, fast path)
      │                     ├→ Firecrawl ($FIRECRAWL_URL) → page markdown (tier 1)
      │                     ├→ Crawl4AI ($CRAWL4AI_URL)  → page markdown (tier 2, optional)
      │                     ├→ Raw HTTP + Readability     → page markdown (tier 3 fallback)
      │                     └→ Wayback Machine (opt-in)  → archived page markdown (tier 4, $WAYBACK_ENABLED)
      └── summarize (opt.) →  Ollama ($OLLAMA_URL)        → synthesized summary ($OLLAMA_SUMMARIZE_MODEL)

flowchart TD
    entry["fetchPage(url)"]
    cache{"Valkey cache hit?"}
    cached["→ return cached { title, url, text }"]
    github{"github.com?"}
    gh_fetch["GitHub API / raw.githubusercontent.com\n→ return"]
    llms{"llms.txt domain?"}
    llms_fetch["Probe /llms-full.txt\nextract matching section\n→ return"]
    kiwix{"Kiwix host?\nKIWIX_URL set"}
    kiwix_fetch["Local Kiwix ZIM\nWikipedia · Stack Overflow · Arch Wiki\n→ cache + return"]
    pdf{".pdf URL?"}
    robots["robots.txt pre-check — tiers 1–3\ndisallowed → RobotsDisallowedError (cached 24h)"]
    tier_skip(["Per-domain tier skip\nsuccess rate <30% over ≥10 tries\nor tier_skip operator override"])
    t1["Tier 1 — Firecrawl\n$FIRECRAWL_URL"]
    t2["Tier 2 — Crawl4AI\n$CRAWL4AI_URL · optional"]
    t3["Tier 3 — Raw HTTP + Readability\nfallback: raw HTML slice"]
    t4["Tier 4 — Wayback Machine CDX API\narchived snapshot · WAYBACK_ENABLED=true"]
    post["Post-extraction\nJSON-LD Article · title cascade\nog:title → twitter:title → title → h1 → URL"]
    result["→ return { title, url, text }"]

    entry --> cache
    cache -->|hit| cached
    cache -->|miss| github
    github -->|yes| gh_fetch
    github -->|no| llms
    llms -->|yes| llms_fetch
    llms -->|no| kiwix
    kiwix -->|yes| kiwix_fetch
    kiwix -->|no| pdf
    pdf -->|"yes — skip tier 1"| t2
    pdf -->|no| robots
    robots --> tier_skip
    tier_skip --> t1
    t1 -->|success| post
    t1 -->|"empty / error"| t2
    t2 -->|success| post
    t2 -->|"empty / error"| t3
    t3 -->|success| post
    t3 -->|"empty / error"| t4
    t4 -->|success| result
    post --> result

    style entry fill:#ffffff,stroke:#333333,color:#000000
    style cache fill:#ffffff,stroke:#333333,color:#000000
    style cached fill:#ffffff,stroke:#333333,color:#000000
    style github fill:#dae8fc,stroke:#6c8ebf,color:#000000
    style gh_fetch fill:#dae8fc,stroke:#6c8ebf,color:#000000
    style llms fill:#dae8fc,stroke:#6c8ebf,color:#000000
    style llms_fetch fill:#dae8fc,stroke:#6c8ebf,color:#000000
    style kiwix fill:#fff9c4,stroke:#b8860b,color:#000000
    style kiwix_fetch fill:#fff9c4,stroke:#b8860b,color:#000000
    style pdf fill:#ffffff,stroke:#333333,color:#000000
    style robots fill:#ffffff,stroke:#333333,color:#000000
    style tier_skip fill:#f5f5f5,stroke:#666666,color:#000000
    style t1 fill:#d5e8d4,stroke:#5a8a4a,color:#000000
    style t2 fill:#d5e8d4,stroke:#5a8a4a,color:#000000
    style t3 fill:#d5e8d4,stroke:#5a8a4a,color:#000000
    style t4 fill:#f8cecc,stroke:#a03030,color:#000000
    style post fill:#e1d5e7,stroke:#7a5a8a,color:#000000
    style result fill:#ffffff,stroke:#333333,color:#000000

SearXNG and Firecrawl are required. Crawl4AI, Valkey, Ollama, Kiwix, and the reranker are optional — the server degrades gracefully when any of these are unavailable.

Adblock at tier 1

The firecrawl-puppeteer service used by Firecrawl runs a custom image (docker/puppeteer-adblock/) that layers @ghostery/adblocker-puppeteer over the upstream trieve/puppeteer-service-ts. EasyList + EasyPrivacy are loaded at startup and refreshed every 168 hours; the blocker is applied to every page Firecrawl creates. Speeds up fetches of ad-heavy sites and shrinks rendered DOM size.

Env vars:

The base image is pinned by SHA256 digest. To deploy a change, rebuild and restart the service:

docker compose -f ~/docker/firecrawl-simple/docker-compose.yml up -d --build firecrawl-puppeteer

Per-domain bypass: domains.json reserves an adblock_skip slot for future operator overrides. Wiring isn't implemented yet — it would require Firecrawl to forward a custom header through to the puppeteer-service, which isn't part of its current API. Tracked as scope-creep item I.

Data-driven tier routing

Before invoking the fetch cascade, searxng-mcp reads the domain's tier_stats_30d (see domain capability database) and skips any tier with success rate below 30% over at least 10 attempts. Cold-start domains (<10 attempts) keep the default cascade. Each skip emits a searxng.fetch.tier.skipped NATS event with reason: low_success_rate and increments searxng_fetch_total{outcome=skipped}.

Operator override. Add a tier_skip map to domains.json to force-skip tiers regardless of stats:

{
  "tier_skip": {
    "example-bot-blocked.com": ["tier1"],
    "another-site.example": ["tier1", "tier2"]
  }
}

tier_skip keys can be bare domains (example.com matches the domain and all subdomains) or domain + path prefix (example.com/api/). The file is hot-reloaded — no restart needed. Manual overrides emit reason: operator_override.

Domain capability database

Every fetch records what searxng-mcp learns about the target domain to Valkey under domain:<hostname> (90-day TTL, schema_version 2). Captured per record:

• tier_stats_30d.{tier1,tier2,tier3}.{attempts, ok, fail, last_fail_reason, window_start_ms} — fetch success rate per tier over a rolling 30-day window; counters reset when the window expires
• capabilities.robots_txt.{present, fetched, allows_us} — robots.txt presence and whether it permits us
• capabilities.llms_full_txt.{present, size_bytes, last_checked} — whether the domain serves /llms-full.txt
• capabilities.json_ld_article.{sampled, present, last_sampled_at} — how often Article-schema JSON-LD is found
• capabilities.og_title.{sampled, present, last_sampled_at} — same for <meta property="og:title">
• preferred_strategy — currently set to llms_full_txt when a present probe lands; future phases will use this to skip the tier cascade

Inspect a record with the bundled CLI:

pnpm dump-domain docs.anthropic.com

Concurrent updates for the same hostname (the tier-attempt, robots-probe, and post-extract-sample recorders that fire in parallel during one fetch) are serialized through an in-process write queue per hostname.

llms.txt fast path

For whitelisted documentation domains in domains.json (llms_txt array), fetchPage tries <origin>/llms-full.txt first and extracts the section matching the requested URL before invoking any tier. This avoids running puppeteer against well-instrumented docs sites and returns a clean markdown section directly. Cached probe outcomes live in Valkey (llms:<origin>:full, 24 h / 7 d for present/absent); the large body is held in-process for the lifetime of the MCP. Default whitelist: docs.anthropic.com, docs.openai.com, docs.stripe.com, docs.crawl4ai.com, docs.firecrawl.dev, docs.cursor.com. Extend by editing domains.json — the file is hot-reloaded.

Kiwix fast path

When KIWIX_URL is set, fetch requests for known offline-capable hosts are intercepted
before the Firecrawl/Crawl4AI cascade and served from the local Kiwix
ZIM archive. This eliminates the 100% tier-1 failure rate for sites like Wikipedia (which
blocks headless scrapers) and returns clean readable content with zero external network traffic.

Supported hosts and ZIM books (kiwix-serve must run with --nodatealiases / -z):

The Kiwix path runs after the llms-txt fast path and before the robots gate. If the Kiwix
request fails or returns empty, the full tier cascade runs as normal. When KIWIX_URL is
unset the feature adds zero overhead — isKiwixHost() returns false immediately.

Set KIWIX_URL to your kiwix-serve base URL (e.g. http://localhost:8292).

Fetch quality

After any tier returns content with raw HTML, a post-extraction pass improves title and body quality:

• JSON-LD Article extraction — Schema.org Article / NewsArticle / BlogPosting / TechArticle blocks supply cleaner headline and articleBody than tier-1 chrome scraping (size-capped at 1 MB per script tag).
• Title cascade — falls back through og:title → twitter:title → <title> (with publisher-suffix stripping) → first <h1> → URL.
• Tier-2 Readability comparison — when Crawl4AI returns markdown, JSDOM+Readability also runs over its raw HTML and is preferred when its text is longer (or unconditionally when Crawl4AI returns less than 500 chars).

Observability (opt-in)

Tracing, metrics, and event publishing are entirely opt-in — with none of the env vars below set, the server has zero observability overhead and never loads the OpenTelemetry or NATS packages at runtime.

OpenTelemetry (traces + metrics) — set OTEL_EXPORTER_OTLP_ENDPOINT to your collector's HTTP endpoint and the server emits:

• Spans (per request): tool.<name> → expand_query? → searxng_request → rerank → fetch (×N) → tier1_firecrawl | tier2_crawl4ai | tier3_rawfetch → post_extract; plus summarize_llm for search_and_summarize.
• Counters: searxng_search_total{profile, expand}, searxng_fetch_total{tier, outcome}, searxng_cache_total{namespace, outcome}, searxng_errors_total{stage, error_type}.
• Histograms: searxng_search_duration_seconds{profile}, searxng_fetch_duration_seconds{tier, outcome}.

Standard OTEL env vars apply (OTEL_SERVICE_NAME defaults to searxng-mcp).

NATS events — set NATS_URL (e.g. nats://localhost:4222) and the server publishes a structured event on every search, fetch, cache hit/miss, robots skip, and error. Subjects:

Each envelope includes request_id and (when OTel is enabled) trace_id so subscribers can join the two streams. Subject prefix overridable via NATS_SUBJECT_PREFIX. Search queries flow through search.* events — downstream consumers are responsible for any PII scrubbing.

Politeness

• Honest User-Agent — outbound requests identify as searxng-mcp/<version> (+https://github.com/TadMSTR/searxng-mcp; personal research).
• robots.txt compliance — /robots.txt is fetched once per origin and cached for 24 hours in Valkey under robots:<origin>. Disallowed paths are skipped before any tier runs and logged as skipped_robots url=… reason=….

Transport

stdio (compatible with Claude Code MCP plugin and LibreChat stdio config).

Prerequisites

• Node.js 20+
• pnpm (or npm)
• A running SearXNG instance
• A running Firecrawl instance
• A running reranker exposing a Jina-compatible /v1/rerank endpoint (optional)
• A running Valkey or Redis-compatible instance (optional, for result caching)
• A running Ollama instance with qwen3:4b and/or qwen3:14b pulled (optional, for query expansion and summarization)

SearXNG

SearXNG must have JSON output format enabled. In settings.yml:

search:
  formats:
    - html
    - json

Reranker

The reranker must expose a Jina-compatible /v1/rerank endpoint. A lightweight FlashRank wrapper works well — see the docker/reranker/ reference in homelab-agent.

Firecrawl

Any Firecrawl-compatible instance works. The local firecrawl-simple deployment is sufficient. Set FIRECRAWL_API_KEY if your instance requires authentication (defaults to placeholder-local for local deployments that skip auth).

Crawl4AI

Crawl4AI is an optional second-tier fetch fallback used when Firecrawl returns empty content (bot-blocked pages, JS-heavy sites). Set CRAWL4AI_URL to enable it. If unset, the cascade skips to raw HTTP fetch.

docker run -d -p 11235:11235 unclecode/crawl4ai:0.8.6

If your instance requires API token authentication, set CRAWL4AI_API_TOKEN.

On the search_and_summarize path, Crawl4AI requests use fit_markdown for noise-filtered content extraction. Other callers (search_and_fetch, fetch_url) use raw_markdown.

Kiwix (optional)

kiwix-serve serves ZIM archives over HTTP. Download
the required ZIM files and run kiwix-serve with --nodatealiases (-z) so book names are
stable:

kiwix-serve --port 8292 --nodatealiases /path/to/zims/

Required ZIM files for each supported host:

• Wikipedia: wikipedia_en_all_mini (or maxi)
• Stack Overflow: stackoverflow.com_en_all
• Arch Wiki: archlinux_en_all_maxi

ZIM files can be downloaded from library.kiwix.org.

Valkey / Redis

Any Redis-compatible instance. Valkey is recommended. Search results are cached for 1 hour; fetched pages for 24 hours. If unavailable, the server operates without caching.

Ollama

Required for expand and search_and_summarize. Pull the required models:

ollama pull qwen3:4b   # query expansion
ollama pull qwen3:14b  # summarization

Set think: false behavior is handled automatically — no extra Ollama configuration needed.

Configuration

All service URLs are configurable via environment variables.

Install

npm (recommended)

npm install -g @tadmstr/searxng-mcp

Or run directly with npx:

npx @tadmstr/searxng-mcp

From source

git clone https://github.com/TadMSTR/searxng-mcp.git
cd searxng-mcp
pnpm install
pnpm build

Output: build/src/index.js

MCP Client Configuration

Claude Code (CLI)

The recommended approach uses claude mcp add-json to register the server with full env var support:

claude mcp add-json searxng --scope user '{
  "command": "npx",
  "args": ["-y", "@tadmstr/searxng-mcp"],
  "env": {
    "SEARXNG_URL": "http://localhost:8081",
    "FIRECRAWL_URL": "http://localhost:3002",
    "RERANKER_URL": "http://localhost:8787",
    "OLLAMA_URL": "http://localhost:11434",
    "VALKEY_URL": "redis://localhost:6379",
    "CACHE_TTL_SECONDS": "3600",
    "FETCH_CACHE_TTL_SECONDS": "86400",
    "EXPAND_QUERIES": "false",
    "CRAWL4AI_URL": "http://localhost:11235"
  }
}'

This writes to ~/.claude.json. Do not add searxng to ~/.claude/settings.json — that file is not used for MCP env var injection in Claude Code.

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "searxng": {
      "command": "npx",
      "args": ["-y", "@tadmstr/searxng-mcp"],
      "env": {
        "SEARXNG_URL": "http://localhost:8081",
        "FIRECRAWL_URL": "http://localhost:3002",
        "RERANKER_URL": "http://localhost:8787",
        "OLLAMA_URL": "http://localhost:11434",
        "VALKEY_URL": "redis://localhost:6379",
        "CRAWL4AI_URL": "http://localhost:11235"
      }
    }
  }
}

LibreChat (librechat.yaml)

mcpServers:
  searxng:
    type: stdio
    command: node
    args:
      - /path/to/searxng-mcp/build/src/index.js
    env:
      SEARXNG_URL: http://localhost:8081
      FIRECRAWL_URL: http://localhost:3002
      RERANKER_URL: http://localhost:8787
      OLLAMA_URL: http://localhost:11434
      VALKEY_URL: redis://localhost:6379
      CRAWL4AI_URL: http://localhost:11235

GitHub URLs

github.com URLs are handled natively without Firecrawl:

• Repo root (github.com/owner/repo) — fetches the README via the GitHub API
• File blob (github.com/owner/repo/blob/branch/path/to/file) — fetches raw content from raw.githubusercontent.com

Unauthenticated requests are rate-limited to 60/hour. Set GITHUB_TOKEN to raise this to 5,000/hour.

Security

URL safety

The fetch_url and search_and_fetch tools enforce a URL allowlist — private/internal IP ranges (10.x, 192.168.x, 172.16-31.x, localhost, 127.x), IPv6 private ranges (::1, fc00::/7, fe80::/10), and non-HTTP protocols are blocked. This prevents the server from being used as an SSRF proxy into your local network.

Redirect protection

HTTP redirects in raw fetch requests are blocked to prevent SSRF bypass via redirect chains to internal addresses.

Dependency auditing

CI runs pnpm audit on every push. The lockfile (pnpm-lock.yaml) is committed for reproducible, auditable builds.

Credential handling

No credentials are stored or logged by the server. API keys (FIRECRAWL_API_KEY, GITHUB_TOKEN, CRAWL4AI_API_TOKEN) are read from environment variables and used only in outbound requests to their respective services.

Input validation

Environment variables are validated at startup — RERANK_RECENCY_WEIGHT warns on NaN, negative, or >1.0 values. Numeric tool parameters use z.coerce.number() with range constraints.

License

MIT