SlopSearX

Cloud-native, stateless, AI-agent-first meta search engine. Drop-in SearXNG replacement for the GroktoCrawl stack.

SlopSearX is a horizontally scalable, stateless meta search engine designed for AI agent consumption. It replaces SearXNG in the GroktoCrawl stack with:

• HTML output by default — SearXNG-compatible browser responses with JSON/CSV/RSS negotiation
• Human search portal — a responsive, server-rendered landing and results experience with Dark / Darker themes
• YAML+Markdown native output — structured + readable for AI agent contexts via format=yaml
• SearXNG-compatible API — drop-in replacement for existing consumers
• Plugin engine adapters — one file per engine, @register_engine, zero orchestrator changes
• Category routing — SearXNG-compatible taxonomy with sub-categories and env-var overrides
• Distributed rate limiting — Valkey-backed sliding windows, correct at 50+ replicas
• Response caching — Valkey-backed, category-aware TTL, 150x speedup on cache hits
• OpenMetrics observability — /metrics endpoint, per-engine counters + latency + status
• Stateless, cloud-native — no local volumes, no persistent DB, all replicas interchangeable
• Hybrid config — env vars for secrets, optional mounted file for engine tuning

API

Europe PMC is opt-in; see configuration, attribution, and evaluation.

Engines (51)

The table below is maintained to match the live adapter registry (51 registered
adapters as of this writing). The MCP server's slopsearx_list_capabilities
tool and the slopsearx://capabilities resource are generated from that same
registry at runtime — treat them as authoritative.

General / Web

Developer / Package Registries

GitHub authentication is sub-category-specific: public repository and
issue/PR searches may run without a token, while code search requires
ENGINE_GITHUB_API_KEY. The capability catalog reports authentication at the
engine level, so its single auth.class cannot express this distinction;
the adapter behavior is authoritative for the selected sub-category.

Science & Research

Medical / Health

Security / Threat Intelligence

Finance / Economics

Media & Entertainment

Geography / GIS

Legal

Jobs / ATS

Adding a new engine: See docs/ENGINE_ADAPTERS.md for the full adapter reference — contract rules, data types, lifecycle hooks, and the category system.

Capability metadata: every adapter additionally declares an audited
capability surface (supported_result_types, failure_classes, cost_class
from the closed vocabularies in slopsearx/adapter.py). The MCP capability
catalog exposes these for enabled and disabled engines; a declaration is a
static capability hint and must not be confused with configured availability
(enabled), authentication state (auth.class/auth.configured), or
observed health (last_known_status). See docs/MCP_CONTRACT.md §7.4.

MCP Server (agents)

SlopSearX ships a Model Context Protocol server for AI agents. It exposes
intent-level search (no URL strings), capability discovery, scope
explanation, snapshot-based pagination, asynchronous research jobs, scheduled
change detection, and attributed retrieval receipts —
built on the same pipeline as the HTTP API. Search results carry a
machine-readable retrieval handoff record so a downstream reader (e.g.
GroktoCrawl) can capture pages and link them back to the originating result
(see docs/RETRIEVAL_HANDOFF.md).

# Install (already included in the package dependencies)
pip install -e ".[dev]"

# Run over stdio (default transport)
slopsearx-mcp            # or: python -m slopsearx.mcp

# Run over HTTP for remote clients (e.g. Hermes Agent on another host)
MCP_TRANSPORT=http MCP_HOST=0.0.0.0 MCP_AUTH_TOKEN=change-me slopsearx-mcp

# Remote gateway: stdio MCP server that proxies to a remote SlopSearX server
slopsearx-mcp --remote http://<slopsearx-host>:8000/mcp   # token: MCP_REMOTE_TOKEN=…

# Gateway against an OAuth-mode remote: --oauth runs the standard MCP OAuth
# flow (loopback callback; tokens persisted for reuse)
slopsearx-mcp --remote http://<slopsearx-host>:8000/mcp --oauth

# OAuth 2.1 mode (required by OAuth-only clients, e.g. Claude Web connectors)
MCP_TRANSPORT=http MCP_OAUTH_ENABLED=1 MCP_OAUTH_ISSUER_URL=https://mcp.example.com slopsearx-mcp

• 38 tools: slopsearx_search, slopsearx_search_targeted,
  slopsearx_search_jobs, slopsearx_search_security,
  slopsearx_search_science, slopsearx_list_capabilities,
  slopsearx_explain_search_scope, slopsearx_get_service_status,
  slopsearx_read_results, slopsearx_read_result, slopsearx_read_entities,
  slopsearx_plan_research, slopsearx_plan_query_variants,
  slopsearx_plan_research_followup, slopsearx_start_research, slopsearx_get_job, slopsearx_cancel_job,
  slopsearx_retry_research, slopsearx_extend_research,
  slopsearx_update_research,
  slopsearx_create_saved_search, slopsearx_get_saved_search,
  slopsearx_update_saved_search, slopsearx_pause_saved_search,
  slopsearx_delete_saved_search, slopsearx_read_saved_search_reports,
  slopsearx_read_saved_search_events, slopsearx_ack_saved_search_events,
  slopsearx_submit_retrieval_receipt,
  slopsearx_read_retrieval_receipts, slopsearx_export_research_manifest,
  slopsearx_preview_staged_search, slopsearx_search_staged,
  slopsearx_get_staged_search, slopsearx_retry_staged_search,
  slopsearx_start_dependency_dossier, slopsearx_get_dependency_dossier,
  slopsearx_get_artifact_lineage
• Resources: slopsearx://capabilities, slopsearx://capabilities/{engine},
  slopsearx://routing-profiles, slopsearx://health/summary
• Artifact references and bounded workflow lineage are documented in
  docs/ARTIFACT_LINEAGE.md.
• Supported artifact-to-workflow transitions, source lifecycle errors, and
  conflict rules are documented in
  docs/WORKFLOW_COMPOSITION.md.
• Specialist tools (jobs, security, science, research, saved searches and events, retrieval receipts,
  staged search, dependency dossiers) are disabled until
  the operator grants them (MCP_GRANT_JOBS=1, MCP_GRANT_SECURITY=1,
  MCP_GRANT_SCIENCE=1, MCP_GRANT_RESEARCH=1,
  MCP_GRANT_SAVED_SEARCHES=1, MCP_GRANT_RETRIEVAL_RECEIPTS=1,
  MCP_GRANT_SAVED_SEARCH_EVENTS=1,
  MCP_GRANT_STAGED_SEARCH=1, MCP_GRANT_DEPENDENCY_DOSSIER=1).
• Sensitive engines (hibp, dehashed) are unreachable from generic
  routing, categories, and intent profiles, and are rejected by every
  explicit-engine search path (generic explicit engines, targeted, jobs,
  security, science) unless the operator sets MCP_TARGETED_SENSITIVE_ALLOWED=1
  (deliberate, uniform policy boundary enforced by one shared gate).
• There is no separate "advanced search" tool. Richer include/detail
  (card vs. full record) semantics on the existing search/read tools and the
  per-engine capability matrix (slopsearx_list_capabilities) cover
  requires_* needs — see docs/MCP_SERVER.md §6.14.

Full installation, configuration, client setup (Claude Desktop, Cursor,
Hermes Agent — see "Hermes Agent (Nous Research)" in the guide — and
generic MCP clients), and agent usage guidance:
docs/MCP_SERVER.md.

Quick Start

Build from source

The Dockerfile requires a unique security-refresh value so cached builds cannot silently reuse an old Debian package layer:

DEBIAN_SECURITY_REFRESH="$(date +%s)" docker compose build

VPN and proxy deployments

Google and DuckDuckGo are best-effort HTML-scrape adapters. VPN, proxy, and
datacenter IPs can receive consent, challenge, or block pages; configure a
Brave API key (ENGINE_BRAVE_API_KEY) for a reliable API-backed web-search
source. A Brave key supplements the other active Tier-1 engines; it does not
disable them.

Optional Jev routing and result reranking

Set TYPESAFE_API_KEY to enable TypeSafe Jev specialist routing and result
reranking automatically. SlopSearX
keeps its ordinary general-engine base and asks Jev which eligible specialist
engines can add distinctive evidence for the query. Every specialist meeting
the 0.65 threshold is added; there is no arbitrary engine-count cap. Without
the key, or if Jev is unavailable, the existing deterministic routing path is
used unchanged. Explicit engine, category, and media scopes bypass Jev routing.

After retrieval, Jev orders a bounded result-card shortlist using ordinary Score,
across general and specialist source tiers. A configured reranker disables
specialist promotion, including when advice fails. Without
the key, or on invalid/unavailable reranking advice, results retain the configured
deterministic presence/RRF order. Providing the key opts into sending query,
bounded titles, sanitized URLs and snippets to TypeSafe; sensitive-engine scopes
are excluded. See Jev result reranking for payload bounds,
timeouts, caching, privacy and the replaceable provider contract.

See TypeSafe Jev specialist routing for configuration,
failure behavior, policy boundaries, and routing-card maintenance.

To surface successful scrape responses that parse to zero results, enable the
opt-in diagnostic flag. These entries appear in meta.empty_engines; they are
warnings rather than failures because a search can legitimately have no matches.

FEATURE_EMPTY_SCRAPE_DIAGNOSTICS=true

Pre-built Docker images are available from GitHub Container Registry for linux/amd64 and linux/arm64. Docker selects the native architecture automatically, including on Apple Silicon. Builds run on every push to main (latest and a short commit SHA) and on version tags (X.Y and X.Y.Z, plus a short commit SHA). Each architecture is built on a native runner, Trivy-scanned, and smoke-tested before the combined manifest is published. Pull requests validate both architectures without publishing. Existing deployment digest pins retain their original platforms until promoted to a new multi-platform image digest.

The repository's own deployment surfaces (docker-compose.yml, k8s/deployment.yaml) pin the image by digest, so they always run exactly the artifact that CI built, Trivy-scanned, and smoke-tested. To deploy the same verified artifact as a given commit, resolve its short-SHA tag to the current digest:

docker buildx imagetools inspect ghcr.io/magnus919/slopsearx:<short-sha> --format '{{json .Manifest.Digest}}'

# Pull and run with Valkey for caching and rate limiting
docker run -d --name valkey valkey/valkey:8-alpine
docker run -d --name slopsearx -p 8080:8080 \
  -e VALKEY_URL=redis://valkey:6379/0 \
  --link valkey \
  ghcr.io/magnus919/slopsearx:latest

# Try it
curl 'http://localhost:8080/search?q=hello+world&format=json'

Standard HTML, CSV, JSON, and RSS formats are controlled by search.formats
in config.yaml (or SEARCH_FORMATS=html,json). YAML is an additive
SlopSearX format and is always available. A requested standard format that is
not enabled returns HTTP 403; malformed search parameters return HTTP 400.

Browser portal

Open / for the human-facing search portal. A query submitted there uses the
same search service, routing, cache, ranking, and engine policy as the API; use
/search directly for a result page or for compatibility clients. The portal
defaults to Dark mode and provides a Darker mode toggle. Operators can
set SLOPSEARX_PORTAL_DEFAULT_THEME=darker to change the initial theme. An
explicit user choice is retained in browser storage when available. The portal
does not grant capabilities that the server-side policy would reject. Explicit
selection of a sensitive engine (currently hibp and dehashed by default)
also requires MCP_TARGETED_SENSITIVE_ALLOWED=true; the same operator policy
is applied before either the browser or MCP surface dispatches a search.

Public search remains account-free. Operators can separately enable the OIDC
and Valkey-backed /workflows supervisor console for tenant-scoped workflow
inspection and explicitly granted actions. The workflow routes are disabled by
default and do not change /, /search, or MCP contracts. See
docs/WORKFLOW_PORTAL.md for its direct HTTPS
deployment, identity, two-gate authorization, and rollback requirements.

See docs/PORTAL_DEPLOYMENT.md for the browser
URL map, proxy/access modes, digest-pinned deployment, smoke check, rollback,
and troubleshooting runbook.

License

MIT — see LICENSE.

Scholarly results are grouped by identified work before reranking, with bounded internal source provenance and optional SearXNG Paper metadata.

Caller-directed query planning previews decomposition, terminology variants and evidence-linked follow-ups without dispatch. See query planning.

The v5 PR review canary uses non-binding COMMENT mode and a pinned JavaScript publisher action. Its first publication and same-review replay were confirmed, and containment/re-enable checks succeeded. This configuration enables COMMENT, APPROVE, and REQUEST_CHANGES and disables only Droid Auto Review. It is deployed only when this change is merged after checks and review. Droid Tag and Wiki remain unchanged. Source coverage has been partial and semantic review accuracy is unknown. For run receipts, bounded history, and rollback, see PR review operations. No workflow automatically merges a pull request.