ILTER — AI Gateway

Deploying AI models to production brings critical operational challenges such as unpredictable costs, data privacy risks, and API outages. ILTER is an independent gateway that sits between your application and AI providers; transforming your entire AI traffic into a secure, optimized, and fully controlled infrastructure without requiring any architectural changes in your code.

• 🧩 MCP Gateway & Marketplace: Connect AI to your APIs, CRM, or MCP servers — zero changes required.
• ⚡ Smart Router: Every request is scored for complexity in real time and routed to the right model tier.
• 🌐 Smart Fallback: Provider down or rate-limited? ILTER fails over to another key or provider, no changes.
• 💰 Budget Control: Hard daily/monthly spending limits per key — traffic cuts off the instant a limit is hit.
• 🪪 PII Guard: Emails, SSNs, and other sensitive data are masked before they ever leave your network.
• 🧠 Semantic Cache: Queries are served from cache — vector search with SHA-256 exact-match fallback.
• 🛡️ Prompt Guardrails: Blocks injection, toxic content, and off-topic requests before they reach the model.
• 🌀 Agent Loop Detector: Catches loops by rate, fingerprint, cost, or depth before burning budget.
• ⏱️ Cron Engine: Schedule recurring AI workflows with standard cron expressions — no external queue needed.
• 📡 Observability: Prometheus metrics and OpenTelemetry tracing, ready for Grafana, Datadog, or Honeycomb.
• 📊 Dashboard: Embedded web UI for KPIs, audit logs, API keys, and built-in chat playground — no server.

All this infrastructure requires no external server, Node.js/Python environment, or complex dependencies. It comes as a zero-setup, single static executable (binary) ready to run in seconds.

⬇️ Download the latest release

./ilter serve
# Proxy: http://localhost:8181/v1/chat/completions
# Dashboard: http://localhost:9191
# Metrics: http://localhost:9192/metrics

Dashboard

Overview Live KPIs, daily cost trend, provider/model breakdown, and a one-click toggle hub for every feature.	Chat Built-in playground to test prompts against any registered model without leaving the dashboard.	MCP Marketplace Browse and one-click install community MCP servers — filesystem, GitHub, Postgres, and more.
Smart Router Manage routing strategies — cost-first, quality-first, or custom rules — and switch the active one live.	Logs Full request history with cost, latency, and status, filterable by time range and provider.	Jobs Cron-scheduled AI workflows with next/last run status, right from the embedded scheduler.

Features

Every business integrating AI faces uncontrolled costs, privacy concerns, and unpredictable agent behaviors. ILTER solves these directly at the gateway level.

🧩 MCP Gateway & Marketplace

Connecting AI to your internal APIs or CRM usually requires heavy client-side modifications.

• Zero Client Changes: Automatically injects tools from registered MCP servers into every chat completion request.
• Interception: ILTER catches the tool_call, executes the MCP server tool locally (stdio/SSE), and returns the result to the model.
• OpenAPI Bridge: Convert any REST API to an MCP tool list instantly.
• OAuth PKCE Support: Standard OAuth PKCE authorization endpoints (/.well-known/*, /authorize, /token, /register) on port 8181 for remote MCP clients.
• Tri-Protocol Support: Speaks all three published MCP revisions — 2024-11-05, 2025-03-26, 2026-07-28 — on both inbound (as MCP server) and outbound (as MCP client) sides, negotiating newest-first and bridging whatever versions each side actually supports. See docs/architecture.md for the negotiation rules, per-version OAuth policies, and the async Tasks engine.

⚡ Smart Router

Using expensive models for every simple question wastes engineering time and money. ILTER routes requests dynamically based on context.

• Real-Time Scoring: <0.03ms heuristic complexity score (0–100) based on word count, reasoning-required phrases, code blocks, and tool calls.
• Automatic Tier Selection: Routes dynamically between economy, standard, and premium models.
• Custom Overrides: E.g., complexity > 50 → gpt-4o or prompt contains "analyze" → claude-sonnet.
• Load Balancing: Weighted round-robin, cost-optimized, latency-optimized, and priority-based strategies across healthy provider circuits.

🌐 Smart Fallback

Don't get locked into a single vendor or disrupted by provider outages.

• Single Endpoint: Hides 9 providers behind one uniform OpenAI-compatible API.
• Circuit Breaker: RetryTransport → CircuitBreakerTransport. 5 consecutive failures open the circuit and trigger automatic fallback.
• Seamless Failover: Switching from OpenAI to Anthropic during an outage requires zero code changes.

🧠 Semantic Cache

• Vector Search: Redis Stack-backed similarity search to catch and serve repetitive queries.
• Exact-Match Fallback: SHA-256 caching works even without an embedding model.
• Async Operations: Cache writing does not block the primary request path.

🛡️ Prompt Guardrails

• Security: Prompt injection detection and toxicity filtering.
• Topic Control: Keyword-based and custom regex rules to block specific topics.
• Actions: Set per-rule severity to block, warn, or mask.

💰 Budget Control

Per-token pricing creates end-of-month surprises. ILTER tracks costs in real-time for each API key and enforces hard limits.

• Hard Kill Switch: Configurable daily and monthly spending limits (USD). Returns 429 budget_exceeded the millisecond the limit is breached.
• Warning Thresholds: Default 80% usage alerts.
• Real-time Tracking: Cost breakdowns by model, provider, and individual keys via the dashboard.

🪪 PII Guard

Sending customer data to cloud providers violates GDPR and HIPAA. ILTER masks this data before it ever leaves your network.

• Triple-Layer Engine: Bloom Filter + Aho-Corasick Trie + Regex working together.
• Detected Types: Names (EN/TR), Emails, Phones, SSN/National IDs (TCKN), Credit Cards (Luhn validated), IPs.
• Three Modes: Mask ([PII_EMAIL]), Reversible (PII:EMAIL:a8b9f1 — restored in response/SSE stream), or Block (422 pii_blocked).
• Ultra-Fast: <0.04ms latency overhead with <5MB memory footprint.

🌀 Agent Loop Detector

Agentic workflows can enter infinite loops, generating massive bills overnight. ILTER catches them while you sleep.

• Rate Limit: Blocks if >30 requests/sec.
• Fingerprint Match: Blocks if the exact same prompt is sent >5 times in a 20-request window.
• Cost Accumulation: Blocks if >$5 is spent within 5 minutes on a single session.
• Session Depth: Throttles if a single session exceeds 100 requests.

⏱️ Cron Engine

Running periodic AI tasks (summaries, reports) usually requires external queues and workers. ILTER embeds this directly in the binary.

• No External Queue: Uses standard cron expressions (e.g., 0 9 * * 1).
• Workflow State: Pass data between steps using {{.Input}} and {{.prev}}.
• Robust Execution: Webhook triggers, exponential backoff, and dead-letter queues included.

📡 Observability

• Metrics: Native Prometheus metrics on /metrics (Port 9192).
• Tracing: OpenTelemetry (OTLP push) ready for Grafana Cloud, Datadog, or Honeycomb.
• Audit Logs: Full request logs, cost trends, and PII event indicators.

📊 Others

No npm, no build step, no separate server. A modern web UI embedded directly into the Go binary via embed.

• Overview: KPIs, monthly spend, routing decisions, and feature toggle hub.
• Chat Playground: Use any registered model directly from the dashboard.
• Management: Create API keys, configure thresholds, register MCP servers, and manage Cron jobs. (Accessible at Port 9191).

Quickstart

# Requires nothing — it just works
./ilter serve

For the interactive setup wizard:

./ilter init

To see the dashboard filled with mock costs, requests, and PII events:

./ilter init --demo && ./ilter serve

Your first real request (Point your existing OpenAI SDK to ILTER):

curl -X POST http://localhost:8181/v1/chat/completions \
  -H "Authorization: Bearer <ilter-api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Hello!"}]}'

Supported Providers

Architecture

Your Application
    │  OpenAI API format
    ▼
ILTER :8181
    │
    ├─ Auth              — ilter-xxxx key (Argon2id/SHA-256 + LRU cache)
    ├─ Rate Limiter      — Redis / in-memory token bucket, RPM/TPM per key
    ├─ Budget Enforcer   — Monthly spend limit, hard reject
    ├─ Prompt Injection  — Inject system prompts from DB
    ├─ PII Masker        — Bloom + Aho-Corasick + Regex, <0.04ms
    ├─ Guardrails        — Injection detection, topic blocking
    ├─ MCP Inject        — Tool injection + tool_call interception + OAuth PKCE
    ├─ Smart Router      — Real-time complexity scoring & tier selection
    ├─ Loop Detector     — Rate / fingerprint / cost / session
    ├─ Semantic Cache    — Redis Stack vector search + SHA256 fallback
    └─ Provider Router   — Weighted round-robin, circuit breaker, fallback
           │
           ├─ OpenAI / Anthropic / Gemini / DeepSeek
           ├─ OpenRouter / Ollama / Qwen / OpenCode
           └─ ...

ILTER :9191  →  Dashboard (Astro + React, Go embed)
ILTER :9192  →  /metrics  (Prometheus, OpenTelemetry bridge)

• Language: Go 1.26.3, single binary, CGo-free (CGO_ENABLED=0), goroutine concurrency.
• Router: chi v5.
• Database: SQLite — modernc.org/sqlite, pure Go, WAL mode.
• Cache: Redis Stack 7+ — optional, graceful degradation.
• Config: Compiled defaults + ILTER_* env vars — no configuration file required.

→ Architecture details: docs/architecture.md
→ Design decisions & FAQ: docs/faq.md
→ Gateway comparison: docs/comparison.md

Docker & Container Deployment

# Single container (scratch-based image, <20MB)
# Admin key + a provider key are enough to boot straight into `serve` — no
# `ilter init` step needed. The SQLite DB is created on first start, and the
# provider whose ILTER_PROVIDER_<NAME>_API_KEY is set is enabled automatically.
docker run -d \
  -p 8181:8181 -p 9191:9191 -p 9192:9192 \
  -v $(pwd)/data:/app/data \
  -e ILTER_ADMIN_API_KEY=<your-own-random-secret> \
  -e ILTER_PROVIDER_OPENAI_API_KEY=sk-... \
  ykocaman/ilter:latest

# Full local stack: ILTER + Redis Stack + Ollama
docker compose up -d

Without ILTER_ADMIN_API_KEY and a provider key both set, serve refuses to start — there'd be no way to authenticate or route requests — and exits with a message telling you to run ilter init or set both.

Image size: <20MB (3-stage build: Bun web → Go UPX → empty scratch base image). This applies to the Docker image specifically — the plain binary from Releases (no UPX) is ~35-40MB.

Configuration

Zero configuration by default. Three levels of overrides:

→ All options: docs/configuration.md

CLI

Development

make build      # Go binary + web assets
make check      # Build + lint (Go + web)
make test       # go test -race -count=1 ./...
make fix        # gofumpt + biome format

Contribution guidelines: CONTRIBUTING.md

Performance

Apple M4, Go 1.26.3:

License

Apache 2.0 with Commons Clause — see LICENSE.