pickysteve

Picky about what he loads into context, including what he refuses to load.

▶ Watch the trailer

https://github.com/user-attachments/assets/8750946b-36be-4c48-bf73-79513451d1f5

PickySteve is a lightweight orchestration layer. A cheap model figures out which skill a request
actually needs, retrieves that one skill, and hands a small, focused, untrusted-data-boundaried
context bundle to a capable model. It does not dump every tool and document you own into context
on every request.

This repo is Phase 1 (MVP), built to an architecture spec. Phase 2 work (tracing platform,
standing eval harness, credential vault, sandbox) is not built yet. Each piece gets added only
when a real Phase 1 failure justifies it.

30-second quickstart

# from the repo root (uv 0.10+; on Windows the venv python is .venv/Scripts/python.exe — substitute it throughout)
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -r requirements.txt

# choose your model — local Ollama, OpenAI, Claude, OpenRouter, or any OpenAI-compatible endpoint
.venv/bin/python -m pickysteve.setup

# calibrate the reranker floor on the labeled set
.venv/bin/python eval/calibrate.py

# run one request
.venv/bin/python -m pickysteve "review my Rust endpoint for security and REST design"

Bring your own model. python -m pickysteve.setup asks which model to use and saves it.
Runs on anything that speaks the OpenAI-compatible API: local Ollama (offline, no key),
OpenAI, Claude, Gemini, Llama, etc. via OpenRouter / LiteLLM / their native
compat endpoints. The published benchmarks were measured on local qwen3:8b; a different model
just needs a eval/calibrate.py re-run.

Note: this is currently a uv / git clone install. There is no PyPI package yet, so
uvx pickysteve and pipx install pickysteve do not exist. If that changes, this section
gets a one-liner. For now, the fastest path to a real coding agent is the connector installer
below.

Wire it into your agent (one command)

python -m pickysteve.connectors.install --list   # see which of 18 agents are detected
python -m pickysteve.connectors.install --all    # wire every detected agent (backs up configs first)

Supports Claude Code, Codex, Cursor, Windsurf, Cline, Roo Code, Gemini CLI, Qwen Code, Goose,
OpenHands, GitHub Copilot, Kimi Code, OpenCode, ZeroClaw via MCP stdio, and Aider, Hermes,
OpenClaw, NanoClaw via an OpenAI-compatible proxy on :8077/v1. Full per-agent config snippets
and the connectivity matrix are in INTEGRATIONS.md.

How it works

flowchart TD
    A[Request] --> B[Security Gate\nscan raw request]
    B -->|clean| C[Router\ncheap model → search query]
    B -->|injection| X1[Abort]
    C --> D[Retrieval\nBM25 + embeddings, RRF fused]
    D --> E[Security Gate\nscan every retrieved doc]
    E -->|clean| F[Rerank\ncross-encoder vs original request]
    E -->|poisoned| X2[Abort / drop candidate]
    F --> G[Floor + Dedupe\nbelow floor → clarify, don't guess]
    G --> H[Knowledge Graph\nconfused_with edges + distinguishers]
    H --> I[Judge\nLLM reads full skill bodies + KG notes]
    I --> J[Compat Check\nflag conflicts, don't merge]
    J --> K[Assembly\nnonce-wrapped untrusted-data boundary]
    K --> L[Execution\ncapable model does the work]
    L --> M[Log\nfull trace to logs/runs.jsonl]

Ten stages: gate, route, retrieve, gate again on retrieved content, rerank, floor/dedupe,
knowledge-graph context, judge, compat-check, assemble, execute, log. The second gate pass scans
every retrieved candidate, not just the user's request. Most similar projects skip that pass, and
it is the highest-risk surface: a poisoned skill doc is attacker-controlled content sitting right
next to your execution model.

The stack (and why)

Total Phase-1 dependencies: stackone-defender, rank-bm25, sentence-transformers,
openai, numpy. That is the minimal set the spec prescribes.

Two decisions the spec left open (decided and documented)

• Retrieval unit (§2.3): each markdown file is one retrieval unit. A skill folder with
  several files (see registry/rag-architecture/) yields multiple units sharing a skill_id.
  After reranking, units from the same skill collapse to the best one in assembly, so the
  execution model never receives three chunks of one skill.
• Gate policy on a poisoned retrieval (§2.1): default RETRIEVED_INJECTION_POLICY=abort. If
  a retrieved candidate trips the gate (high-risk), the whole request aborts. The documented
  alternative is drop, which discards just that candidate and continues. For allowed-but-
  sanitized content, the pipeline uses the Tier-1-sanitized text downstream (defense in depth)
  and logs that sanitization happened.

Refinements after a 21-agent adversarial review

The first validation surfaced three failures. Fixing them, and adversarially reviewing the
fixes, added these mechanisms. See FINDINGS.md for the full before/after.

• Tier-3 escalation (gate, request path only): a legitimate question about prompt injection
  was being blocked. The request gate now enables the defender's Tier-3 LLM hook over the gray
  band [0.64, 0.85), just above the model's calibrated 0.64 block threshold. A cheap
  adjudicator can rescue a would-be block but never flip a would-be allow, while near-certain
  attacks (≥0.85) still hard-block without consulting it. Retrieved third-party content never
  escalates (strict gate).
• Multi-intent router with §2.4-safe rescue: the router emits sub-queries and retrieval
  unions across them for recall. Reranking stays governed by the original request (§2.4). Only a
  genuinely compound request (two or more distinct sub-intents) also maxes over its sub-queries,
  to surface a secondary intent the full-request score would bury.
• Relative-dominance gate: a secondary skill is kept only if it scores at least
  DOMINANCE_RATIO (0.08) times the top skill. This keeps PickySteve picky instead of dumping
  marginal tag-alongs.
• Honest #13 fix: a correct skill that the reranker under-scored was fixed by enriching the
  skill doc with real symptom vocabulary, not by lowering the floor onto leaked data. The floor
  is recalibrated on a leakage-free labeled set with hard-negatives.

Benchmarks

All numbers below come from this repo's own eval docs and logs.

Routing accuracy, the trifecta (DEEP_CONTEXT.md):

The heldout2 set is kept deliberately hard and unsaturated. New confusable-pair tasks get added
faster than the router/rerank stack is re-tuned, so it acts as a running canary for regressions
rather than a suite that is expected to hit 100%.

On a 40-request held-out accuracy set with no calibration overlap (TEST_REPORT.md):
90% overall correct, 100% top-1 accuracy (30/30 answerable), 96.7% full recall,
MRR 1.000, 100% off-domain rejection (haiku/recipe requests correctly get no_confident_match).

Two-tier gate (recall-all + conformal abstention). The cheap local judge routes singleton
predictions directly; ambiguous cases escalate to a frontier judge (logs/two_tier.out):

Security, red-team detection (SECURITY_AUDIT.md, TEST_REPORT.md):

• 180-payload corpus (129 attack / 51 benign, 14 evasion families): 100% attack detection,
  zero bypasses after hardening. The baseline was 86%.
• Separate 115-attack corpus: 97.6% detection on the request path, 96.5% on retrieved
  content, up from 87.1%. Benign false-positive rate held at 0.0% throughout.
• On the adversarial 180-payload corpus, benign-allow rate is 61% (39% false-positive on
  deliberately tricky security-flavored prompts). On the real skill registry, false positives
  are 0/43, verified by a startup-time warm pass the server refuses to serve without.

Trap-registry ranking test (SIM_REPORT.md): 24 skills built to confuse a
naive matcher, 14 tasks. The gold skill outranked every trap 13/13 (100%), correct top-1 on
12/13 answerable tasks, correct no-match handling 1/1.

Why not just RAG or LangGraph?

• It does not retrieve everything and let the model sort it out. The floor, dedupe, and
  dominance-ratio gate exist so the execution model never sees marginal tag-along documents. The
  goal is picking one thing, not five plausible things.
• It is not a bigger orchestration framework. There is no state machine and no LangGraph-style
  graph runtime. Phase 1 is five Python modules (retrieval.py, rerank.py, router.py,
  security_gate.py, pipeline.py). See "Phase 1 non-goals" below for what is left out (no
  knowledge-graph-as-default, no standing eval harness, no sandbox) until a real failure justifies
  adding it.
• The security gate is not a bolt-on. Most RAG setups treat retrieved documents as trusted
  once they clear a similarity threshold. PickySteve scans retrieved content through the same
  injection gate as the user's request, fail-closed, before it ever reaches assembly.

[!IMPORTANT]
Two-scan, fail-closed by design. Every request is scanned twice: once raw before routing,
and once per retrieved candidate before assembly. Either scan can abort the request or drop a
single poisoned candidate. On timeout, error, or an ambiguous LLM adjudication, the gate fails
closed. Nothing ambiguous reaches the execution model silently.

[!IMPORTANT]
Untrusted content never becomes instructions. Retrieved skill docs are wrapped in a
per-call random-nonce boundary (<<UNTRUSTED-{nonce}>>...<<END-{nonce}>>) before being handed
to the execution model, so a poisoned doc cannot forge a [SYSTEM]: directive or close the
boundary early. This was hardened after a real finding: static delimiters were forgeable by a
crafted skill body (see SECURITY_AUDIT.md, "assembly.py" row).

Live visualizer: open assets/pickysteve_live.html in a browser
to watch Steve step through gate, router, retrieval, rerank, judge, and assembly on a sample
request. For the terminal-native version, eval/run_examples.py streams the same stage-by-stage
trace to logs/runs.jsonl as it drives the 18 example requests end to end.

Core principle

Confidence and relevance scores measure topical similarity, not correctness. Nothing here claims
a retrieval was right, only that it was plausible. All retrieved content is treated as low-trust
data, never as instructions.

Known limitations

[!WARNING]
PickySteve can make mistakes. Do not trust it blindly on critical tasks. It routes to a
plausible skill, not a guaranteed-correct one. Review what it picks before acting on it.

• qwen3-calibrated thresholds. The reranker floor and the Tier-3 escalation gray band are
  calibrated against qwen3:8b as router/judge. Swapping the local model requires re-running
  eval/calibrate.py. Thresholds are not portable across judges by assumption.
• Latin-script non-English prompt injection residual. Spanish-language injection can still
  bypass the bundled English-only classifier in some cases. This is a gap in the bundled ONNX
  model, not a logic bug in the gate wiring.
• Heldout2 residual item. One genuine miss (task #12, a compound canary/feature-flag-vs-
  blue-green case) where the trap outranked gold. See the benchmarks table above and
  logs/heldout2_final_run.log for the full trace.
• The reranker (bge-reranker-base) takes roughly 2s per 8 candidates on CPU and dominates
  end-to-end latency.
• The router can occasionally over-decompose a single intent into multiple facets, surfacing a
  marginal secondary skill.
• Five spec-level open logic gaps remain by design (see below) and TEST_REPORT.md §6.

Open logic gaps (carried forward from the spec)

1. Confidence is not correctness. The rerank score is topical similarity, not outcome quality. There is no outcome feedback loop; that needs labeled real results over time.
2. The router can be wrong. Intent decomposition for vague or compound requests is a hard reasoning problem.
3. Skill-conflict resolution is unsolved. The compat check flags conflicts rather than resolving them.
4. "Compatible skills can be combined" has no concrete definition. There is no automatic skill-merging.
5. No recency or trust weighting in retrieval. A stale skill ranks the same as a fresh one at equal relevance. Staleness is flagged, not down-weighted.

Phase 1 non-goals (intentionally absent)

No knowledge graph or LightRAG as the default path, no LangGraph or state-machine framework, no
external tracing (Laminar/Langfuse), no credential vault, no automated eval harness
(DeepEval/Ragas), no sandbox runtime. Each is added in Phase 2 only when a real Phase-1 failure
justifies it.

FAQ

How is the judge not gamed?
Two independent judges run across the eval suites: a local qwen3:8b, and a Claude-blind-judge
mode where Claude picks the root-cause skill without seeing the labeled answer. When they
disagree, it is informative. On the hardest adversarial subset, Claude scored lower (86%) than the
local judge (91%) because it disputed a couple of debatable labels. That divergence is treated as
a signal the label is ambiguous, not proof the judge is wrong (DEEP_CONTEXT.md).
Every model call in the eval pipeline is cached, so a given pass rate is deterministic and
reproducible.

Isn't this just RAG?
Retrieval is one stage out of ten. The stages that are not RAG (the two security gates, the
reranker floor/dominance-ratio gate, the compat check, and the nonce-wrapped untrusted-data
boundary) are where most of the engineering and most of the fixed bugs went. Plain RAG does not
refuse to answer when nothing clears a calibrated floor, and it does not re-scan its own retrieved
documents for injection before use.

What happens on no confident match?
PickySteve returns no_confident_match and asks a clarifying question instead of guessing. The
floor is calibrated on a labeled good/bad set rather than hand-tuned, and the documented
philosophy is that confidence measures topical similarity, not correctness. When nothing clears
the bar, a question beats a wrong pick. Off-domain rejection has tested at 100% across every
held-out suite.

Does it work with tools that don't support MCP?
Yes. An OpenAI-compatible proxy (pickysteve.connectors.http_server, port 8077) sits in front of
any tool that takes a custom OpenAI base URL (Aider, Hermes, ZeroClaw, OpenCode, OpenClaw,
NanoClaw). There is also a REST /pick endpoint and a direct Python import for anything else. See
INTEGRATIONS.md for the full connectivity matrix across 18 agents.

What are the known limitations?
See the "Known limitations" section above and TEST_REPORT.md §6.

Setup

# from this directory (uv 0.10+, Ollama with qwen3:8b running locally)
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python -r requirements.txt

On Windows, the venv python lives at .venv/Scripts/python.exe; substitute it in every
command below.

Use

# 1) Calibrate the reranker floor on the labeled set (writes eval/calibrated_floor.json)
.venv/bin/python eval/calibrate.py

# 2) Run a single request
.venv/bin/python -m pickysteve "review my Rust endpoint for security and REST design"

# 3) Run the 18 example requests end to end (traces -> logs/runs.jsonl)
.venv/bin/python eval/run_examples.py          # add --no-exec to skip the execution model

# 4) The mandatory security-gate test
.venv/bin/python tests/test_security_gate.py

Config is all environment variables (PS_*). See pickysteve/config.py.

Connect it to your coding agent

# MCP (Claude Code, Codex, Cursor, Windsurf, Cline, Roo, Gemini CLI, Qwen Code, Goose, ...):
.venv/bin/python -m pickysteve.connectors.mcp_server      # exposes pick_context + list_skills

# OpenAI-compatible proxy (Aider, Hermes, ZeroClaw, ...): point the tool's base URL at :8077/v1
.venv/bin/python -m pickysteve.connectors.http_server     # /pick + /v1/chat/completions

Full per-agent config snippets, the one-command installer, and the Obsidian second-brain export
(python -m pickysteve.connectors.obsidian --vault <path>) are documented in
INTEGRATIONS.md.

Contributing

See CONTRIBUTING.md for dev setup, the eval/test suite layout (what is a
fast pre-commit check vs. what needs a live Ollama), the rule that any routing-affecting change
must re-run the base/harder/held-out trifecta before merge, and code-style conventions.

Credits

Trailer music: "Powerful Emotional Trailer" by MaxKoMusic, via Chosic,
licensed under CC BY-SA 3.0.

License

MIT. See the LICENSE file for the full text.