openevidence-tools

繁體中文版 → README.zh-TW.md

Two pieces that work together to use OpenEvidence as an
evidence source you can actually trust:

1. An MCP server that exposes OpenEvidence to any MCP client (Claude Code, etc.) as four
   tools — ask_openevidence, list_recent_oe_queries, fetch_oe_result, ask_oe_followup.
2. The /oe-verify skill (optional, recommended), which cross-checks every citation in
   an OE answer against primary sources and flags what OE left out. The MCP server works without
   it; installing it is one copy into your skills folder — see
   Optional: the /oe-verify skill.

Why both? OpenEvidence answers clinical questions fast, but it is a RAG pipeline (vector
search + LLM synthesis), and its citations can be wrong even when the underlying claim is real.
The classic failure is transitive citation: OE retrieves a chunk of a review's full text
in which the review quotes another trial's number, then credits that number to the review
itself. The claim is true; the attribution is false. Never absorb an OE answer unverified —
that is the whole reason /oe-verify exists.

What the MCP server exposes

preserve_citations=True — the provenance channel /oe-verify consumes

With preserve_citations=True, the answer keeps its inline [N] markers and appends a
## Citation Data JSON block. Each entry carries the metadata the verifier needs — title,
doi, pmid, journal, authors, publication_types, impact_score, recency_days,
is_guideline, url, plus two provenance fields:

• origin — the RAG corpus the cited chunk was retrieved from. OE ships this ROT-1
  obfuscated (every byte shifted +1, so _ arrives as a backtick); the server decodes it, e.g.
  mbodfu`gvmmufyu`tdsbqfe`w3 → lancet_fulltext_scraped_v2.

• origin_risk — a transitive-citation risk tier derived from origin:

  Tier	Corpora	Meaning
  HIGH	*_fulltext_scraped_* (lancet, nejm, aan, …)	Scraped review full text — the quoted sentence is often the review citing someone else's trial. This is OE's main failure mode.
  MEDIUM	wiley_cdsr_fulltext (Cochrane), wiley_research_fulltext, guidelines_fulltext_*, media_annotated_gemini	Publisher full text / AI-annotated figures — real chunks, still not necessarily the paper's own finding.
  LOW	pubmed_abstracts_*	Abstract-level; little room to mis-attribute.
  UNKNOWN	no span metadata	Provenance unavailable — treated as MEDIUM, never as LOW.

/oe-verify uses origin_risk to aim its attribution check: a HIGH/MEDIUM chunk backing a
quantitative claim is the danger zone, and each such citation gets traced back to whether
the number is really that paper's own finding.

One paper can legitimately hold several [N] indices — OE indexes per retrieved chunk, not
per paper (observed: 10 indices over 7 papers). The server maps markers to citation objects
positionally (the i-th [N] occurrence == the i-th citation object in document order) and, on
any count mismatch, degrades explicitly rather than guessing.

The /oe-verify skill

Optional — the MCP server runs without it. To install, see
Optional: the /oe-verify skill.

A three-layer verification pass over one OE answer (skills/oe-verify/SKILL.md):

• Step 0.5 — existence gate: every DOI is checked against CrossRef before any semantic
  work. A DOI that doesn't resolve is FABRICATED; no agent tokens are wasted on it.
• Layer 1 — per-citation accuracy (parallel cheap agents): look each paper up in
  Semantic Scholar / PubMed, compare what OE claimed against what the source says, and — for
  HIGH/MEDIUM-provenance quantitative claims — run the attribution check that catches transitive
  citation.
• Layer 2 — completeness (one stronger agent): anchor on the best recent systematic
  review / meta-analysis and ask what OE omitted, in both directions, plus an applicability
  check against your own practice context.
• Layer 3 — internal contradiction scan: do the papers OE itself cited actually agree with
  each other? Only runs when ≥2 papers make quantitative claims on the same outcome.

The output is a compact Markdown report with a per-citation verdict table (✅ accurate, ⚠️
overstated, ❌ misrepresented, 🚫 fabricated, ↗️ transitive, 🔄 secondary-untraced, ❓
unverifiable) and an evidence-landscape summary.

Requirements

• An OpenEvidence account (OpenEvidence is intended for clinicians). You log in yourself,
  once, in the browser window the server opens for you.

• Edge, Chrome or Chromium installed — on Windows, macOS or Linux. The server does not
  scrape or spoof anything: it runs each OpenEvidence API call as a page-context fetch()
  inside a logged-in openevidence.com tab, so every request uses your own account and
  session — exactly what would happen if you clicked in the tab yourself. OpenEvidence has no
  public API and its session cookie is httpOnly, which is why an in-tab fetch (rather than a plain
  HTTP client) is used at all.

  By default (transport cdp) the server starts a dedicated copy of your browser on its own
  profile (~/.openevidence/browser-profile) and drives it over the Chrome DevTools Protocol
  on 127.0.0.1:9222, through Playwright. It is launched on demand and left running; nothing is
  downloaded, and your everyday browser profile is never touched. No extension, no third-party
  daemon.

• Python 3.10+ and the deps in requirements.txt (mcp, pinned <2 — see below — and
  playwright). Only the Python package is needed: skip playwright install, it attaches to
  the browser you already have.

mcp must stay below 2.0. mcp 2.0 removed mcp.server.fastmcp, so an unpinned install
resolves to 2.x and server.py dies at import with ModuleNotFoundError — which an MCP client
only ever surfaces as Failed to connect — Connection closed, sending you hunting in the wrong
place. requirements.txt pins it; if you install by hand, pin it too.

🔒 Log in to OpenEvidence only, in the dedicated browser. While it runs, any program on
your own machine can drive that browser through its debugging port (the port listens on
127.0.0.1 only, never on the network). That is exactly why it gets a separate profile: keep
it for OpenEvidence, and do not sign in to your bank, mail or anything else there.

Versions up to 0.3.x drove your everyday browser through
Kimi WebBridge (a browser extension plus a local
daemon on http://127.0.0.1:10086/command). It still works: set OE_TRANSPORT=http, or set
OE_BRIDGE_URL to any daemon that speaks the same {action, args, session} JSON contract
(setting OE_BRIDGE_URL selects this transport by itself, so existing configurations keep
working). Note there are unrelated third-party projects using the name kimi-webbridge, some of
which expose an unauthenticated port 10086; this means the official one. The daemon auto-starts
only on Windows (or wherever OE_BRIDGE_BIN points at a binary that accepts start); on
macOS/Linux start the bridge yourself. A debugger-based bridge holds your logged-in identity on
every site at once — if you use one, consider
kimi-webbridge-lockdown to fence your
bank, brokerage and mailbox out of its reach.

Configuration (environment variables)

Install

pip install -r requirements.txt

# Log in to OpenEvidence once, in the dedicated browser this opens:
python mcp/server.py --login

# Register with an MCP client, e.g. Claude Code:
claude mcp add openevidence -- python /path/to/openevidence-tools/mcp/server.py

# Optional, recommended: install the /oe-verify skill (next section).

The login is kept in the dedicated profile; repeat --login only if OpenEvidence logs you out.
From then on the server is self-healing: it launches the browser if it is closed, reconnects if
it was restarted, and opens/reuses an OpenEvidence tab as needed. A Not authenticated error
means the session expired — run --login again.

Check the transport on your machine without touching OpenEvidence (it starts a throwaway browser
profile against a local stand-in server, then closes it):

python tests/smoke_cdp.py              # Linux without a display: xvfb-run -a python tests/smoke_cdp.py
python -m unittest discover -s tests   # unit tests, no browser needed

Both run in CI on Windows, macOS and Linux.

Optional: the /oe-verify skill

The skill is not needed to use OpenEvidence through the MCP server, but it is what makes an
answer safe to rely on (see why). It is a single folder,
skills/oe-verify/; installing it means copying that folder to where your client loads skills.
--login reminds you of this until it is installed.

For Claude Code, as a personal skill available in every project:

# macOS / Linux
mkdir -p ~/.claude/skills && cp -r skills/oe-verify ~/.claude/skills/

# Windows (PowerShell)
New-Item -ItemType Directory -Force "$HOME\.claude\skills" | Out-Null
Copy-Item -Recurse skills\oe-verify "$HOME\.claude\skills\"

To make it available in one project only, copy it to <your project>/.claude/skills/ instead.
Start a new Claude Code session afterwards; then run /oe-verify after an ask_openevidence
answer. The copy does not update itself: after git pull, copy the folder again (or symlink it
instead of copying). Other MCP clients: put the folder wherever that client discovers skills.

Responsible use

• Your own account, your own session. This tool automates your logged-in browser tab. Use
  it only with an account you are entitled to use, and follow OpenEvidence's terms of service.
• Respect the quota — it is a gap, not a bucket. Since the 2026-09 site revamp OpenEvidence
  no longer throttles by "N per hour" but by a long minimum gap between submissions, and a
  throttled request appears to re-arm that gap rather than being free. Measured on one account
  2026-09-02: submissions ≥11 min apart all succeeded (4/4 at 11, 13, 14 and 16 min); a 6-min gap
  failed, as did every burst fired within a couple of minutes. So the server serialises
  submissions and paces them at OE_SUBMIT_SPACING_SEC (default 700s), persisting the last
  attempt to OE_DATA_DIR/submit_state.json so that restarting the server does not reset the
  clock. Only new questions and follow-ups are paced; re-fetching a prior answer is free.
• Never retry straight after a 429. OE's own Expected available in 0 seconds is always 0 and
  cannot be used. Retrying on it does not recover — it extends the lockout. The server therefore
  reports the time to come back instead of retrying, and you should do the same by hand.
• Verify before clinical use. Run /oe-verify (or otherwise check the primary sources) before
  you rely on an OE answer. That is the entire point of this repo.
• Not medical advice. These tools help a clinician evaluate evidence; they do not replace
  clinical judgment, and their output is not medical advice.

When OpenEvidence changes its site

There is no public API here — the server talks to the same private endpoints the website
talks to, so a site revamp can break it. The 2026-08-26 revamp (new sidebar, Discover,
Tools, voice mode) broke every part of the path at once. All five faults are fixed in 0.3.0,
and they are worth writing down because the same shapes will recur:

<follow-up-question> / <follow-up-topic> / <visual> are new wrapper tags too; the old
"Would you like to explore" sentence match no longer fired, so that trailing chrome was
leaking into the answer body. Both forms are now stripped.

If something breaks again, the fastest route is the one used here: watch what the website
itself sends from your own browser's network panel, and diff it against what this server
sends.

Credit where it's due — htlin222

This project started independently: I wanted OpenEvidence answers to flow into my note-writing
workflow, and the first version drove the site with my own browser automation. That version died
to bot detection, and it was while looking for a way forward that I found
Hsieh-Ting Lin (htlin222)'s work. Much of what this repo does
well now comes from his more recent updates, and it is worth being specific about what came from
where.

From audit-oe-skill (MIT) came the core insight
this whole repo is built around: that OpenEvidence's origin field is ROT-1 obfuscated, that it
names the RAG corpus a chunk was retrieved from, and that the corpus therefore predicts how
likely the citation is to be mis-attributed. He also named the failure mode — transitive
citation, where a review quotes another trial's number and OE credits the review — and built
the verification architecture we adopted: one cheap agent per citation in parallel, a trace-back
stage for quantitative claims in review full text, and a single stronger agent scanning the cited
papers for contradictions with each other, graded CRITICAL / WARNING / NOTE. Our origin_risk
tiers are his risk table; our Layer 1 and Layer 3 are his Phase 3 and Phase 4. When you read our
verification report, most of the questions it asks are his questions.

From openevidence-mcp (Apache-2.0) came the
transport idea that got us unstuck: OpenEvidence has no public API and an httpOnly session cookie,
so don't scrape it — run the call inside the clinician's own already-logged-in browser tab. Our
implementation is our own (and now drives a dedicated browser over the DevTools Protocol
rather than an extension), but the direction was his.

We wrote our own tool layer rather than forking, for a practical reason: our verifier needs
publication_types, impact_score, recency_score, is_guideline and per-chunk origin
returned through the MCP contract itself, and we drive the browser through a general-purpose
channel rather than a dedicated extension. We also added a completeness layer his skill doesn't have (what did
OE omit?), a CrossRef existence gate, and a marker↔citation index mapping our architecture
needs and his does not. Those are additions to his foundation, not a replacement for it.

skills/oe-verify/ is derived from audit-oe-skill, MIT © 2026 Hsieh-Ting Lin; see
NOTICE. Thank you, Hsieh-Ting.

Works with textbook-to-note

/oe-verify is designed to plug into the
textbook-to-note note-writing workflow as its
optional evidence-enrichment stage: when a note's management/treatment section needs current
guidance, ask_openevidence fetches it and /oe-verify gates it before it enters the note.
Standalone use — just asking OE questions and verifying the answers — is equally fine.

🌱 Start here if you're new to AI agents ／ AI agent 新手起點

This tool is one piece of my personal AI workflow. If you want to learn how to use AI agents like Claude Code from zero (no programming background needed), I wrote a beginner series (in Traditional Chinese):

這個工具是我個人 AI 工作流的一部分。想從零開始學怎麼用 Claude Code 這類 AI agent（不需要程式背景），可以從我的入門系列開始：

1. 從零開始：安裝、看懂 GitHub、跑起你的第一個工具
2. 怎麼跟 AI agent 講話：心法、元技能與規則檔
3. 自動化流程不是設計出來的，是長出來的

Full map of my tools and posts ／ 所有工具與文章的全貌 → drpwchen.com/map