Research Agent Harness

An autonomous research agent that takes a topic to a submission-ready paper — and doesn't trust itself.

Built with OpenProgram (Agentic Programming): Python controls the workflow, the LLM provides the judgment — 89 functions across 10 research stages, with a deterministic verification layer standing between the model and your paper.

This harness is an OpenProgram program — it runs inside OpenProgram.
Install OpenProgram first, then add this harness to it.

Why this harness

• ⚖️ Built for cross-model review — your paper can be written by one model and reviewed by a different one (e.g. Claude as author, GPT via Codex as reviewer) so it isn't self-grading. Cross-model is the default in the dedicated research-review CLI and one flag away in the autonomous run (--review-provider openai-codex). At hard/nightmare difficulty the reviewer keeps memory across rounds and debates the author under a strict concession protocol; at every difficulty it audits whether promised revisions actually happened.
• 🔍 Trust-but-verify, deterministically — fabricated citations are caught by code, not vibes: every BibTeX entry is checked against four bibliographic indexes (Crossref / OpenAlex / Semantic Scholar / arXiv). Quantified claims without citations, citation-dump abuse, and paper numbers with no experiment provenance are flagged by pure-Python lints and gates.
• 🗣️ Dialogue when you want it, autonomy when you don't — --chat starts a Socratic planning mentor that asks you one question at a time until your research plan converges, then hands the brief to the autonomous run. Same framework, no slash commands.
• 🧠 9-persona reviewer pool — empiricist, theorist, novelty hawk, methodologist, statistician, reproducibility auditor, devil's advocate, clarity critic, balanced — each a concrete, checkable lens, not a costume. The default round draws four (empiricist, theorist, novelty hawk, devil's advocate).
• 🧰 Everything is an editable function — every step is a plain Python file whose docstring is the prompt. Open it, read it, change it. No hidden chains.
• 📜 Everything leaves a trace — operation log, cumulative review log with full transcripts, PRISMA flow report computed from the literature loop's real ledger (every number auditable), commitment ledgers, integrity reports.

What's inside — 89 functions, 10 stages

research-harness --list is the always-current catalog. Every function is a plain Python file — the docstring is the prompt; edit it and the behavior changes.

The verification layer

Checkpoints sit between the model and your paper. Four are pure Python, zero tokens; the integrity gate uses one bounded LLM pass to map each extracted claim to machine-readable run_record.json provenance:

The review loop (ARIS design, upgraded)

Three protocols keep multi-round review honest (enforced in code, adapted from ARS protocol specs). The concession-threshold debate and reviewer memory activate at hard/nightmare difficulty; the commitment ledger and score trajectory run at every difficulty:

• Concession threshold — a weakness is withdrawn only when the rebuttal scores 5/5 on evidence; after any concession the bar rises. Pressure doesn't move scores.
• Commitment ledger — every revision-plan item is re-audited next round; unaddressed items are carried forward verbatim and cannot silently drop.
• Score trajectory — per-dimension deltas across rounds; a regression (e.g. soundness fell while presentation rose) is flagged and fed back to the author model.

Difficulty controls information asymmetry: medium (author curates what the reviewer sees) → hard (+ reviewer memory & debate) → nightmare (reviewer reads the repo itself; the author can hide nothing).

Quick Start

1. Install

# 1. Install the OpenProgram host (one command)
git clone https://github.com/Fzkuji/OpenProgram && cd OpenProgram
./scripts/install.sh            # Windows: .\scripts\install.ps1

# 2. Add this harness — clones it into OpenProgram's programs/agentic_functions/
#    and installs its deps. The first-run wizard also offers this.
openprogram programs install research

Restart OpenProgram and research_agent appears in the Functions page / chat. That's the whole install.

OpenProgram walks openprogram/programs/agentic_functions/ at startup and loads
any cloned repo that satisfies the harness contract:

Research-Agent-Harness/              ← cloned into programs/agentic_functions/
├── pyproject.toml                   ← declares THIS repo's own deps only
└── research_harness/                ← importable package
    ├── __init__.py                  ← kept dependency-light
    └── agentics/
        └── __init__.py              ← exposes AGENTIC_FUNCTIONS = [research_agent]

Importing research_harness.agentics fires the @agentic_function
decorators, which self-register the functions. Two rules keep this safe:
the top-level __init__ must import cleanly on a machine without the
harness's optional deps, and pyproject.toml must NOT declare
openprogram as a dependency (the host already provides it; declaring it
re-installs the host from git). Full contract:
docs/installing-harnesses.md.

git clone https://github.com/Fzkuji/OpenProgram.git && pip install -e OpenProgram
git clone https://github.com/Fzkuji/Research-Agent-Harness.git
pip install -e Research-Agent-Harness

pip install -e hard-codes absolute paths into site-packages/*.pth — if
you rename a parent folder, rerun pip install -e . from the new location.

2. Pick LLM providers

Providers are configured on the OpenProgram host, not here — run openprogram providers configure (or the first-run wizard) and follow OpenProgram's provider setup. This harness just chooses which configured provider plays each role:

For cross-model review, point the two roles at different vendors: --provider claude-code --review-provider openai-codex.

The full autonomous experience (functions that save their own artifacts) needs a file-capable executor — claude-code or openai-codex. Pure-API providers still run every Python-side orchestrator (literature loop, citation gate, lints, PRISMA, review parsing).

3. Run

# Autonomous: the agent picks stages and functions to satisfy the task
research-harness --work-dir /abs/path "Survey recent work on LLM uncertainty"

# Dialogue first: Socratic planning, then hand the brief to the autonomous run
research-harness --work-dir /abs/path --chat "LLM uncertainty"

# Cross-model: Claude writes, GPT reviews
research-harness --work-dir /abs/path --provider claude-code --review-provider openai-codex \
    "Review the paper at ./my-project/"

# Focused paper review CLI: one paper in, one structured review out
research-review paper.pdf --venue NeurIPS -o review.json
research-review paper.pdf --venue NeurIPS --mode revise --auto-fix --max-rounds 4

# Deterministic citation check, no LLM
python -m research_harness.citation_gate paper/references.bib

# Everything registered
research-harness --list

In Python:

from research_harness.main import research_agent
from openprogram.providers import create_runtime

rt = create_runtime(provider="claude-code")
rt.set_workdir("/abs/path/to/work-dir")
result = research_agent(task="Survey LLM uncertainty", runtime=rt)

Dialogue mode

$ research-harness --work-dir ~/research/unc --chat "LLM uncertainty quantification"

— Socratic planning dialogue (answer in the terminal; type 'done' to finish early) —

[follow-up] When you say "uncertainty", do you mean the model's calibration
on its own predictions, or epistemic uncertainty about facts?
> calibration on QA tasks

[follow-up] What evidence would convince a skeptical reviewer that your
calibration metric is better than expected calibration error?
> ...

The mentor asks one question at a time (clarifying → probing → structuring → challenging), never answers for you, extracts [INSIGHT] commitments in your own words, and writes RESEARCH_BRIEF.md + the full transcript when the plan converges — then offers to start the autonomous run with the brief. Registered oversight="interactive", so the unattended loop can never wander into it.

Under the hood

Two nested decision loops on OpenProgram's next-step decision mechanism (exec(choices=...)): every routing point hands the model a typed menu, and the framework parses, validates, retries once, and executes the pick. Misbehavior is handled by code — an unresolvable pick fails loudly (never silent success), a model that re-picks the same function with the same arguments is warned and then cut off, and human-in-the-loop functions are hidden from the autonomous catalogs by oversight metadata.

Project structure

Research-Agent-Harness/
├── research_harness/
│   ├── main.py                  # two-level loop + CLI (+ --chat)
│   ├── registry.py              # 89 functions, stages, oversight metadata
│   ├── pipeline.py              # fixed 8-stage pipeline + integrity gate wiring
│   ├── citation_gate/           # deterministic 4-index citation verification (vendored, CC BY-NC 4.0)
│   ├── writing_lint/            # uncited-assertion + citation-context lints
│   ├── references/              # venue scoring, writing principles, citation discipline
│   └── stages/                  # literature / idea / experiment / writing / review /
│                                #   rebuttal / presentation / theory / wiki / integrity /
│                                #   interactive / submission / external / meta
├── skills/                      # Claude Code skill shims (/peer-review, /self-review, …)
└── tests/                       # 288 tests, no network, mocked LLM

Design principles

1. Python controls the loop, the LLM makes the decisions — every routing point is a typed next-step decision with retry and loud failure; every guard (repetition, oversight, gates) is code.
2. The docstring is the prompt — no hidden prompt files; reading a function tells you exactly what the model is told.
3. Different models should disagree — cross-model review is the default in the research-review CLI and one flag (--review-provider) away in the autonomous run; the review protocols are designed for an adversary, not a collaborator.
4. Verify with code wherever code can verify — citation existence, claim provenance, assertion lints, PRISMA counts: deterministic, free, and immune to model mood.
5. Everything leaves a trace — operation log, review transcripts, commitment audits, integrity reports; no work is lost, no decision is unexplained.

Acknowledgments

This harness stands on several open projects. Thank you to their authors:

• OpenProgram — the runtime framework underneath everything here: @agentic_function, Runtime.exec(), and the next-step decision mechanism (exec(choices=...) / decision.make) that drives both levels of the autonomous loop. Agentic Programming is the paradigm it ships.
• ARIS (wanshuiyin) — the cross-model review design this harness's review loop is built on: a GPT reviewer and a Claude author kept adversarial by construction, with difficulty levels controlling information asymmetry.
• awesome-ai-research-writing (Leey21) — the battle-tested writing/polish/translation prompts that seeded the writing stage's 20+ functions.
• Academic Research Skills (Cheng-I Wu, CC BY-NC 4.0) — the deepest single influence on the harness's verification layer, fully absorbed into the harness's own design: the vendored deterministic citation-existence gate (research_harness/citation_gate/) and writing lints; the re-implemented protocols — concession-threshold debate, revision commitment ledger, score-trajectory regression detection, the experiment→writing integrity gate, the enriched reviewer persona pool, oversight-level metadata; and the Socratic mentor strategy behind --chat. ARS itself credits The AI Scientist (Lu et al.) for the autonomous-research failure-mode catalog and PaperOrchestra (Song et al.) for verification ideas — both indirectly shaped this harness too.

License

The harness's own code is MIT. Directories vendoring ARS content (research_harness/citation_gate/, parts of research_harness/writing_lint/) carry CC BY-NC 4.0 (see their LICENSE files), which makes the combined distribution non-commercial.