dsh-continual-evolve

中文 | English

Continual self-evolution for DeepSeek Harness: a versioned, auditable, rollback-safe layer of harness state — prompt notes, memories, skills, and subagent specs — refined from session trajectories.

Status: all phases complete; in long-term maintenance. Phases 1–3
shipped the full evolution loop: the pure-core engine, model tools and
the /evolve command, the automatic review gate (turn-interval +
compaction checkpoints, human approval for global edits), real
system-prompt injection (prompt notes + delegation specs, zero token
cost when empty), and the benchmark-driven validation loop (code-owned
scoring, non-regressive acceptance, rubric ACL). Since then the plugin
keeps growing with usage-driven enhancements — the memory layer (ranked
injection, trajectory citations, archive), per-installation rubric keys,
and plugin-owned file logging. See the Roadmap for the full shipped and
candidate lists.

Background

This project started as a research question: can a harness improve itself,
and what would a production-grade version look like? Three lines of evidence
shaped the answer:

• penguin-harness demonstrated the concept (benchmark → evaluate →
  optimize → accept/rollback) but with zero code-level enforcement — every
  guarantee was a prompt contract. Its report (docs/research/) became the
  hardening checklist this project implements.
• prime-agent /refine proved the engineering shape: versioned harness
  entries, atomic persistence, optimistic concurrency, inverse-op rollback.
  This package is an original implementation of that shape on the DSH plugin
  surface.
• Academic work (Self-Harness, AHE, HarnessOpt-Bench) supplied the discipline:
  frozen evaluation runtime, code-owned aggregation, non-regressive
  acceptance.

The result: the model proposes, the code guarantees. Every mechanical
safety property (schema validation, snapshots, versioning, audit trail,
acceptance decisions) is enforced in code — never by asking the model to
behave.

Why

Agents accumulate reusable experience in every session — repeated failures, durable facts, reusable procedures — and then forget it at the next turn or session. This plugin makes that experience first-class persistent state:

• Versioned entries keyed by kind (prompt / memory / skill / subagent), each with a recorded provenance and version
• Evidence trail: every refinement appends an event carrying trigger / changes / evidence / outcome
• Deterministic rollback: inverse edits are generated from applied results — no LLM re-guessing
• Code-enforced safety, not prompt discipline: schema validation, atomic writes, corrupt-file degrade, optimistic concurrency, immutable base system prompt
• Local (session) and global (cross-session) scopes with merge semantics

Design provenance

Inspired by three bodies of work (see docs/design.md):

• prime-agent /refine (MIT): the state model, atomic persistence, optimistic concurrency, per-edit validation, and inverse-op rollback this package implements — annotated reference source in docs/research/prime-agent-refinement.ts. The code here is an original implementation, written for the DSH plugin surface.
• penguin-harness (Apache-2.0): the benchmark-driven evolution loop — research report in docs/research/penguin-harness-self-evolution.md; its prompt-only contracts are the anti-pattern this package hardens.
• Academic: Self-Harness (arXiv 2606.09498), AHE (arXiv 2604.25850), HarnessOpt-Bench (arXiv 2608.06301).

Tech stack

Project layout

dsh-continual-evolve/
├── package.json          # exports / files / engines / scripts + dsh.bundle manifest
├── cordis.patch.yml      # bundle patch (dsh plugin add activates on install)
├── tsconfig.json / .oxlintrc.json / .editorconfig / .gitignore
├── LICENSE / README.md / README.zh.md
├── docs/
│   ├── design.md               # full design doc (incl. hardening matrix)
│   └── research/               # penguin-harness report + prime-agent reference source
├── src/
│   ├── index.ts          # cordis plugin entry (service mount + wiring)
│   ├── types.ts          # HarnessState / entry / edit / result types
│   ├── state.ts          # atomic persistence, corrupt degrade, merge, concurrency
│   ├── validate.ts       # code-enforced edit validation
│   ├── apply.ts          # per-edit apply pass with optimistic locking
│   ├── rollback.ts       # deterministic inverse-op rollback
│   ├── plan.ts           # proposal JSON parsing (truncation-aware)
│   ├── tool.ts           # evolve_* model-facing tools (5)
│   ├── command.ts        # /evolve command (incl. benchmark subcommands)
│   ├── planner.ts        # ctx.llm planner
│   ├── render.ts         # bounded prompt rendering
│   ├── inject.ts         # dynamic system-prompt section (prompt notes + delegation specs, ranked injection)
│   ├── source.ts         # trajectory citations (sessionId + event seqs of distilled entries)
│   ├── auto.ts           # auto-review gate (turn/compaction triggers + audit, global-aware view)
│   ├── notify.ts         # gate visibility — follow-up notice after an approved auto-refine
│   ├── goal.ts           # goal-driven evolution rounds (/evolve goal)
│   ├── review.ts         # gate LLM judgment (declines local duplicates of globally covered topics)
│   ├── approval.ts       # human approval for global edits
│   ├── skill.ts          # skill materialization ($DSH_HOME/skills/)
│   ├── mount.ts          # hot-mounted skill plugins (loader.create + boot restore)
│   ├── benchmark.ts      # benchmark store
│   ├── rubric.ts         # rubric ACL (AES-256-GCM envelopes, auto-generated local key)
│   ├── logfile.ts        # plugin-owned file logging (JSONL exporter + rotation)
│   ├── score.ts          # code-owned aggregation + acceptance rule
│   ├── evaluate.ts       # evaluation matrix runner (structured-output subagents)
│   ├── pool.ts           # bounded-concurrency worker pool for evaluation runs
│   ├── store.ts          # store layout + snapshots + result history
│   └── service.ts        # evolution engine (onApplied hook)
└── test/                 # 20 files, 184 tests

In-session usage (after restart)

/evolve                  help + current local store
/evolve list [global]    list entries
/evolve history          applied refinements (ids for rollback)
/evolve rollback <id>    deterministically revert a refinement
/evolve plan [msg]       LLM planner against the current store
/evolve archive <id>     hide an entry from injection (data kept, restorable)
/evolve unarchive <id>   restore an archived entry
/evolve log [tail N] [session <id>]  show the recent plugin log (default 50 lines; optional per-session filter)
/evolve export <path>    backup the local store to JSON
/evolve import <path>    restore a store from an export file
/evolve mount <skillId>  hot-mount a skill entry as a live cordis plugin (tool: skill_<name>)
/evolve mount list       list hot-mounted plugins (restored on boot)
/evolve unmount <id>     remove a hot-mounted plugin
/evolve goal             show the evolution goal (round-driven auto-review)
/evolve goal <objective> create/update the evolution goal — while active, the review gate runs EVERY round
/evolve goal done        complete the evolution goal

Model-facing tools: evolve_list, evolve_add, evolve_update, evolve_delete, evolve_rollback.

Memory layer

Beyond the persisted store itself, three features keep injected memory
"understanding you" as entries grow (gap analysis vs. Mem0 / Letta / Zep /
LangMem; no external services — everything is pure functions):

• Ranked injection — when a kind holds more than the 6-entry cap, the
  injected block no longer shows the fixed first six: entries are scored by
  relevance to the agent's most recent direct user messages (keyword/BM25
  level: title hits weigh 2×) and then by recency (updated_at, 30-day
  half-life), so the freshest and most relevant entries fill the cap. The
  empty-store zero-token behavior is unchanged.
• Trajectory citations — every newly created entry records
  metadata.sourceSession + metadata.sourceSeqs pointing at the direct
  user messages it was distilled from (DSH sessions are event-sourced with
  contiguous seqs, so the citation expands back into the durable session
  log). Listings show src=<sessionId>:<seqs>; old entries are not migrated
  and never error.
• Archive — /evolve archive <id> hides an entry from injection
  (metadata.archivedAt, data kept, rollback-compatible) and
  /evolve unarchive <id> restores it. Archived entries are marked
  [archived] in evolve_list and skipped by injection; the overflow count
  excludes them.
• Global-aware gate — the auto-review gate and planner judge the merged
  global + local state with every entry's real scope labeled, so a topic
  already covered by a global entry is declined instead of being re-sedimented
  as a local duplicate.

Logging

Plugin-owned file logging: every cordis log message (from this plugin or any
other) is appended to <dshHome>/evolve/plugin.log as JSONL (0600, rotated to
plugin.log.1 past logMaxBytes). It works no matter how dsh web is
launched — no extra component to install, no startup-script dependency.
View the tail with /evolve log [tail N], or read the file directly:

tail -f ~/.dsh/evolve/plugin.log          # live
/evolve log 100                            # last 100 lines in the chat

For live output in a foreground terminal, the official
@deepseek-ai/cordis-plugin-logger-console plugin can be added to the
profile (optional; the file log remains the baseline that always exists).

Benchmark-driven validation (Phase 3)

/evolve benchmark new <title> [runs]                   create a benchmark (runs = repeats per case, default 1)
/evolve benchmark add-case <bid> <title> <statement> <rubric>
/evolve benchmark list                                 list benchmarks
/evolve benchmark reset <bid>                          clear the scoreboard (re-run reference)
/evolve benchmark status <bid>                         scoreboard + decisions
/evolve benchmark run <bid>                            evaluate current state → reference
/evolve benchmark run <bid> candidate <refinementId>   evaluate post-refinement state → decide

The loop: freeze a reference score → evolve a candidate (/evolve plan) →
run the same case × run matrix against the post-refinement state → the
code-owned acceptance rule keeps the candidate only if the overall mean
strictly improves with no case regressing (Self-Harness style). The model
produces raw per-cell scores only; aggregation and decisions live in
src/score.ts. Rubric isolation is by construction (the planner never sees
rubric files); a rejection is recorded in the scoreboard and the refinement
is rolled back automatically (autoRollbackOnReject, on by default).

Real recorded run (ACCEPT)

A live dsh web session, one case, one candidate — the first genuine
acceptance:

The evaluator does not grade model common sense — it inspects the actual
harness state under test (grep, evolve_list) and scores against it, so a
harness change measurably moves the score. Earlier runs in the same session
produced honest REJECTED decisions (0 → 0 placeholder cases, and 100 → 100
where the baseline was already perfect).

Configuration

Example (profile cordis.patch.yml):

- insert:
    - id: continual-evolve
      name: 'dsh-continual-evolve'
      config:
        autoReview: true
        reviewIntervalTurns: 6

Development

pnpm install        # install dev deps
pnpm build          # tsc -> lib/
pnpm test           # vitest run
pnpm lint           # oxlint src test

Hit a wall? See docs/FAQ.md — real failure/fix records (service planes, schema DSL, structured output, gate counting, verifying prompt injection).

Roadmap

Shipped

• Phases 1–3 (done): pure-core engine (state model, validation, apply, rollback, proposal parsing) → evolve_* tools + /evolve command + ctx.llm planner → auto-refine review gate (turn-interval + compaction checkpoints, visible follow-up notices), global-scope human approval, executable skills, real system-prompt injection (prompt notes + delegation specs, inherited by subagents), benchmark-driven validation loop (code-owned scoreboard, non-regressive acceptance, rubric isolation by construction), hot-mounted skill plugins, goal-driven evolution rounds.
• 2026-08 maintenance wave (done):
  • memory layer — ranked injection (relevance + recency scoring fills the per-kind cap), trajectory citations (metadata.sourceSession + sourceSeqs, shown as src=session:seqs), archive/unarchive (/evolve archive <id>, injection skips archived entries), global-aware gate (declines local duplicates of globally covered topics)
  • per-installation rubric key — auto-generated local key file (<dshHome>/evolve/rubric.key, 0600); no more publicly known dev key
  • plugin-owned file logging — every cordis log message lands in <dshHome>/evolve/plugin.log (JSONL, 0600, rotated), viewable via /evolve log; works with any launch method, no extra component to install
  • trajectory-grounded planning — /evolve plan (and every planner call, including the gate's refine step) now reads the session trajectory: the caller's recent direct user messages are extracted from the session log and fed to the planner as a <session_trajectory> block, so proposals are grounded in what the user actually said (explicit trajectory overrides; empty trajectory is omitted at zero cost)
  • gate-proposed archiving — stale entries are a first-class refine target: the planner can emit action: "archive" (kind + id only), which stamps metadata.archivedAt through the normal apply path — snapshot, version bump, audit event, and a deterministic rollback inverse that restores the pre-archive state. Archive hides from injection but never deletes; re-archiving an archived entry is rejected, and the base system prompt stays immutable
  • automatic rollback on benchmark rejection — the acceptance loop is closed: when the code-owned decision rejects a candidate, the refinement is reverted automatically through the same engine path as /evolve rollback (deterministic inverse edits, snapshotted and audited; configurable via autoRollbackOnReject, on by default). Failures report the manual fallback instead of throwing
  • per-session log filtering — /evolve log [tail N] [session <id>] keeps only the lines mentioning a given session id (exact token match, drawn from the rendered message and raw args); gate records now carry the session id in their log line

The planned/candidates list is empty for now — future work is driven by real usage.

License

MIT. Independent project — not affiliated with DeepSeek.