drover

drover turns herdr into a self-running team of coding agents: a lead assigns beans
(tasks), coders work in their own git worktrees, a reviewer on a different harness reviews, and a watcher
loop moves work between them without the lead in the path.

It is a set of bash scripts, a launchd job, and an agent skill. It was extracted from one person's fleet —
10 to 25 agents across Claude Code, Codex, opencode, Grok, Cursor and Antigravity, working on one production
codebase — and it carries that fleet's scar tissue: almost every guard in it has a dated comment saying what
broke before it existed.

Why

herdr already hosts the agents: each coding CLI runs in a named pane, and herdr agent list|read|prompt
drives them. What it does not do is notice when one of them has finished. Before drover, the lead did the
noticing, and the ledger says what that cost:

• 44 minutes median between one dispatch and the next to the same agent (p90: 285 minutes), measured over
  650 gaps, for tasks that take 10–20 minutes. Every transition — done → review → next task — waited for the
  lead, who polled in 8-minute sleeps.
• fleet status reported done for agents that were mid-turn, and pane text kept stale error banners after
  they were fixed. When opencode's credits ran out, 10 agents reported done with no work, no report and no
  error, and were dispatched into again.
• Reports announced "DONE, landable" for the ninth time — each one a paid round that re-verified
  commits that already existed.
• opencode spent $290, then $376 on two consecutive days, against $27–$82 on each of the four days before.
  The first meter built to watch it (opencode stats --days 1) reported $952: it adds up the lifetime
  cost of every session touched in the window. The cap now sums per-message cost since midnight.

drover's answers: agents announce completion (fleet-done) instead of being polled; state is judged by
artifacts (a commit on the branch, a report on disk), never by status fields; a watcher reacts within a
minute; and each of those failures now has a guard (a spend cap, a no-op detector, credit-dead seat detection).

The loop

   beans list --ready (on main)
            |  top_up_queue
            v
   +---->  queue/  <epoch>-<bean>  <------------------------------+
   |        |  refill: a free coder, harness not broke,           |  requeue: dead seat,
   |        |  bean on main and ready, not owned by another coder |  failed dispatch
   |        v                                                     |
   |   CODER seat  --  own worktree, branch fleet/<coder>-<bean> --+
   |        |  fleet-done <me> <bean> DONE | BLOCKED
   |        v
   |    inbox/  -->  drain_inbox
   |        |   DONE with unchecked items -> PARTIAL -> back to the same coder (3x -> lead)
   |        |   BLOCKED                   -> state/blocked (lead reads it)
   |        |   DONE                      -> one review, on a DIFFERENT harness
   |        v
   |   REVIEWER seat  --  cherry-picks the named commits onto a detached main, builds, tests
   |        |  fleet-done <me> <bean> CHANGES | DONE (verdict APPROVE)
   |        v
   |    drain_inbox
   |        |   CHANGES -> followup to the coder, ahead of anything in the queue --+
   |        |   APPROVE -> state/approved/<bean>: held out of the queue           |
   |        v                                                                     |
   |   LAND  the lead — or, with DROVER_AUTOLAND=1, fleet-land-bean:              |
   |        cherry-pick the bean's own branch into an ISOLATED worktree,          |
   |        gate (typecheck + related tests), fast-forward LOCAL main. Then       |
   |        fleet-verify-main (full build+suite on a clean tree) -> push origin.  |
   +--------------------------------------------------------------- coder <------+

By default landing stays with the human lead — the only party that re-verifies against current main, runs
migrations, and pushes. Set DROVER_AUTOLAND=1 and the watcher does it: fleet-land-bean lands each
approved bean to LOCAL main behind a fast per-bean gate, and fleet-verify-main runs the full suite on a
clean worktree before fleet-push sends the green batch to origin. A cumulative red pauses landing
(state/land-paused) so one bad merge can't cascade. Migrations are still never applied automatically.

Operating model

The loop above assumes the watcher is the one thing dispatching, reviewing, landing and pushing. Running
the lead's own direct assignments alongside the watcher's automated lanes is the one combination this
tool actively prevents you from reaching by accident, because it was tried once and every stall that day
had to be found by a human reading logs: a bean released by one side got re-claimed by the other, a bean
mid-review was interrupted then double-assigned, a third was displaced out from under its coder (see
docs/LESSONS.md, "Two managers collide"). Exactly one thing dispatches, reviews, lands and pushes at a
time.

• Observer mode (touch state/observer) turns the watcher into a pure observer: every tick becomes
  fleet-observe instead of the mutating lanes above — it reads state, logs, and alerts, and never presses
  a key in a pane, reassigns a bean, lands, or pushes (rm state/observer restores the normal lanes). Use
  it when a human or the agents themselves are going to own dispatch/landing directly for a while.
• The lead queue is what a human or lead uses to hand out work while the watcher stays observer-only:
  write a TAB-delimited <bean>\t<harnesses-or-*>\t<prompt> line to state/lead-queue and list eligible
  seats in state/lead-pool (a :named suffix restricts a seat to lines that name it explicitly). Every
  fleet-observe tick hands each free, in-pool seat its next matching line — see docs/RUNBOOK.md.
• Self-land (docs/agent-land-loop.md) is what agents follow once assigned: own the bean end to end —
  branch, code, update the bean file, rebase, get the FINAL diff reviewed by a different model (capped at 3
  rounds, then a self-run SHIP-bar review takes over — see docs/LESSONS.md), push through the one
  fleet-wide push lock (examples/pre-push-lock.sh), and prove the push landed with
  git merge-base --is-ancestor <sha> origin/main before reporting DONE.
• Self-serve (docs/self-serve-playbook.md): seats on a harness listed in DROVER_SELF_SERVE pick their
  own ready bean (claimed atomically through state/owner/<bean>) or run one hunt when no queued line fits
  them, instead of idling beside a queue routed to other harnesses.

fleet-observe --board gives a one-screen read of the fleet in observer mode, and observer ticks still
stamp state/last so the tick-stall alert doesn't fire on a watcher that's running exactly as configured.

What one watcher tick does

fleet-watch run ticks every 60 seconds (FLEET_TICK). In order:

1. Clear known blocking dialogs — only exact matches (an auto-mode prompt, a save confirmation, a
   rate-limit model switch). A generic "Enter to confirm" match once hit a folder-trust dialog whose cursor sat
   on "No, exit" and killed three seats.
2. Budget — fleet-budget reads each non-working pane for out-of-credits / usage-limit / session-limit
   banners (ignoring stale scrollback); the spend guard sums today's per-message opencode cost from its database and compares it with your daily cap; any
   harness listed in state/harness-off is off. A broke harness gets no new dispatches or reviews.
3. Harvest beans that agents filed in their worktrees onto main (the board follows main).
4. Drain the inbox — route each announcement: DONE → a cross-harness review; PARTIAL → back to the coder;
   CHANGES → a followup; APPROVE → held for landing; BLOCKED → the lead. Stale announcements (a bean the seat is
   no longer on) are logged and ignored; a late DONE still routes its review.
5. Retry reviews that found no free reviewer last tick.
6. Reap task files whose claim is provably dead (bean completed on main, re-routed, reported and idle, or
   silent past 45 min for a reviewer / 90 min for a coder). A working or blocked seat is never reaped.
7. Seat scan (optional, TypeSafe) — every 10 minutes, read seats that have held a task for 20+ minutes.
8. Owner sync — record which coder owns each bean; flag any bean held by two coders.
9. Sweep (every 6 h), WIP snapshot (every 20 min), inventory alert (every 30 min: undeployed
   commits and integrity drift), yolo guard (every 5 min). heal_repo at the top of every tick
   restores $DROVER_REPO to main if an agent op drifted it off (otherwise the whole fleet idles).
10. Unblock + ghost reap — every 5 min fleet-unblock triages the held pile (un-hold finished beans,
    keep date-gated ones, digest lead/human decisions, re-dispatch the rest); every 10 min fleet-ghost-reap
    returns in-progress beans no fleet state names to todo.
11. Top up the queue from beans list --ready when it is empty (sprint keywords first, then priority) —
    unless the WIP cap is reached (see Guardrails): backpressure stops starting NEW beans while the
    integration backlog is full.
12. Refill every free coder: a followup beats the queue; then the next queued bean, after re-checking it
    is on main, still ready, not owned by another coder, not awaiting review or landing, within the epic
    train cap, and that the seat is not credit-dead. A free seat that finds the queue empty calls
    fleet-spec-feed, which files one bean from a stuck epic or an orphan OpenSpec change. fleet-refill (a daemon,
    launchd/dev.drover.fleet-refill.plist.in) hands queued items to idle seats, fleet-feed restocks that queue from
    DROVER_FOCUS_BEANS, fleet-batch groups fix items, fleet-prep gives each assignment a clean branch, and
    fleet-cred-check is the standup health check. Optional: fleet-batch-land lands every READY branch with one gate and one push. fleet-preflight checks the publication mode, credentials and production lag; the operating model (work unit, WIP limits, definition of done, rhythm) is in docs/OPERATING-MODEL.md.
13. Push + land (only with DROVER_AUTOLAND=1) — push first: the cumulative gate + push via
    fleet-verify-main (a red pauses landing, with the reason in state/land-paused); then one approved
    bean per tick to LOCAL main via fleet-land-bean, which yields while a push is in flight.

state/last holds a one-screen summary of the latest tick; watch/watch.log has every decision with a
one-word tag (ROUTED, DISPATCH, FOLLOWUP, PARTIAL, SKIP, STALE, BUDGET, APPROVED, NO-OP, SEAT, …).

Guardrails

Optional: TypeSafe detectors

fleet-judge asks TypeSafe's System One API three typed questions. It is off unless
~/.config/typesafe/api_key exists. Without the key, or on any error, it prints nothing and exits 2, and
fleet-watch behaves exactly as it does without it.

eval/ has the seat-state fixtures (real pane tails, rewritten to remove paths and task text) and a script that
scores the shipped detector against them. The numbers above were measured on the unscrubbed originals; the
scrubbed fixtures have not been re-scored. The noop and verdict numbers were measured on the original fleet's
reports, which are not included.

What leaves your machine: report text (first 1,500 chars), review text (1,800) and pane tails (last 2,500), with
credential-shaped strings (sk-…, ghp_…, AKIA…, *_KEY=…, …) stripped first. Paths, branch names and task
titles are sent. Decide whether that is acceptable before adding the key.

Requirements

• macOS (launchd, stat -f, APFS cp -c clones for fast worktree bootstrap)
• bash, python3, jq, git 2.40+
• herdr with the harness CLIs you use (claude, codex, opencode, cursor-agent, grok, agy)
• beans — the task board. Beans must be committed on main to be dispatched.
• Optional: opencode (for the spend cap), a TypeSafe API key, sqlite3 (for ops/oc-compact.sh)

Quickstart

git clone <this repo> ~/src/drover && cd ~/src/drover
./install.sh --dry-run          # shows every symlink, dir, file and plist it would create
./install.sh                    # symlinks bin/* into ~/.local/bin; never overwrites a file it did not create
$EDITOR ~/.config/drover/config # set DROVER_REPO, DROVER_ROSTER, DROVER_BUILD, DROVER_TEST

# inside herdr:
fleet up                        # spawn the roster: one worktree + pane per seat, approval bypass on
fleet rules                     # install the fleet-peers skill for claude/codex/grok/opencode
fleet status

fleet assign <bean>             # hand-dispatch one bean to the least-loaded idle coder, or:
fleet-watch once                # one tick by hand; tops the queue up from beans --ready
./install.sh --load             # let launchd run fleet-watch, fleet stale, and the hourly standup
tail -f ~/.local/share/drover/watch/watch.log

To land: follow docs/RUNBOOK.md. To stop new work while everything in flight finishes:
touch ~/.local/share/drover/watch/state/finish-first.

If another fleet command is on your PATH (JetBrains Fleet installs one), it may shadow this one. drover's own
scripts call each other by absolute path, so only your interactive shell is affected.

Commands

Configuration

Everything is read from ~/.config/drover/config (plain shell, sourced; DROVER_CONFIG to move it). Write
values as ${VAR:-value} so an environment variable still wins. See examples/config.

Spend caps live in ~/.config/drover/caps (opencode_daily_usd=, active_from=), re-read every tick.

Lessons

docs/LESSONS.md is the part of this repository most worth reading even if you never run
it: over fifty dated, measured lessons from running the fleet — why agents are never judged by their status
field, why every bean gets its own branch, why a printf | grep -q under pipefail idled 19 agents for four
hours, why a test that needs no input from the system is documentation, why two managers dispatching at once
collide and observer mode is the fix, why a push lock must never evict a live owner by age. The agent-facing
subset is skills/fleet-peers/SKILL.md.

Limitations

• macOS only. launchd, BSD stat/date/df, and APFS clones are assumed throughout.
• Built for one person's fleet. Roles are inferred from seat names, harness quirks (dialog texts, credit
  banners, resume flags) are the ones that fleet met, and the prompts assume the target repo has its own agent
  rules (CLAUDE.md / AGENTS.md). Expect to read the scripts.
• Landing is manual by default. drover routes work to "approved"; a human lead lands it. DROVER_AUTOLAND=1
  automates landing + push behind the gates above, but it never applies migrations and it pauses on a red suite.
• Seats run with approval prompts bypassed. That is the only way agents run unattended; it is also why
  worktrees, snapshots, and the never-push / never-migrate rules exist. Run it on a machine and a repository
  where that is acceptable.
• Not included: a shared MCP-server hub the original fleet ran to cut per-agent memory (a separate Node
  project). fleet stale still recognises a process named mcp-hub/hub.mjs and leaves its children alone.

License

MIT — see LICENSE.