flowproof

Test an AI agent the way you test everything else: run it once, keep the
recording, and assert against it from then on.

flowproof sits at the model boundary. It captures your agent's real run -
every model request and every tool-call decision it made - into a trace,
then serves that recording back on later runs. Replay makes zero LLM
calls, so a suite that used to cost money on every commit and flake on
sampling becomes free and repeatable.

What you assert is behaviour rather than prose: which tools were called,
with which arguments, in which order, and which were not. Record one
adversarial model response and you can prove from then on, at no per-run
cost, that your agent refused to act on it.

Flows are plain YAML - short enough for an agent to write, readable enough
for a human to review in a diff. The same engine also drives web, desktop
and Citrix, so the interface a workflow ends in is covered by the same
trace.

Adding it to an existing agent? docs/adopting.md is
written to be handed to a coding agent.

Product page: automators.ai/flowproof

One spec, one record against a real model, then run forever with none. The
frames are a capture of scripts/demo/ actually running, not a
mock-up. The same three commands carry the guard path — add
assert_no_tool_call and the call an agent must never make is proven on every
commit (docs/agent-testing.md).

How it works

Agents author, a deterministic engine executes. An agent performs a
flow once from a natural-language YAML spec and records a trace — the
resolved selectors, actions, and assertions. The trace replays
deterministically in CI with zero LLM calls. When the app changes and
a step breaks, healing proposes a reviewable diff — never a silent
mutation.

flowproof is built agent-native: the primary caller is a program
(usually an AI agent), with humans in an oversight role. Every operation
is a library call returning structured results — including a structured
"here is what I'd need to know" payload when a step is too ambiguous to
author. The CLI and MCP server are thin renderings
over the same code paths.

Open-source automation has moved in steps: Selenium made browsers
scriptable, Robot Framework widened automation beyond the browser to
acceptance testing and RPA, Playwright made web automation reliable.
flowproof is built for the next step — the era in which AI agents write
and maintain the automation, and what matters is that their output is
deterministic to execute and cheap to review.

Quick start

npx flowproof --version      # no install, no Python

npm install --save-dev flowproof
# or: pip install flowproof

A spec is natural-language steps and no selectors. This one tests an agent —
a real one, built on the official OpenAI SDK, shipped in
examples/agent-demo/:

# examples/agent-demo/weather-node.flow.yaml
name: Weather assistant answers with the forecast (Node)
app: agent
agent:
  command: node examples/agent-demo/weather_agent.mjs
tools:
  - name: get_weather
    result: { city: Nairobi, sky: sunny, temp_c: 26 }
steps:
  - prompt: What is the weather in Nairobi right now? Use your tools.
  - assert_tool_call: get_weather where city contains Nairobi
  - assert: reply contains sunny

Record once. The only step that calls a real model, so the only one that
needs a key:

npm install openai
export ANTHROPIC_API_KEY=...        # or OPENAI_API_KEY
npx flowproof record examples/agent-demo/weather-node.flow.yaml

Replay for ever. No key, no model, no network to the provider:

npx flowproof run examples/agent-demo/weather-node.flow.yaml

The agent ran for real both times — same client, same tool loop. At record
flowproof captured the exchange at the model boundary; at replay it served
that recording back, so the trajectory is fixed and nothing was billed.
assert_tool_call is the part that fails when the agent regresses: wrong
tool, wrong argument, or a tool called out of order.

Want a green run before you have a key? The GIF's cassette is committed, so
pip install openai && flowproof run scripts/demo/order-status.flow.yaml
passes on a fresh clone — no key, no provider network.

Any OS. For a UI instead of an agent, point app: web at a page, or app: api
at a flow with no UI at all (examples/); the Windows Calculator
walkthrough lives in docs/getting-started.md, which
is the complete version of this section. Add --json for the structured
report on stdout.

Python API

from flowproof import Flow

flow = Flow("calc.flow.yaml")
flow.record()                    # RecordResult(trace_path=..., steps=5)

result = flow.run()              # RunResult — truthy iff the flow passed
result.steps[4].status           # "passed"
result.report_path               # result.json artifact for this run

trace = flow.get_trace()         # inspect the recorded trace programmatically

Agents can drive the same four operations — record, run, get_trace, heal —
over MCP: pip install flowproof[mcp], run flowproof-mcp.

What works today

Author — steps in plain language, resolved two ways:

• A deterministic rules grammar (docs/authoring.md —
  every documented form is enforced by a test) covers the common
  vocabulary: type, click, select, navigate, wait, assert.
• For anything freeform, a model grounds the step against the live app's
  real elements via neutral target tokens — it can never invent a
  selector — and the result still replays with zero model calls.
  Anthropic or any OpenAI-compatible endpoint (vLLM).
• When a step is too ambiguous to author ("make required field changes"),
  recording returns a structured clarification payload — the stuck step
  plus the live screen's field inventory — so the driving agent can
  resolve the ambiguity and re-record (docs/self-help.md).

Execute — deterministic replay with a provenance-tagged
selector ladder (native id → structural → text
anchor) that falls back rung by rung and flags degraded matches for
healing; auto-waiting assertions; --retries for infra flakes. Point
run at a directory to execute a whole suite: one shared browser with an
isolated context per flow, a suite.yaml manifest for shared env,
seed/cleanup hooks, and data minted by an external CLI
(env_from → ${VAR}).

Review — every run writes a bundle: result.json, JUnit XML,
an HTML report, and a trace-synced recording.gif so a human can watch
what the run actually did. flowproof heal re-authors a broken flow
against the live app and proposes a reviewable trace diff with
before/after frames — applied only with explicit --apply.

Reach — adapters behind one spec format:

• app: web — headless Chromium via DevTools protocol, cross-platform
• Windows desktop via UI Automation (calc, notepad)
• app: sap — SAP GUI Scripting over COM: native scripting ids,
  transaction-code navigation (Go to /nVA01), SAP virtual keys; an
  in-memory fake engine keeps the pipeline tested on every platform
• app: vision — pixels-only driving for Citrix/RDP: OCR perception
  (pure-Rust ocrs), spatial text anchors, real input injection
• app: api — no UI at all: flows made of HTTP and SQL assertions
• app: agent: test an AI agent at the model boundary. Record its
  trajectory once against a real model, replay it deterministically with
  zero model calls, and assert the tool calls it makes
  (docs/agent-testing.md). On Linux, an
  agent.command flow can be run under real, unprivileged egress
  containment (seccomp): declare its network with allow_egress and
  certify it with assert_no_egress

Verify beyond the UI — out-of-band truth in any flow: assert_sql
(postgres) and assert_api (status, body matching, JSON request body,
auth headers); and for an app: agent flow, assert_tool_call /
assert_no_tool_call over the agent's trajectory. Secrets travel as
${VAR} references: resolved from the environment when the step fires,
never stored in a trace; password fields are masked in captured frames.

Prove security controls hold: a control is just a property that must
hold, asserted deterministically over a recorded flow. Name one with a
stable control: id; assert an access-control denial as a composed pattern
(become a low-privilege identity from the suite's identities:, prove it is
alive, then assert the denial); catch a leaked secret in agent output with
assert_no_secret_leak: ${VAR} (agent flows in v1); and fold the
control-bearing flows into a pass/fail/capability-error coverage map
with flowproof audit
(docs/authoring.md).

Debug what a tool sends: flowproof capture is a byte-fidelity HTTP
capture endpoint: point a tool-under-test at it and every request is printed
and saved verbatim (method, path, all headers, raw body as text and hexdump,
plus any SAP /BA1/-style namespace field names), answered 200 so the send
side completes. It ships inside the flowproof binary, so it is one command to
run (docs/capture.md).

All of it ships as one wheel (PyO3/maturin): Rust engine, Python API,
CLI, MCP server. Proven in CI on every push: a Notepad flow records and
replays on GitHub's Windows runners, the web adapter drives real Chromium
on Linux, the SAP pipeline runs against the fake engine, and vision runs
real OCR over synthetic screens.

Architecture

spec (natural language, YAML)
   │  flowproof record          ← agent authors: rules first, model fallback,
   ▼                              grounded against the live app
trace (JSON-lines, versioned)  ← selectors + actions + assertions, ${VAR} refs
   │  flowproof run             ← deterministic, zero LLM calls
   ▼
run bundle                     ← result.json · junit.xml · report.html ·
                                 recording.gif · healing diffs on drift

Status

Early, in active development; interfaces may still change between minor
versions. The badges above carry the current release — the Rust crates, the
Python wheel and the npm package move together, so they are the same number.

Real and tested in CI: the record→replay spine, all six adapters (web,
Windows desktop, sap, vision, api, agent), model-grounded authoring,
healing with reviewable diffs, suites, the security-control surface
(flowproof audit), the MCP server, and — at the agent boundary — the
OpenAI-compatible proxy with assert_tool_call plus the MCP tool boundary
over stdio.

Built, but with thinner coverage: the Anthropic Messages dialect, streaming
replay, agent.url services, and the MCP boundary over streamable HTTP.
docs/agent-testing.md names each gap in a
per-capability table rather than leaving "built" to imply "tested".

Two limits to know before you start: an agent flow is one turn, not a
conversation, and egress containment is Linux-only — elsewhere a
command: agent is reported "not contained" rather than silently trusted.

Choosing between flowproof and an existing browser-automation suite?
docs/comparison.md is an honest read on when
flowproof fits and when to keep what you have.

Contributing

See CONTRIBUTING.md, and
docs/design.md for the design notes behind the engine.
Licensed under Apache-2.0.