Dynobox

Cross-harness testing for multi-step agent and skill workflows.

Status: Early access. Dynobox is ready for local skill and agent workflow testing while the CLI, SDK, and report formats continue to evolve before 1.0.

Dynobox runs agent scenarios through local harnesses such as Claude Code, Codex,
and OpenCode, captures observable behavior, and evaluates assertions against what
actually happened. It is designed for testing skills and agent flows where you
care about tool usage, file effects, transcripts, final answers, and behavior
across harnesses.

Why Use It

Use Dynobox when you want to answer questions like:

• Does this skill call the expected tools?
• Does it avoid dangerous or unrelated commands?
• Does it create or preserve the right files?
• Does its final answer include required information?
• Does the same task work under Claude Code, Codex, and OpenCode?

Quick Start

Requires Node.js 22 or later.

Install the CLI and a starter dyno file, then run it:

npm install -g dynobox
dynobox init        # writes dynobox/example.dyno.mjs
dynobox discover    # prints the *.dyno.* files that run would load
dynobox run         # discovers and runs every *.dyno.* file under the cwd

dynobox run with no argument discovers *.dyno.{mjs,js,ts,mts,yaml,yml} files
recursively under the current directory. Pass a directory or a single file to scope it:

dynobox run examples/local-observability
dynobox run my-skill.dyno.yaml

Pick a harness at runtime when needed (each authored file declares its own
default list):

dynobox run --harness claude-code
dynobox run --harness codex
dynobox run --harness opencode
dynobox run --harness claude-code,codex,opencode

Run each selected scenario/harness pair more than once to measure pass rates:

dynobox run --harness claude-code,codex,opencode --iterations 5

The selected harness executable must already be installed, authenticated, and
available on PATH.

Example: A Dyno File

// my-skill.dyno.mjs
import {artifact, command, defineDyno, finalMessage, tool} from '@dynobox/sdk';

export default defineDyno({
  name: 'package-script-skill',
  harnesses: [{id: 'claude-code', permissionMode: 'default'}],
  scenarios: [
    {
      name: 'detects test script',
      setup: [
        `cat > package.json <<'JSON'
{
  "name": "fixture",
  "scripts": {"test": "vitest run"}
}
JSON`,
      ],
      prompt:
        'Use `cat package.json` and tell me whether this project has a test script.',
      assertions: [
        command.called('cat', {args: ['package.json']}),
        tool.notCalled('edit_file'),
        artifact.contains('package.json', 'vitest run'),
        finalMessage.contains('test'),
      ],
    },
  ],
});

The same dyno authored in YAML:

# my-skill.dyno.yaml
name: package-script-skill
harnesses:
  - id: claude-code
    permissionMode: default
scenarios:
  - name: detects test script
    prompt: >-
      Use cat package.json and tell me whether this project has a test script.
    setup:
      - |
        cat > package.json <<'JSON'
        {"scripts":{"test":"vitest run"}}
        JSON
    assertions:
      - label: reads package.json
        type: command.called
        executable: cat
        command: {args: [package.json]}
      - type: tool.notCalled
        tool: edit_file
      - type: artifact.contains
        path: package.json
        text: vitest run
      - type: finalMessage.contains
        text: test

See Getting Started for the full walkthrough and
Config Authoring for the YAML / TS assertion
reference.

Documentation

• Docs index
• Getting Started
• Config Authoring
• CLI Reference

Current Capabilities

• Discover and run *.dyno.{mjs,js,ts,mts,yaml,yml} files with
  dynobox run [path] — no arg = cwd, directory = recursive, file = single
  run. Legacy explicit-file paths (e.g. dynobox.config.ts) keep working.
• Scaffold a starter file with dynobox init (--yaml for YAML, --harness
  to pin the starter harness).
• Authenticate with dynobox login, verify with dynobox whoami, and remove
  saved tokens with dynobox logout.
• Author dynos in TypeScript / JavaScript with @dynobox/sdk helpers
  (defineDyno, defineScenario, tool, skill, artifact, transcript,
  finalMessage, sequence, http, dyno) or in YAML using the same shape
  with type-discriminated assertion objects.
• Automatically copy adjacent fixtures/ directories for JS/TS dynos authored
  with defineDyno(...), and automatically copy SKILL.md for dynos authored
  under .agents/skills/<name>/ or .claude/skills/<name>/.
• Run locally against Claude Code, Codex, OpenCode, or any combination.
• Select harnesses at runtime with --harness claude-code, --harness codex,
  --harness opencode, or comma-separated values while preserving configured
  model metadata.
• Override selected harness models positionally with --model, such as
  --harness claude-code,opencode --model sonnet,openai/gpt-5.5.
• Filter scenarios at runtime with --scenario <pattern>.
• Repeat each scenario/harness pair with --iterations <count> and view
  inline pass-rate sparklines such as .F....
• Configure harness permission behavior with permissionMode or
  --permission-mode; harness-specific access elevation is opt-in.
• Emit newline-delimited JSON reports with --reporter json.
• Upload compact dashboard summaries with --save-run when authenticated.
• Assert tool calls with tool.called(...) and tool.notCalled(...).
• Match file-oriented tool calls by path, such as
  tool.called('read_file', {path: 'package.json'}).
• Assert skill instruction file references with skill.referenced(...).
• Assert normalized shell commands with command.called(...) and
  command.notCalled(...); raw shell string matchers remain available as
  escape hatches.
• Assert HTTP requests to declared endpoints with http.called(...) and
  http.notCalled(...).
• Express valid alternative behavior paths with anyOf(...), including nested
  verify.command(...) branches.
• Assert ordered tool-call sequences with sequence.inOrder(...).
• Assert work-directory artifacts with artifact.exists(...),
  artifact.notExists(...), artifact.contains(...), and
  artifact.unchanged(...).
• Assert harness transcript and final response text with
  transcript.contains(...) and finalMessage.contains(...).
• Run post-harness checks with verify.command(...) for generated artifacts.
• Stream live grouped progress and tool events in interactive terminals.
• Use default grouped output, --quiet dots-and-failures output, --verbose
  phase/assertion details, and --debug work-directory, artifact, transcript,
  raw chat JSONL, and normalized tool-event paths.

HTTP endpoint declarations and HTTP assertions can evaluate local child-process
traffic that honors proxy and CA environment variables. Harness-native web
tools and binaries with their own trust stores may bypass local HTTPS capture.

Packages

This repository is a pnpm monorepo. Published packages live under packages/.

@dynobox/run-schema is published to GitHub Packages under the @dynobox scope. @dynobox/runner-local and @dynobox/evaluators are private workspace packages bundled into the published dynobox CLI instead of exposed as public dependencies.

Project Site

dynobox.xyz

Contributing

See CONTRIBUTING.md for local development commands and
checkout workflows.

License

Apache-2.0 for all code in this repository. See LICENSE.