Multi-Agent Workflow Lab

An open-source testing and observability framework for multi-agent delegation, tool execution, MCP workflows, permissions, sandboxed actions, prompts, and runtime behavior.

Status: Experimental — v0.1.0 release candidate. Suitable for local development, evaluation, and policy testing; not production-hardened infrastructure.

Why this exists

Most model evaluation stops at input → model → output. Multi-agent systems add behavior that a final answer cannot explain:

• Which agent delegated a task, to whom, and why?
• What context, permissions, tools, MCP servers, and budget did the child receive?
• Did an agent attempt privilege escalation, repeat work, or enter a loop?
• Was the delegation efficient, and was the child result actually integrated?
• Can the run be inspected, compared, or replayed without repeating side effects?

MAWL makes those decisions explicit, policy-controlled, and traceable. Prompts are versioned assets; delegation is a first-class runtime event; deterministic rules evaluate behavior independently from optional model judges.

Key capabilities

Architecture

flowchart TD
  Input["User or test input"] --> Parser["Input parsers"]
  Parser --> Runtime["Agent runtime + task graph"]
  Runtime --> Delegation["Delegation engine"]
  Delegation --> Guards["Identity + permission + policy"]
  Guards --> Agents["Child agents"]
  Guards --> Tools["Tool executor"]
  Tools --> MCP["MCP connectors"]
  Tools --> Sandbox["Sandbox providers"]
  Agents --> Events["Runtime events"]
  MCP --> Events
  Sandbox --> Events
  Events --> Observe["Trace + metrics + monitors"]
  Events --> Evaluate["Rules + optional judge + replay"]

Quick Start

Requirements: Node.js 22 or newer and pnpm 11.

From a clone or an extracted release archive:

cd multi-agent-workflow-lab
pnpm install --frozen-lockfile
pnpm demo

The demo requires no API key or paid service. A public clone URL can be added after the maintainer chooses and publishes the repository location.

What the demo shows

pnpm demo runs a deterministic review chain through the CLI:

status: completed

orchestrator
  → researcher
    → analyst
      → reviewer
        → evaluator

delegation score: 88.75/100
workflow completed

The actual CLI output is JSON and includes the run ID, every task/agent/status, all eight evaluation dimensions, derived metrics, and the JSONL trace path.

Delegation observability

A trace records behavior rather than only the final answer:

orchestrator
  delegation.requested → researcher
  delegation.created   context + permissions + budget + depth

researcher
  tool.requested       → mcp.mock.search
  policy.allowed       explicit decision evidence
  tool.completed       size + redaction metadata

orchestrator
  delegation.result.accepted → child task integrated

Use these commands after a run:

pnpm mawl trace show <run-id>
pnpm mawl evaluate <run-id>
pnpm mawl graph <run-id>
pnpm mawl replay <run-id> dry-run

Evaluation scores are normalized diagnostics, not mathematical truth or a security guarantee.

Example workflows

Run any positive example with:

pnpm mawl run workflows/02-parallel-research.yaml

The negative examples intentionally return a non-zero exit code when the requested action is blocked.

Good vs. bad orchestrator

The lab compares a bounded, least-privilege orchestration trace with a synthetic bad trace containing duplicate work, leaked context, escalated permission, and an unauthorized tool request. RunComparator and DelegationEvaluator report changes in:

• agent selection and delegation edges;
• task decomposition and depth;
• context and permission minimization;
• tool use, retries, tokens, runtime, and failures;
• result integration and overall diagnostic score.

The executable coverage is in tests/lab.test.ts, with scenario inputs under scenarios/.

CLI

In this workspace, prefix CLI arguments with pnpm mawl:

pnpm mawl run <workflow.yaml> [json-input]
pnpm mawl agents list
pnpm mawl workflow inspect <run-id>
pnpm mawl task inspect <task-id>
pnpm mawl inspect agent permissions <agent-id>
pnpm mawl inspect mcp server [server-id]
pnpm mawl trace show <run-id>
pnpm mawl evaluate <run-id>
pnpm mawl graph <run-id> [output.mmd]
pnpm mawl replay <run-id> [exact|model-rerun|tool-rerun|dry-run]
pnpm mawl compare <run-a> <run-b>
pnpm mawl test <workflow-test.yaml>
pnpm mawl prompts list
pnpm mawl prompts inspect <prompt-id> [version]
pnpm mawl permissions inspect <agent-id>
pnpm mawl doctor

tool-rerun blocks tool names or manifests classified as external side effects unless the embedding application supplies explicit permission for that exact tool. The CLI supplies no side-effect permissions.

Prompts are first-class assets

This is not a directory of free-form prompt snippets. YAML assets under prompts/ include:

• stable ID and semantic version;
• owner, type, purpose, and known risks;
• declared input schema and expected output;
• allowed actions and recommended capabilities;
• content hash verified by PromptRegistry;
• provenance and trust classification when assembled at runtime.

Runtime and security policy layers remain separate from untrusted user, tool, MCP, and child-agent content. Prompt quality tests detect duplicate IDs, invalid versions, missing metadata, variable/placeholder errors, unexpected hashes, and undeclared references.

Security model

• Permissions and contextual policies deny unmatched requests by default.
• Agent execution identities are issued by the runtime and linked to workflow/task/session IDs.
• Child context, authority, and budget are reduced from the parent.
• MCP responses and model/tool/child output are untrusted data, not authority.
• Secret references and output redaction limit accidental exposure.
• Tool requests pass manifest resolution, validation, permission, policy, optional approval, output checks, and audit.
• Delegation depth, ancestry, task count, retries, calls, tokens, output, and runtime are bounded.
• Replay refuses external side effects without an explicit per-tool permission.

The restricted local sandbox is intended for development and policy testing. It is not equivalent to hardened container, VM, microVM, or kernel isolation, and network denial is best-effort. MAWL does not claim to be prompt-injection-proof, sandbox-escape-proof, or fully zero-trust. Read the threat model and security policy.

Using a real model provider

The default path always uses MockModelProvider. An optional OpenAI-compatible chat-completions adapter is available in @mawl/providers:

cp .env.example .env

Populate MODEL_PROVIDER, MODEL_ENDPOINT, and MODEL_API_KEY from a secret manager, then inject OpenAICompatibleProvider into the runtime. The CLI intentionally does not auto-load credentials. See provider integration.

Repository structure

multi-agent-workflow-lab/
├── apps/                    CLI and runnable examples
├── packages/
│   ├── core/                Schemas and provider contracts
│   ├── runtime/             Agent loop, tasks, delegation, scheduler, replay
│   ├── evaluation/          Rules, assertions, scores, specs, comparison
│   ├── observability/       Events, traces, graphs, metrics, budgets, monitors
│   ├── tools/ mcp/ sandbox/ Controlled execution surfaces
│   ├── auth/ permissions/ policy/ security/ secrets/
│   └── prompts/ parsers/ providers/ storage/ testing/
├── agents/                  Agent definitions
├── prompts/                 Versioned prompt assets
├── workflows/               Executable workflow examples
├── examples/                Compatibility examples and guide
├── scenarios/ redteam/      Deterministic failure and attack inputs
├── tests/                   Unit, security, lab, and end-to-end tests
├── config/                  Example pricing configuration
└── docs/                    Architecture, operations, security, and release docs

Testing and development

All default tests use local mock providers and require no external API access:

pnpm install --frozen-lockfile
pnpm typecheck
pnpm lint
pnpm build
pnpm test
pnpm test:e2e
pnpm test:security
pnpm demo
pnpm mawl doctor

To extend the framework, add YAML agents under agents/, versioned prompts under prompts/, workflows under workflows/, tool manifests through ToolRegistry, MCP adapters through McpConnection/McpConnector, rules through RuleEvaluator, and scenario coverage under tests/ or scenarios/. See the documentation index and contributing guide.

Roadmap

• distributed execution and atomic shared budgets;
• hardened remote container or microVM sandbox providers;
• OpenTelemetry exporters and remote observability storage;
• additional model, identity, and MCP adapters;
• workflow visualization UI and benchmark datasets.

Contributing, security, and license

Contributions to agents, prompts, tools, MCP connectors, workflow scenarios, evaluators, and security tests are welcome. Read CONTRIBUTING.md and CODE_OF_CONDUCT.md.

Report vulnerabilities privately as described in SECURITY.md. Do not place sensitive vulnerability details in public issues.

Licensed under the MIT License.