AgentMeasure

Measure agent usage without counting retries as users.

AgentMeasure separates execution facts from logical operations, evidence
from inference, and economic settlement from value — an open measurement
layer for the Agent Capability Economy. One number for your dashboards
(logical operations), the evidence for what each number counts, and explicit
disclosure for what it cannot prove.

Today: measure agent-facing capability usage — attempts, operations, retry
inflation, success rates with numerators you can audit.
Next: make capabilities comparable and meterable.
Long term: provide the measurement foundation for Capability as a Service (CaaS).

Reach → Choice → Use → Utility → Value

AgentMeasure is not a payment protocol, marketplace, or universal ranking system.
It standardizes the facts and measurement semantics those systems can build on.

Website · Free 7-day audit — apply · Whitepaper · 白皮书（中文） · Core Specification · 中文

Start with the story: When the Software Consumer Becomes an Agent (EN) · 当软件的消费者变成 Agent (ZH)

Try it in 2 minutes

./examples/demo-e2e.sh

Mock MCP server → canonical observations → local metrics, all on your machine, no
cloud. The demo is reproducible: it runs in an isolated workspace (never touches
~/.agentmeasure) — same fixture + same policy = same result (42 calls → 84
canonical observations, caller claims claude:14 · codex:14 · unknown:14).

Then read why we audit the ecosystem's usage claims:
Benchmark Run #001 — six real claims profiled with
the Evidence Profile (multi-axis, no composite scores), and
Pipeline Validation #001 — our own fixture,
kept out of the ranking as a reference baseline.

Run a preregistered experiment — AgentMeasure Lab

python3 lab/am lab selftest                                   # planted uplift recovered + honest null
python3 lab/am lab init                                       # workspace + example experiment
python3 lab/am lab preregister am-lab/experiments/example-manifest.json
python3 lab/am lab run am-lab/experiments/example-manifest.prereg.json

The open experiment engine (lab/): task set × harness matrix × factor
variants → Reach → Choice → Success → Consumption funnel → effect sizes with
confidence intervals, guardrails, honest nulls, and an offline HTML report that opens
with a bilingual decision-maker one-pager. Preregistration is enforced (hypothesis /
primary metric / guardrails / analysis plan hashed before the run — with a scale /
power / budget preview), seeds replay deterministically, and a budget circuit breaker
stops safely with the data it has. Selection uplift that loses consumption is
rejected at the decision exit (unverified_growth — do not ship), and candidates
that make no more money at higher cost are flagged as dominated.

The shipped demo runs on a synthetic harness (planted ground truth at realistic,
literature-scale amplitudes, disclosed in every report) — it validates the engine,
not real-agent claims. Real harness adapters (Claude Code / Codex) are runner plugins
against the same interface; that is the highest-value contribution right now.
Docs: lab/README.md · formats:
lab/schemas/ (experiment manifest / funnel events / report).

Why the split matters, in one line

1 user intent
2 provider attempts
1 final success

That is 1 operation, 2 attempts, operation success 100%, attempt success 50%,
attempts per operation 2.

It is not "2 operations with 50% success".

Attempts are execution facts. Operations are logical intents. Mixing the two is how agent
reliability reports get distorted.

External provider trials

AgentMeasure is looking for 3 MCP/API providers for short external measurement runs.

The goal is not to prove the model works. It is to find where it breaks on real provider traffic.

A trial can stay fully local, requires no agent-side install, and can start with one capability.

Best feedback: a concrete trace, counterexample, or field that cannot be measured safely.

→ The formal offer, schedule, and FAQ live in product/AUDIT.md · apply in Issue #2 — Looking for 3 MCP/API providers for external measurement runs.

Why capabilities need a new measurement layer

The software consumer is changing from humans to agents, and the economic unit is shifting from software seats toward callable capabilities.

Skill / MCP / CLI / SDK
        ↓
Describe / expose / distribute a capability

Capability
        ↓
Data / Compute / Action / Permission / Transaction
        ↓
Creates scarce economic value

Interfaces may become cheap to create; capabilities remain scarce to deliver.

The first generation of capability distribution is already here — open Skills, open MCP adapters, open CLIs. The scarce layer underneath them is what the next economy is built on: proprietary data, compute, execution, permissions, and real-world fulfillment.

From software to capability economy

Human Software Economy
User → UI → SaaS → Seat / Month

             ↓

Agent Capability Economy
Agent → Capability → Execution → Outcome
                       ↓
              Usage / Value / Transaction

If a capability is to become an economic unit that agents can discover, compare, and eventually purchase automatically, it must first be identifiable, measurable, and comparable under a shared semantics. That is what AgentMeasure provides.

Traditional usage metrics cannot support this economy — the old chain breaks at every link, and the last link is new:

Install ≠ Available
Available ≠ Presented
Presented ≠ Selected
Selected ≠ Used
Used ≠ Useful
Useful ≠ Incremental Value
Measured Usage ≠ Billable Usage

The last inequality is why the Operation/Attempt model matters commercially: 3 attempts of one operation are not 3 billable operations — unless the metering policy says so.

The extended thesis — economic units, scarcity, measurement before monetization,
metering semantics, and the evidence that commerce is arriving before measurement —
lives in docs/CAPABILITY-ECONOMY.md.

The measurement view: Reach → Choice → Use → Utility → Value

AgentMeasure defines metric families, not a universal KPI. A search capability, a booking API, and a compute job have different value structures.

The five layers are the measurement view. The same facts map onto an economic view:

Claim discipline throughout: observed choice is not preference. A selection can be made by the model, a router, a workflow, the user, a policy, or the platform; Observed Selection Rate reports what was observed, and Observed Head-to-Head Choice Share is an observed head-to-head choice share under comparable candidate conditions — comparable means the same candidate set, category, choice mode, and decision axes (Decision Authority / Selection Constraint) are declared.

How AgentMeasure works in production

Agent Runtime                     Capability Provider

Claude / Codex
      │
      │ MCP / API
      ▼
                         ┌──────────────────────┐
                         │ Customer Capability  │
                         │                      │
                         │ AgentMeasure SDK     │
                         │ Business Handler     │
                         └──────────┬───────────┘
                                    │
                               observations
                                    │
                                    ▼
                              Collector
                                    │
                                    ▼
                            AgentMeasure Cloud

Four things a real developer needs to know:

Your software does not need to be open source.

MCP is not required — it is the first reference surface.

Third-party agents do not need AgentMeasure installed for provider-side usage measurement.

AgentMeasure is not on the critical request path.

Product architecture (Provider SDK → local buffer → hosted ingestion → dashboard):
product/ARCHITECTURE.md.

What AgentMeasure measures today

• Decision Opportunity / Candidate Set / Presentation / Selection — the four objects of choice; the Observed Selection Rate denominator is Presented, not Available
• Software Entity → Capability → Interaction Surface — what exists, what it can do, and the observable interface; observation happens on surfaces, attribution resolves to entities through the machine-readable registry
• Operation / Attempt — one logical use vs. one execution; retries are multiple attempts of one operation, not a validity class (they are kept as reliability signals, not counted as distinct logical uses)
• Qualified Usage — production usage after excluding test, benchmark, synthetic, replay, duplicate and other invalid traffic according to policy
• Result Consumption — defined, reference partial: the result was used by the task
• Effect Confirmation — domain model defined, metric planned for Draft 0.5: the intended world-state change was confirmed
• Measurement Label — a nutrition label for every public number (coverage / sampling / policy / method)

The full model lives in the Core Specification, Metrics, Entity, and Quality. The README is not a spec summary.

From measurement to CaaS

AgentMeasure is progressively standardizing the measurement chain from discovery
and choice through execution, utility and value: core usage semantics are defined
today; utility/value and commercial metering remain active drafts. Metering and
commercial attribution are future extensions; payment rails can be provided by
existing payment infrastructure.

AgentMeasure standardizes economic facts, not money movement.

Extended thesis: docs/CAPABILITY-ECONOMY.md ·
Economic semantics: extensions/COMMERCIAL.md（Experimental）

Harness profiles — what each runtime can (and cannot) observe

Portable semantics need a public record of observation blind spots. Per-harness
profiles map each runtime's native objects to AgentMeasure semantics:

As harnesses compose software at runtime, one behavior gets described by
different objects and units in different runtimes. Experiment
D turns that into
evidence; Proposal: Delegation
defines the agent-to-agent boundary the object model was missing.

What AgentMeasure is / is not

Who it is for

Try the standard

git clone https://github.com/roy-tong/AgentMeasure && cd AgentMeasure
python3 conformance/runners/run_metrics.py   # metric vectors (M2.2 / M2.5 / M4.1)
python3 verify_vectors.py                     # verification / correlation / operation vectors
python3 registry/validate_entities.py         # validate the machine-readable registry

Product MVP — first real measurement (in development)

The first product path is Remote MCP / API Capability Measurement: an
AgentMeasure Provider SDK (@agentmeasure/mcp) that emits observations from
the provider side (no agent-side install), feeding a local collector. Local analytics
run without any cloud:

npm install https://github.com/roy-tong/AgentMeasure/releases/download/v0.1.1/agentmeasure-mcp-0.1.1.tgz
# (npm registry publish pending scope/token — tarball is the current install path)
# wrap your MCP server's tool handlers: server.tool = (name, schema, mw.wrapTool(name, handler))
node examples/mcp-integration.js          # synthetic traffic → local JSONL
python3 product/local-analytics.py ~/.agentmeasure/events/agentmeasure-events.jsonl

Status: SDK v0.1.1 — External-Ready (canonical output, non-blocking spool with
loss accounting, per-request caller, MCP v1/v2, 21 tests, deterministic fixture) +
local analytics implemented; hosted ingestion and dashboard next. First real
external Provider = Product Gate A (ROADMAP.md, MVP.md).

Scope and acceptance: product/MVP.md · SDK contract:
product/PROVIDER-SDK.md · Deployment:
product/DEPLOYMENT.md

Repository map

AgentMeasure/
├── standard/          # the normative standard (CORE / METRICS / QUALITY / DATA / ...)
├── extensions/        # experimental, non-normative profiles (COMMERCIAL.md)
├── product/           # product architecture (SDK / hosted analytics, in development)
├── whitepaper/        # methodology papers (EN/CN)
├── conformance/       # language-neutral test vectors + runners
├── reference/         # reference implementation (collector + adapters)
│   ├── collector/     #   normalization, correlation, aggregation, evidence
│   └── adapters/      #   codex / claude / dsh / mcp observation adapters
├── schemas/           # machine-readable schemas (entity registry)
├── registry/          # machine-readable registries (entities / project identity)
├── experiments/       # empirical experiment designs
├── reports/           # public reports (Discrepancy Report)
├── proposals/         # standard change proposals (AUP)
└── archive/           # retired early documents

The standard is the artifact; the code is a reference implementation. Using the standard does not mean uploading data to any central server.

Current status & roadmap

Draft 0.4.4（Canonicalization & Reference Convergence） — 唯一 Canonical
Observation（schemas/observation.schema.json，6 类 payload）；Choice/Execution 从同一
Envelope 派生；M3.1 只计已解析 operation（无回退）；Attempt 级 qualification 派生；
metrics.yaml 单一事实源；四维正交（Evidence/Caller/Use Profile/Billing）。

The roadmap runs on two tracks — the standard (0.4 objects & quality → 0.5 utility & economic semantics → 1.0) and the product (Remote Capability Analytics → Provider SDK + hosted analytics → metering). See ROADMAP.md.

Contribute

• Join the community: Discussions — categories: Metric Semantics · Runtime Observation · Experiments · Capability Economy · Implementers; ground rules in docs/DISCUSSIONS.md
• Where to start: the open debate on Strict Qualified Usage as the default, or onboard the first external Provider
• Propose standard changes: proposals/ (AUP: Draft → Discussion → Accepted → Experimental → Stable)
• Report measurement discrepancies: reports/ (Discrepancy Report template)
• Fix the reference implementation: PRs must pass all conformance/ vectors

AgentMeasure does not define who owns the truth. It defines what evidence, under what rules, can support what conclusions.