Agent Skill Runtime Intelligence

English · 简体中文 · 繁體中文 ·
Français · Deutsch · Italiano ·
Español · 日本語 · 한국어 ·
Русский · Português (Brasil) ·
Türkçe · Polski · Čeština ·
Magyar

Turn SKILL.md into checkable runtime expectations. See what actually
happened, where behavior first diverged, and the evidence behind the judgment.

Agent Skill Runtime Intelligence is a passive runtime-intelligence system for
Agent Skills. It extracts conservative, inspectable constraints from the
current Skill definition, matches them to runtime activity, and reconstructs
the result as an evidence-graded Skill Run Panorama. It can run on a developer
workstation or as an authenticated self-hosted remote service. Collection is
non-intervening and read-only by default: SRI does not proxy model requests,
take over the Agent loop, block Agent actions, or modify the observed source
workspace. Evidence remains inside the operator-controlled deployment boundary
unless trace import or OTLP/HTTP export is explicitly configured.

Quick start

Install and start the latest public release on macOS or Linux:

curl -LsSf https://raw.githubusercontent.com/hellogxp/skill-runtime-intelligence/main/scripts/install.sh | sh

No clone, account, sudo, or GitHub CLI is required. The installer verifies
the GitHub Release checksum, detects supported Agents and Skills, explains
every path it will read, asks once before enabling observation-only hooks, and
opens the local UI at
http://127.0.0.1:4317. Runtime data stays under
~/.skill-runtime unless you explicitly configure an export.

Declining Hook access does not prevent SRI from starting or indexing supported
existing records. You can grant access later from Settings → Collection
health, or run skill-runtime setup --enable-hooks.

You can inspect the public installer before running it.

See your first live SkillRun

1. Accept the optional fail-open Hook setup when the installer asks.
2. Restart the Agent and begin a new task. In Codex, review the managed
   commands in /hooks first; existing tasks do not hot-load new Hooks.
3. Use a Skill normally, then confirm the integration and open the UI:

skill-runtime doctor
skill-runtime status

An integration is Live only after the Collector receives a real runtime
event. A configured but unobserved Hook is Pending—never presented as live
evidence. Open http://127.0.0.1:4317, or see the
Getting Started guide for Agent-specific
instructions and troubleshooting.

To run directly from a source checkout:

python3 -m venv .venv
.venv/bin/python -m pip install -e .
.venv/bin/skill-runtime install --enable-hooks
.venv/bin/skill-runtime start

How it works

Skill Runtime observes the workflow you already use. Versioned adapters turn
Agent-native events into a stable Skill lifecycle, while raw source envelopes,
normalized events, relationships, and inferences remain separate. The
diagnosis engine checks explicit Skill constraints against that evidence,
identifies the earliest observable deviation, and keeps systemic adapter blind
spots separate from run-specific findings. It does not invent model intent or
causal effectiveness. The Collector, evidence store, and UI may run together on
a developer workstation or inside an authenticated self-hosted deployment;
optional observability import and export do not determine that placement.

Supported first-party adapters are versioned independently:

Exact capability limits are documented in the
adapter capability matrix. Unsupported and
not-observed stages stay visible instead of being converted into failures.

The problem

Installing a Skill does not prove that an agent discovered it. Discovery does
not prove activation. Activation does not prove that the full instructions and
resources were loaded. Loading instructions does not prove that the Agent
followed them. Execution does not prove that the Skill improved the outcome.

Today, these failures are often silent. Developers are left asking:

• Was the Skill available to this agent?
• Did it activate for this request?
• Which instructions, references, scripts, and assets were loaded?
• Which explicit Skill requirements were followed, missed, or impossible to evaluate?
• Which tools, MCP calls, subagents, files, and artifacts were involved?
• Where did the run fail, retry, or lose context?
• Did the Skill help, or did it only add cost and latency?

Skill-specific diagnosis

The primary diagnostic object is a SkillRun, not an entire Agent session:

User request
    ↓
Skills discovered
    ↓
Skill selected / not selected
    ↓
SKILL.md activated
    ↓
References and scripts loaded
    ↓
Tools / MCP / subagents executed
    ↓
Files and artifacts produced
    ↓
Observable outcome

The UI keeps the lifecycle ordered, typed, and evidence-graded. Missing
activation telemetry means “not observed” or “unsupported”; it does not mean
the Agent definitely skipped the Skill.

Evidence discipline

The UI must never present an inference as a runtime fact:

• Observed — explicitly present in a source event or file.
• Derived — deterministically connected from observed evidence.
• Inferred — a plausible explanation with uncertainty.
• Experimental — an effect measured through controlled paired evaluation.

A single trace can support execution attribution. It cannot prove causal effectiveness. Claims such as “this Skill improved success rate” require repeated with-Skill/without-Skill evaluation.

Product principles

• Operator-controlled by default, on a developer workstation or in a
  self-hosted remote deployment.
• Non-intervening, read-only collection; never take over the Agent loop, block
  Agent actions, or modify the observed source workspace.
• No model proxy and no mandatory cloud service.
• No blocking, approval gate, or policy enforcement in the default product.
• Explicit, opt-in trace import and OTLP/HTTP export, independent of deployment
  placement.
• Explicit provenance and evidence grading.
• Progressive disclosure: simple narrative first, raw events on demand.
• Adapter-based support for changing agent transcript formats.

Current scope

The runtime supports Codex, Claude Code, Qoder, and OpenCode through
independent, versioned adapters and provides:

• installed Skill discovery and validation;
• real-time official Hook/plugin collection plus labeled session fallback;
• Skill activation, resource loading, and tool-call timelines;
• subagent, MCP, file, and artifact relationships;
• duration, token, error, retry, and status summaries when available;
• conservative behavior constraints extracted from the current SKILL.md;
• evidence-bounded conformance, verification, and runtime-failure checks;
• concrete instruction, resource, tool, artifact, and outcome inventories;
• Runtime Overview with systemic coverage limits separated from run findings;
• first-boundary diagnosis;
• a panorama DAG, event timeline, and evidence inspector;
• capability-aware same-Agent and cross-Agent comparison;
• a separate Inferred Analysis surface that cannot rewrite runtime facts;
• opt-in OTLP/HTTP export and supported observability-trace import.

Deployment placement and observability interoperability are independent. SRI
can run locally or as an authenticated self-hosted remote service; remote
placement does not require connection to an external observability platform.
See Self-hosted remote deployment.

The MVP does not include a marketplace, universal agent runtime, security enforcement, enterprise governance, or causal-effect claims.

Detailed installation

For the shortest supported path, use the one-line release installer in
Quick start. The complete first-run flow, Agent-specific
restart/trust steps, privacy behavior, and troubleshooting live in the
Getting Started guide.

For development, the baseline implementation has no runtime dependencies
beyond Python 3.9+. From the repository root:

python3 -m venv .venv
.venv/bin/python -m pip install -e .
.venv/bin/skill-runtime install --enable-hooks
.venv/bin/skill-runtime start

Then open http://127.0.0.1:4317.

The one-time install command:

1. scans user, project, and cached-plugin Skill locations;
2. detects Codex, Claude Code, Qoder, and OpenCode without changing their
   configuration;
3. shows which Agent and Skill paths will be read;
4. downloads a checksum-verified low-startup native sender for the current
   platform, falling back to a local C build and finally the Python sender,
   and prewarms a fresh native binary once during installation;
5. creates ~/.skill-runtime/config.json and the local SQLite index.

The first index imports existing compatible Agent sessions. On a long-lived
workstation this can take longer than a fresh installation; later starts are
incremental and the UI becomes available while the background refresh runs.

When run interactively, it asks once before adding fail-open Agent hooks.
--no-hooks keeps transcript import as the labeled fallback, while
--enable-hooks records explicit consent and installs only managed entries.
For Codex, open /hooks after installation, review the exact managed commands,
and trust them. Codex intentionally requires this explicit review for hooks
added outside managed enterprise configuration. Start a new Codex task/session
after trusting the Hooks, then run:

.venv/bin/skill-runtime doctor

Qoder loads Hook configuration at startup, so restart Qoder after first
installation. OpenCode discovers the managed observation-only plugin from its
global plugin directory; restart OpenCode if the current process predates
installation. Neither integration reads or changes model requests.

The integration becomes Live only after the database receives a real
official_hook event. Merely writing ~/.codex/hooks.json is shown as
Pending, never Connected. start launches the Collector, transcript
fallback watcher, retention worker, SQLite store, and live UI as a managed
background process. No model request is proxied.

Lifecycle commands:

skill-runtime status
skill-runtime doctor
skill-runtime restart
skill-runtime stop
skill-runtime config --set retention_days=30
skill-runtime config --set network_export.endpoint=https://collector.example/v1/traces
skill-runtime config --set network_export.enabled=true
skill-runtime uninstall --keep-data

uninstall removes only managed Hook entries and Skill Runtime-owned files.
Without --keep-data, it requires interactive confirmation (or --yes) before
removing ~/.skill-runtime; Agent sessions and Skill sources are never removed.

To index and serve separately:

PYTHONPATH=src python3 -m skill_runtime_intelligence index
PYTHONPATH=src python3 -m skill_runtime_intelligence serve

Import an existing trace export from a mainstream observability system:

PYTHONPATH=src python3 -m skill_runtime_intelligence import \
  ./trace-export.json \
  --format auto

The versioned import profiles currently recognize OTLP/Phoenix, Langfuse,
LangSmith, W&B Weave, and Datadog JSON shapes. They only create a SkillRun when
the source carries explicit Skill semantics; generic span names are not treated
as activation evidence.

Export normalized, Skill-specific runtime evidence to any OTLP/HTTP traces
endpoint:

.venv/bin/skill-runtime start \
  --otlp-endpoint https://collector.example/v1/traces \
  --otlp-header Authorization='Bearer …'

Export is disabled unless an endpoint is explicitly configured. Checkpoints,
retry status, and destination health are shown in Settings. Raw prompts, tool
payloads, credentials, and Skill resource contents are not exported.
For authenticated background export, provide standard
OTEL_EXPORTER_OTLP_HEADERS in the environment before skill-runtime start;
headers are never written to Skill Runtime configuration or process arguments.

Send live runtime evidence

skill-runtime start includes a local Collector. Native telemetry adapters,
official hooks, lightweight fail-open hooks, and SDK integrations can append a
single event or a bounded batch to POST /api/events:

curl -X POST http://127.0.0.1:4317/api/events \
  -H 'Content-Type: application/json' \
  -d '{
    "event_id": "evt-example-activation",
    "event_type": "skill.activated",
    "occurred_at": "2026-07-29T05:00:00Z",
    "session_id": "agent-session-example",
    "turn_id": "turn-1",
    "activation_mode": "explicit_tool",
    "skill": {"name": "pdf"},
    "source": {
      "adapter": "example-agent",
      "adapter_version": "1.0",
      "collection_mode": "official_hook",
      "source_event_id": "source-event-1"
    },
    "evidence": {
      "grade": "observed",
      "confidence": 1.0,
      "basis": "Official runtime hook"
    },
    "payload": {"tool_name": "Skill"}
  }'

The endpoint redacts common credentials before persistence, deduplicates by
event_id, preserves a separate redacted raw envelope, and returns the
resulting skill_run_ids. GET /api/collector/schema exposes the supported
event vocabulary and collection modes. The UI listens to /api/stream using
SSE, with polling only as a reconnect fallback.

The source indicator distinguishes primary runtime evidence from
Transcript fallback and imported traces. A Collector endpoint alone does not
claim native telemetry: every producer must declare whether its event came
from native telemetry, an official hook, a lightweight hook, or an SDK.

Optional Agent hooks

Inspect the exact paths and events first. This command is read-only:

.venv/bin/skill-runtime setup

Hook installation requires an explicit flag:

.venv/bin/skill-runtime setup --enable-codex-hooks
.venv/bin/skill-runtime setup --enable-claude-hooks

The installer backs up the Agent configuration, preserves existing hooks, and
adds only entries carrying a Skill Runtime management marker. The hook adapter
stores minimal lifecycle fields rather than full prompts or tool payloads.
For completed tool calls it extracts only exact SKILL.md, standard Skill
resource, and changed-file paths in memory; raw commands, patch bodies, prompts,
and tool outputs are discarded before persistence.
While the runtime is active, a permission-restricted Unix socket is the fast
path; an optional native sender avoids Python startup. When the runtime is not
active, the standalone fail-open path appends redacted evidence to
~/.skill-runtime/queue/events.jsonl. skill-runtime start replays that queue
with event-ID deduplication.

Codex events use its official Hook API (SessionStart, SessionEnd,
UserPromptSubmit, PreToolUse, PostToolUse, PreCompact, PostCompact,
SubagentStart, SubagentStop, and Stop). Codex currently executes command
hooks synchronously, so Skill Runtime uses a local Unix socket/native sender
with a bounded timeout. Any delivery failure is swallowed and queued; it never
changes an Agent decision. See the
official Codex Hook documentation.

Remove only the managed entries with:

.venv/bin/skill-runtime setup --remove-codex-hooks
.venv/bin/skill-runtime setup --remove-claude-hooks

The server binds to 127.0.0.1 by default. Full transcript messages and tool
payloads are not copied into the index. Common secret patterns are redacted
before normalized summaries are persisted.

Run the dependency-free test suite with:

PYTHONPATH=src python3 -m unittest discover -s tests -v

Release engineering

GitHub Actions runs Python 3.9–3.13 tests, JavaScript validation, native sender
compilation, and a real install/start/doctor/stop/uninstall smoke test. A v*
tag builds wheel/sdist packages plus checksum-protected Linux and macOS native
senders. The CLI installer downloads the matching release asset, so end users
do not need a compiler.

Run the first product-linked diagnostics experiment:

python3 experiments/runtime_diagnostics/run_benchmark.py

It fault-injects lifecycle evidence gaps, explicit failures, incomplete runs,
and unverified outcomes, then evaluates the same deterministic diagnosis
engine used by the API and UI. See the
PAI-DSW experiment plan for the experiment
ladder, non-interference tests, and reproducibility contract.

After building the wheel, run the isolated packaged lifecycle smoke with:

PYTHONPATH=src python3 experiments/product_lifecycle/run_benchmark.py

It installs into a temporary virtual environment and temporary home, exercises
the full local lifecycle without enabling hooks, and verifies project and
Agent configuration non-interference.

Experiment-driven product design

Product behavior follows four experiment-driven constraints: evidence before
conclusions, the first observable boundary before severity, typed relationships
before flat logs, and deterministic reconstruction before probabilistic
assistance.

Reproducible evidence and its limitations are maintained in the
experiment report. Bounded results
include:

• 2,400/2,400 Collector events accepted without input/output mutation;
• 14/14 deterministic fault-corpus diagnoses with no unsupported causal claim;
• relational diagnosis representation at 13/14 exact and F1 0.963, while flat
  lifecycle retrieval reached 1/14 exact and F1 0.080;
• a privacy-safe real-run audit that explicitly remains unsuitable for
  confirmatory product-effect claims because verified outcomes, balanced
  cross-Agent coverage, and human labels are missing.

These results validate mechanisms and representation choices, not deployment
generalization or human benefit. Real second-Agent studies, cross-platform
tail latency, real-fault calibration, and participant diagnosis studies remain
open evidence gaps.

The research direction is also grounded in adjacent primary work:
SkillsBench and
SWE-Skills-Bench motivate diagnosis because
Skill effects vary and can regress; Harness-Bench
motivates capability-aware cross-Agent comparison; and the
execution provenance survey motivates typed
evidence relations, trace provenance, and privacy-aware audit infrastructure.

Documentation

Product and research references: product definition,
MVP specification,
observability interoperability,
experiment results, and the
research agenda.

Community and governance

• Read Contributing before changing evidence semantics,
  adapters, or product behavior.
• Follow the Code of Conduct in all project spaces.
• Report vulnerabilities privately through the Security policy,
  not a public issue.
• Use the structured issue tracker
  for reproducible bugs and scoped feature proposals. Never attach private
  runtime databases or session transcripts.

Roadmap

1. v0.3.0 — Next release: checkable Skill behavior constraints, concrete runtime
   activity, evidence-bounded assessment, systemic coverage diagnosis, and the
   existing live Panorama and Compare workflow.
2. Next — Adapter and diagnosis hardening: broader Agent/version coverage,
   real-fault calibration, cross-platform tail-latency validation, and
   participant diagnosis studies.
3. Later — Effect evaluation: controlled with-Skill/without-Skill paired
   evaluation, kept explicitly separate from single-run diagnosis.

Project status

The current source tree targets v0.3.0; use the release badge above to identify
the latest published build. The runtime includes checkable
Skill behavior constraints, concrete activity summaries, installed-definition
inventory, consent-driven official Hook adapters for Codex, Claude Code, and
Qoder, an observation-only OpenCode plugin, labeled transcript fallback,
active-scope attribution, exact file/artifact paths, redaction, separate
source/relationship/inference layers, SQLite storage, retention, deterministic
diagnosis, live UI, and cross-run/cross-Agent comparison. OTLP/Phoenix,
Langfuse, LangSmith, W&B Weave, and Datadog exports can be imported;
normalized evidence can be exported live through opt-in OTLP/HTTP.

Candidate discovery inside the model, model-internal selection reasons,
semantic effectiveness, and causal outcome claims remain explicitly
unsupported unless a source or controlled experiment provides that evidence.