otelcol-genai-sketches

genaisketch is an OpenTelemetry Collector connector and ready-to-run distribution.
It turns GenAI spans into token metrics, keyed rankings of users and sessions, and
summary files, while keeping user, session and prompt identities out of metric
labels. Your existing trace backend keeps receiving the original spans.

Use it to track LLM token usage by user, session and model, and to see where usage
data is missing, in Prometheus or any OTLP metrics backend.

Synthetic demo traffic.

Try It

You need Docker with Compose v2, Git, and a POSIX shell (macOS, Linux, or WSL).
The demo is synthetic and needs no model account or API key.

Clone the repository and start the stack:

git clone https://github.com/llm-measurement/otelcol-genai-sketches.git
cd otelcol-genai-sketches
sh examples/demo.sh up

You'll get a sample app, this collector, a Prometheus metrics server, and Grafana.
Open the dashboard printed by the command to see request rate, reported token
rate, distinct activity, and missing usage. The first build takes several minutes;
allow one minute after startup for the first measurement window.

See the token-heavy prompt signatures and their bounds:

sh examples/demo.sh topk

Then run two controlled investigations:

sh examples/demo.sh investigate

You'll see why adding tool spans leaves model-request counts unchanged, and why
missing usage can make reported tokens fall while consumption stays the same.

Watch the walkthrough, or open the
demo guide for ports, secret handling, dashboard reading
order, and cleanup. The token-consumption playbook gives
queries for investigating your own traffic, including unexpected "token maxing."

Find Your Guide

How It Works

applications -> Collector fan-out -> current trace backend
                                  -> genaisketch -> Prometheus metrics
                                                -> keyed top-k snapshot logs
                                                -> optional summary files -> fleetdiff

Keep Datadog, Langfuse, Alloy, or another OTLP destination as your trace backend.
The connector counts model attempts separately from agent, tool, retrieval, and
MCP spans, and records missing token usage explicitly.
It uses llm-sketchkit to hash
configured identities, estimate distinct activity, and rank heavy contributors.
Compatible summary files let fleetdiff
compare windows across collectors without uploading raw traces.
Fan-out preserves the original traces, so disable content capture or redact
before fan-out when raw content must stay local.

See Shadow Mode for fan-out configurations and
Accounting for request and token rules.
The metrics reference, configuration guide,
and deployment FAQ
cover the available signals and required input fields.

Install

For a persistent deployment, use the signed release image or Helm chart.
Follow Deployment to verify signatures and attestations,
pin immutable digests, configure the hashing secret, and start the collector.

The image supports Linux amd64 and arm64. For Kubernetes, the deployment guide
includes the verified chart install and an existing-Secret configuration.

Already build your own collector? Add the standalone connector module to your
OpenTelemetry Collector Builder
manifest:

connectors:
  - gomod: github.com/llm-measurement/otelcol-genai-sketches/connector/genaisketchconnector v0.3.2

Use genaisketch as a traces-pipeline exporter and a metrics-pipeline receiver;
the component README and
configuration guide have the connector settings.
Read Sizing for capacity planning and
Upgrading for feature availability, restarts, and rollback.

Privacy And Limits

• Keyed hashes are pseudonymous. Values are linkable under the same secret;
  secret holders can test candidate values. Restrict access to snapshots and
  summary files. Key rotation starts a new comparison period.
• Slice labels are bounded and cleartext. Choose non-sensitive dimensions
  such as model, team, or route. Excess slice values share an overflow bucket;
  user/session hashes and top-k items stay out of metric labels.
• Estimates carry uncertainty. Distinct counts are estimates; frequent-item
  rankings include lower and upper bounds. Use compatible summary state to
  combine measurements, rather than adding distinct-count gauges.
• Use exact records for billing and quota enforcement. Reported usage and
  coverage support investigation; optional Bloom deduplication can undercount.
  Keep provider records and an exact ledger for financial or admission decisions.

See Security for private vulnerability reporting and
Accounting for measurement semantics.

Project

Status: Alpha. Signed images, a Helm chart, SBOMs, provenance, and upgrade
guidance are available. Start with an evaluation deployment and representative
traffic; configuration and metric semantics may change before 1.0.
See the changelog and release and support policy.

The benchmark report records reproduction commands, workloads,
machine details, and all recorded accuracy, throughput, and soak runs.

Contributing covers development and signed, signed-off commits.
For questions or feedback, open an issue.

Licensed under the Apache License 2.0.