Arbiter

Live AI judge agent for hackathons. Watches demos in real-time via camera and microphone, delivers entertainment commentary with a British accent, scores projects against a rubric using a multi-model ensemble, detects prompt injection attempts, and runs cross-team deliberation with full memory of every demo.

Built for the NEBULA:FOG 2026 security hackathon, where it judged 25 live demos on March 14, 2026.

"Twenty-five teams walked in with 24 hours of code and a live demo slot. What they didn't expect: an AI judge watching every second."

What It Does

• Real-time observation — Connects to Gemini Live API, streams audio/video, generates observations as presenters speak
• Multi-layer injection defense — Regex denylist, semantic classifier (rubric echo, self-eval, fabricated evidence), multi-language detection (7 languages), XML boundary tags, dual-LLM privilege separation
• Runtime Alignment Control Plane — Every consequential effect (crossing the privileged-LLM boundary, scoring, speaking, writing a scorecard) is gated by an expiring, signed behavior lease. Preflight proves the detection policy still works before each demo, an independent monitor re-checks what the sanitizer produced, and effects fail closed with a hash-chained audit trail — see docs/racp.md and how it works
• AI commentary — Generates sharp, persona-driven reviews delivered via Cartesia TTS (British voice)
• Multi-model scoring — Gemini, Claude, and Groq independently score each demo, aggregated with outlier detection
• Theatrical score reveal — Animated criterion-by-criterion reveal on the audience display
• Q&A generation — Generates pointed follow-up questions based on what was actually said in the demo
• Cross-team deliberation — After all demos, compares every team against every other team for final rankings
• Operator dashboard — React WebSocket control panel for managing demo lifecycle
• Audience display — React real-time display for commentary, scores, and leaderboard

Architecture

Capture Layer ──→ Defense Pipeline ──→ Commentary ──→ Scoring ──→ Deliberation
  (audio+video)    (injection guard)    (persona TTS)   (MoE rubric)  (memory)
                          │                  │              │
                          └────────── RACP action gateway ──┘
                                (behavior lease required per effect)

Dual-LLM privilege separation: A quarantined Gemini Live session processes raw input. The privileged judging LLM only sees sanitized observations — never raw camera frames or audio.

Quick Start

Prerequisites

• Python 3.11+
• uv (Python package manager)
• bun (JS package manager, for frontend builds)
• A Gemini API key

Setup

git clone https://github.com/basicScandal/arbiter.git
cd arbiter
uv sync
cp .env.example .env
# Edit .env — at minimum set GEMINI_API_KEY

Build frontends

cd operator-dashboard && bun install && bun run build && cd ..
cd audience-display && bun install && bun run build && cd ..

Run

# Live mode (requires camera + microphone)
uv run python -m src.main

# Rehearsal mode (no hardware or API keys needed)
uv run python -m src.main --rehearsal

URLs

Configuration

All configuration is via environment variables (see .env.example):

Testing

# Full suite (1451 tests)
uv run pytest

# Smoke tests only (fast go/no-go gate)
uv run pytest -m smoke

# Specific module
uv run pytest tests/test_semantic_detection.py -v

Security

Arbiter is designed to operate in an adversarial environment where participants actively try to hack the judge. The defense stack has four layers:

The system was red-teamed post-event by three parallel AI agents that identified 11 findings (all fixed in v1.1.0). See the full report and slide deck.

How Scoring Works

Each demo is scored across three criteria:

Plus a track-specific bonus (+10%) based on the team's chosen challenge track.

Scoring uses a Mixture of Experts (MoE) approach: Gemini, Claude, and Groq independently evaluate each demo. Scores are aggregated with outlier detection — if one model disagrees wildly, it's flagged and its influence is reduced.

Event Day Results

The data/ directory contains all event outputs:

• data/scores/ — Per-team scorecards with criterion breakdowns
• data/observations/ — Raw observations and transcripts from each demo
• data/deliberation/ — Cross-team comparative analysis
• data/events.jsonl — Complete event log
• data/audit.jsonl — Full audit trail

Docs

• How We Built Arbiter — full writeup with architecture, what broke, and lessons learned
• Red Team Report — 11 prompt injection findings from 3 parallel AI red team agents
• Red Team Slides — 10-slide HTML deck summarizing findings
• Architecture
• Operator Guide
• Pre-Event Checklist
• Judge Instructions
• Prize Breakdown
• Zoom Capture Setup

See It In Action

• SINGULARITY 2026 Results — full scoreboard with 25 teams
• PRIME 2025 Results — retroactive scoring of 15 demos from video
• YouTube: NEBULA:FOG — demo recordings

License

MIT