EKOS — Enterprise Knowledge Operating System

EKOS is the open-source knowledge layer for AI agents: a compiler that turns source code, SQL,
documentation and Git history into evidence-backed knowledge that agents query over MCP. Project
site and documentation: ekos.dev. Install: ekos.dev/docs/getting-started/installation
(a setup wizard in your browser on Linux, Windows 11 and macOS — see Installation).

Unlike traditional enterprise systems that manage data, documents or metadata independently, EKOS treats the entire enterprise as a living knowledge system — a permanently evolving semantic model that can be trusted by both humans and AI.

First benchmark: on a real 2,022-file open-source repo (plausible/analytics), cold ingestion takes 34 seconds, and answering real questions from the compiled ledger costs 67-93% fewer tokens than raw grep-based search over the source — measured with a standard tokenizer (tiktoken), not a hand-rolled estimate, with the one case grep wins included rather than hidden. Full methodology, every command, and every raw output: The First Benchmark Number.

About

EKOS is a compiler for enterprise knowledge, not a database or document store. It observes an
enterprise's existing systems — source code, Git history, SQL schemas, GitHub issues/PRs,
Confluence, local PDF/DOCX documents — without interpreting them, compiles
those observations through deterministic passes into a Canonical Knowledge Model, and stores the
result in an append-only ledger where every conclusion carries the evidence it was derived from. AI
agents (Claude Code among them) read that ledger through a read-only Model Context Protocol server
(ekos mcp serve, RFC 0013) — they never touch raw enterprise systems directly.

The project follows an RFC-first workflow (docs/rfcs/): every capability is designed in writing
before it's implemented, and the devlogs/devlog_*.md files are the running record of
what shipped, why, and what was learned building it. It is written in Rust (2024 edition) as a
Cargo workspace, and is licensed under the MIT License.

The Problem

Modern enterprises contain enormous amounts of valuable knowledge distributed across disconnected systems: source code, databases, data warehouses, documentation, wikis, Git repositories, infrastructure-as-code, APIs, runtime logs, and monitoring systems. Every system contains only a partial description of reality. Documentation becomes outdated. Employees leave. Business logic remains hidden inside production code. AI assistants receive fragmented, inconsistent, and often contradictory information.

Enterprises continuously lose knowledge.

The Insight

The enterprise already contains its own documentation — embedded inside source code, SQL, infrastructure definitions, APIs, logs, deployment history, schemas, and runtime behaviour. The problem is not missing information. The problem is the absence of a compiler capable of transforming enterprise reality into enterprise knowledge.

EKOS is that compiler.

Architecture

          Enterprise Systems
 Git   SQL   APIs   Confluence   Logs   Cloud   Monitoring
                        |
                 Observation Layer        ← collects facts, no interpretation
                        |
               Knowledge Compiler         ← multi-pass: normalize → analyze → recover → verify
                        |
          ┌─────────────┴─────────────┐
   Knowledge Recovery          Identity Resolution
          └─────────────┬─────────────┘
                        |
          Canonical Knowledge Model (CKM)  ← language/storage/AI-provider independent
                        |
           Semantic Knowledge Ledger        ← append-only, every fact traceable to evidence
                        |
          ┌─────────────┴─────────────┐
    Knowledge Runtime          Knowledge Services
          └─────────────┬─────────────┘
                        |
            AI Agents & Enterprise Applications

Semantic Primitives

The ledger stores four immutable primitives:

Primitive	Description
Object	Identity of a concept: Customer, Product, Dataset, Service, Business Rule
Relationship	Semantic connection between objects (first-class, not just a foreign key)
Event	Immutable change — the only mechanism that mutates enterprise state
Evidence	Origin of knowledge: SQL query, source code, Git commit, log line, API spec

Every semantic conclusion is supported by evidence. Every change is auditable.

Key Invariants

• The Observation Layer collects facts only — it never interprets business meaning.
• The ledger is append-only — knowledge is never modified in place.
• The Runtime is read-only — it reconstructs and interprets state, never modifies it.
• AI systems consume reconstructed knowledge through the Runtime; they never touch raw enterprise systems directly. The one explicit, scoped, audited exception is the gated ekos_clickhouse_query MCP tool (RFC 0056, off by default) — see below.
• Every compiler pass is deterministic and side-effect-free.
• Every artifact is content-addressable (id + checksum + metadata + dependencies + version).
• Secrets and PII are never observed or stored (RFC 0043) — a built-in baseline redacts known
  secret shapes (AWS/GitHub/Slack/Google/Stripe tokens, private keys, JWTs, generic
  key/password assignments) from all observed content, and excludes files like .env/*.pem/
  id_rsa entirely. ekos.toml's [security] section can only extend this baseline, never
  disable it.

Implementation

Language: Rust (2024 edition), Cargo workspace.

Crates (ekos/crates/): compiler-core, compiler-sdk, observation-sdk, artifact, kir,
scheduler, sql-dialect-sdk, ledger, runtime, identity, recovery, ekl, semantic,
marketing, docs-gen, dbt-gen, common, cli, demo-server, segment-backend / cluster /
distributed (RFC 0111/0113's partitioned + horizontally-distributed storage — see below),
simulation (RFC 0047-0055's opt-in World Engine — see below), clickhouse-query (RFC 0056's
opt-in live NL-to-SQL query engine — see below), evals (RFC 0138's answer-grading harness — see
below), session (RFC 0151 agent session memory), plpgsql (RFC 0163's PL/pgSQL parser and
procedural IR), and EKOS Migrate's migrate, migrate-dq, migrate-logic, migrate-validate, migrate-approval,
migrate-report, migrate-target-clickhouse and pg-live (RFC 0154–0162 — see below). The same workspace also builds ekos with a generic extension seam (RFC 0149) that out-of-tree
builds plug extra connectors and MCP tools into.

Connectors (ekos/plugins/): File, Git, GitHub issues/PRs (live-verified against a real
repo, 1,600 real issues/PRs — RFC 0062), Confluence, local documents
(PDF/DOCX/text/Markdown/HTML/email — text, tables, image OCR), Pentaho Kettle (.ktr/.kjb —
RFC 0027), Python/PySpark source (real AST parsing, DataFrame chains recovered into the
Transformation IR — RFC 0038/0040), Rust source (real AST parsing, real function-call graph —
RFC 0041), Elixir and JavaScript/TypeScript source (real AST + Calls recovery — RFC 0081/0082),
Perl source (.pl/.pm/.t/.psgi, plus .cgi with a real perl shebang — packages, subs,
use/require dependencies, use parent/use base/@ISA inheritance, POD descriptions and
source spans; RFC 0147), compiled .NET and JVM binaries with no source available
(.dll/.exe/.class/.jar/.war/.ear — types, methods, signatures, fields, the real call
graph, branch structure, string/numeric constants and external I/O boundaries, read in-process
from CLI metadata and bytecode with no .NET SDK or JRE required; for .NET, per-method statement
recovery — if/loop/switch/try with the IL offset of every line, labelled with the fidelity actually
reached — plus a per-method migration spec, a Python-rewrite parity check and sandboxed characterization tests that
run the original and check the rewrite against what it did, RFC 0150; RFC 0148 — shipped
as a separately licensed extension build, not part of this open-source repository, RFC 0149), ClickHouse (real HTTP client, schema metadata plus an opt-in live query engine — RFC
0056), plus scaffolded proof-of-concept clients for Salesforce, SAP, Oracle,
Microsoft Fabric, and Snowflake (real API shapes, mock-tested — none yet exercised against a live
account). SQL in the PostgreSQL, SQL Server (T-SQL), MySQL, Snowflake, Databricks and ClickHouse dialects is
parsed from files (RFC 0031); live PostgreSQL / SQL Server database connectors and a Jira connector remain planned.

Installation

With the web console: the setup wizard (Linux, Windows 11, macOS — RFC 0185)

The full package holds the ekos command, the web console and the Python that runs it; nothing
else is needed. Installing it opens a setup page in your browser:

curl -fsSL https://raw.githubusercontent.com/alexeyban/EKOS/main/install.sh | sh        # Linux, macOS

powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/alexeyban/EKOS/main/install.ps1 | iex"   # Windows

Or download ekos-<version>-<target>-full from the releases page, unpack it and start
setup.sh (Linux), setup.command (macOS: right-click → Open the first time) or setup.cmd
(Windows: More info → Run anyway the first time; the program is not code-signed).

The wizard asks where EKOS goes, whether the console is for you alone (local mode: this machine
only, signed in with a one-time code) or for a team (OIDC), which port, a first project and its
source directories, which agents to register it with (Claude Code, Claude Desktop, VS Code —
their config files are backed up and other entries kept), and whether to start the console at
login. It shows the exact list of steps before doing anything. Afterwards:

ekos console open | status | stop | start | restart | logs
ekos setup                  # change any of it later
ekos uninstall              # the program, login entry, PATH line, agent entries; keeps projects
ekos uninstall --ui         # the same on a page in the browser: what goes, your projects, confirm

The console has the same: Settings → Uninstall EKOS (local mode, an install made by setup)
opens that page, which keeps working while it removes the console itself (RFC 0188). A project's
top bar has Init & compile: init (in-place workspaces), build, recover, resolve (conflicts
reported, not stopping), compile, commit and coverage as one run (RFC 0189).

ekos setup --answers answers.toml applies the same answers without a browser (servers,
scripts); --dry-run lists the steps. The release workflow builds the full package for Linux
x86_64 and aarch64, macOS on Apple silicon and Windows (Intel Macs: the command only); scripts/package-full.sh assembles one, and
scripts/package-release.sh --full builds an offline set for Linux x86_64. Releases from v1.1.0 on carry the full packages.

Prebuilt binary, command only

curl -fsSL https://raw.githubusercontent.com/alexeyban/EKOS/main/install.sh | sh

Installs a single ekos binary into the EKOS home, ~/.ekos/bin (RFC 0179), and creates
~/.ekos/data, where projects keep their config and compiled data. EKOS_HOME moves the home;
EKOS_INSTALL_DIR puts the binary somewhere else. An earlier install in ~/.local/bin is
upgraded where it is. No Rust toolchain, no compiler, nothing else to install. The script resolves the latest release, verifies
the download against the release's SHA256SUMS before unpacking it, and never uses sudo.

Piping a script into a shell is a real supply-chain decision, so the script is short enough to
read first:

curl -fsSL https://raw.githubusercontent.com/alexeyban/EKOS/main/install.sh -o install.sh
less install.sh && sh install.sh

The script also takes EKOS_VERSION to pin a release (EKOS_VERSION=v1.1.0 sh install.sh) and
EKOS_INSTALL_DIR to change where it lands. To run EKOS as a shared service with the web
console, use the container image instead.

Prebuilt binary, downloaded by hand

If you would rather not pipe a script into a shell, every release is a plain archive on the
releases page. Pick the one for your machine:

Your machine	Asset
Linux, Intel/AMD 64-bit	ekos-<version>-x86_64-unknown-linux-gnu.tar.gz
Linux, Intel/AMD, old or unusual distro	ekos-<version>-x86_64-unknown-linux-musl.tar.gz
Linux, ARM 64-bit (Raspberry Pi 4+, Graviton, Ampere)	ekos-<version>-aarch64-unknown-linux-gnu.tar.gz
macOS, Apple Silicon (M1 and later)	ekos-<version>-aarch64-apple-darwin.tar.gz
macOS, Intel	ekos-<version>-x86_64-apple-darwin.tar.gz
Windows, 64-bit	ekos-<version>-x86_64-pc-windows-msvc.zip

uname -sm tells you which you are on if you are not sure: Linux x86_64, Darwin arm64, and
so on.

glibc or musl? Take the gnu build first — it is the faster of the two. If it refuses to
start with something like version 'GLIBC_2.x' not found, your distribution is older than the
one the release was built on: take the musl build instead, which is fully static and depends on
nothing. (install.sh does this fallback automatically.)

Linux and macOS

VERSION=v1.1.0
TARGET=x86_64-unknown-linux-gnu          # from the table above
ASSET="ekos-${VERSION#v}-$TARGET.tar.gz"
BASE="https://github.com/alexeyban/EKOS/releases/download/$VERSION"

curl -fsSLO "$BASE/$ASSET"
curl -fsSLO "$BASE/SHA256SUMS"

# Verify before unpacking. Feeding one line to -c keeps it quiet about the five assets
# you did not download, and works with both GNU sha256sum and macOS's shasum.
grep "$ASSET" SHA256SUMS | sha256sum -c -        # macOS: shasum -a 256 -c -

tar xzf "$ASSET"
mkdir -p ~/.local/bin                    # install(1) will not create it for you
install -m 755 "ekos-${VERSION#v}-$TARGET/ekos" ~/.local/bin/ekos

That prints <asset>: OK and exits 0. Anything else — FAILED, or nothing matched — means do
not unpack it; open an issue instead.

Worth being precise about what this proves: releases are checksummed but not signed — there
is no Sigstore/cosign signature or SLSA provenance yet. SHA256SUMS tells you the archive
arrived intact and matches what the release publishes; it does not, on its own, prove who built
it.

~/.local/bin needs to be on your PATH; add export PATH="$HOME/.local/bin:$PATH" to your
shell profile if ekos --version comes back "command not found". Any directory on PATH works —
/usr/local/bin needs sudo, which is the only reason it is not the default here.

macOS, one extra step. These binaries are not code-signed or notarised. A file downloaded
with a browser is quarantined by Gatekeeper and refuses to run ("cannot be opened because the
developer cannot be verified"). Clear the flag on the extracted binary:

xattr -d com.apple.quarantine ~/.local/bin/ekos

Downloading with curl — as above, and as install.sh does — does not set that flag, so this
step only applies if you clicked the link in a browser.

Windows

$Version = "v1.1.0"
$Asset   = "ekos-$($Version -replace '^v','')-x86_64-pc-windows-msvc.zip"
$Base    = "https://github.com/alexeyban/EKOS/releases/download/$Version"

Invoke-WebRequest "$Base/$Asset" -OutFile $Asset
Invoke-WebRequest "$Base/SHA256SUMS" -OutFile SHA256SUMS

# Verify before unpacking.
$actual = (Get-FileHash $Asset -Algorithm SHA256).Hash.ToLower()
$line   = Select-String -Path SHA256SUMS -SimpleMatch $Asset
if (-not $line) { throw "$Asset is not listed in SHA256SUMS - do not use this file" }
$expected = $line.Line.Split()[0]
if ($actual -ne $expected) { throw "checksum mismatch - do not use this file" }

Expand-Archive $Asset -DestinationPath .

ekos.exe is inside the extracted folder. Move it somewhere on your PATH, or add that folder
to PATH. SmartScreen may warn the first time you run it, for the same reason as macOS: the
binary is unsigned.

(The Linux/macOS block above was run verbatim against the real v1.0.3 release. This PowerShell
one was not — no Windows machine was available — so treat it as carefully written rather than
proven, and please open an issue if it misbehaves.)

Offline, from an install package

For a machine without internet access, or a build of a commit that is not released yet, build a
package on a Linux x86_64 machine with a checkout:

scripts/package-release.sh          # → dist/ekos-<version>-release.tar.gz

It builds ekos from the committed tree (uncommitted changes are left out) for glibc on the
host and fully static for musl in the pinned Rust image (needs Docker; --no-musl skips it),
packs both the way a release does with one SHA256SUMS, install.sh and an INSTALL.md, and
checks that the package installs. <version> is the tag on a tagged commit, otherwise
<crate version>-dev.<commit>. Copy that one file to the target machine, then:

tar xzf ekos-<version>-release.tar.gz && cd ekos-<version>-release
EKOS_VERSION=v<version> EKOS_BASE_URL="file://$PWD" sh install.sh

The installer verifies the archive and installs into ~/.ekos/bin exactly as it does from a
release. INSTALL.md in the package covers installing by hand, the first run, agents, upgrading
and removing.

First run

However you installed it:

ekos --version
cd /path/to/your/repo
ekos init --detect     # writes an ekos.toml that matches what's actually in this repository
ekos doctor
ekos build && ekos recover && ekos resolve && ekos compile && ekos commit
ekos coverage          # confirms every input kind actually produced objects
ekos ask "what does this system do?"

ekos init --detect is worth using rather than plain ekos init: it detects the SQL dialect your
schema is written in, excludes third-party and generated directories that would otherwise be
compiled as if they were your own code, and prints an inventory of what it found. See
RFC 0152.

Projects kept outside the sources (RFC 0178)

By default ekos.toml and the .ekos/ state directory sit in the repository. A project in the
EKOS home keeps both elsewhere, so the sources are only read and one machine can hold many
projects, each with its own ledger:

ekos project add shop --source /path/to/your/repo --detect
ekos --project shop build && ekos --project shop recover && ekos --project shop resolve \
  && ekos --project shop compile && ekos --project shop commit
ekos --project shop ask "what does this system do?"
ekos project list
ekos mcp serve --project shop

~/.ekos/                          $EKOS_HOME, or --home
  data/<tenant>/<project>/
      ekos.toml                   [workspace] root = "/path/to/your/repo"
      state/                      what .ekos/ is for an in-place workspace

• --project works from any directory; EKOS_PROJECT and EKOS_TENANT do the same.
• The tenant is default unless --tenant names one. Two tenants may each have a project
  called shop; they share nothing.
• Nothing is written into the sources. Generated docs and dbt models default to the project's
  out/ directory.
• ekos project remove shop moves the project's data to ~/.ekos/trash/; --purge --yes
  deletes it.
• ekos init and --config are refused together with --project.

Several repositories and folders in one project. Name each input and they compile into one
ledger, so knowledge links across them and every object says where it is from:

ekos project add shop \
  --source api=/repos/payments-api --source web=/repos/storefront --source handbook=/srv/handbook
ekos --project shop source list
ekos --project shop build --source handbook      # re-observe one source only
ekos --project shop ekl "FIND Object WHERE source = 'handbook'"

• A file of the same name in two sources is two objects. Each object carries source (and
  sources when it was found in more than one).
• ekos source add <name> --path <dir> and ekos source remove <name> edit the project's
  ekos.toml and keep its comments. Removing a source does not remove compiled objects.
• A project already compiled without source names cannot gain them; create a new project.
• Sources are local directories and git repositories. Confluence and GitHub are still configured
  once per project.

Users, groups and roles are in the web console
(below). See
RFC 0178.

Is the ledger still current? (RFC 0171)

The ledger describes the source as it was at the last ekos commit. ekos build records every
observed file's size and modification time, and a successful commit marks that list as "what the
ledger reflects". From then on EKOS compares it with the files on disk. It reads metadata only,
never file contents, and a file that was only touched also counts as changed. With [freshness] content-hash = true (off by default) ekos build also records a content hash per file, and a file
whose time moved but whose bytes did not is no longer reported; files excluded as secrets are never
hashed.

ekos status            # … Source : fresh — ledger reflects the source as of 2026-10-09 12:35 UTC (git c4f01c7)
ekos freshness         # what changed since, and which compiled objects cite those files

Changed (1):
  schema.sql
    may be stale: File schema.sql, Table customers, Table orders
Removed (1):
  app.py
    may be stale: File app.py, PythonSymbol total

ekos compile and ekos commit warn FRESH001 when the source moved after ekos build.
ekos doctor includes a "Source freshness" line. Over MCP, ekos_status carries a freshness
block, and when the source has changed every read tool's answer gets one extra note saying the
ledger may be behind, so an agent knows not to trust facts about those files. [freshness] enabled = false turns all of this off; ttl-seconds (default 30) sets how long the MCP server reuses one
check.

Upgrading and removing

Re-running the install command (or installing a newer package) replaces the binary in place with the latest release; there is no
package database and nothing else to update. To remove it, delete the binary
(rm ~/.ekos/bin/ekos, or ~/.local/bin/ekos for an install made before RFC 0179). EKOS keeps
files in two places only: the .ekos/ directory inside each workspace you compiled in place,
and the EKOS home (~/.ekos) for projects. Delete either separately.

Container image (RFC 0179)

Deploying for a team? docs/deployment.md is the step-by-step guide:
sign-in, tenants, roles, agents, model credentials, backups and upgrades.

One image holds the ekos binary, the console API and the built console UI, and serves both on
port 8000. All state is in one volume, the EKOS home. Sources are mounted read-only.

cp deploy/.env.example deploy/.env       # set EKOS_SOURCES and the two tokens
docker compose -f deploy/docker-compose.yml up -d --build
open http://127.0.0.1:8000

• EKOS_SOURCES is the host directory holding your repositories and document folders. It
  appears in the container as /sources.
• In the console, Add a workspace → New project from a source directory with a path such as
  /sources/payments-api. The project's config and compiled data go to the volume
  (/var/lib/ekos/data/default/<project>/); nothing is written into the source directory.
  Then run the pipeline from the project's Run tab.
• A project may only read directories under /sources, both when it is created and when its
  config is edited.
• The CLI is in the container too: docker compose -f deploy/docker-compose.yml exec ekos ekos project list.
  A project created that way appears in the console after a restart.
• The container runs as a non-root user and listens on loopback over plain HTTP. For access from
  other machines put a reverse proxy with TLS in front.
• A model server on the host (Ollama) is reachable from a project's ekos.toml as
  http://host.docker.internal:11434.

Tagged releases publish the image as ghcr.io/alexeyban/ekos:<version>; until the next tag,
build it from a checkout as above. See RFC 0179.

Tenants, users and roles (RFC 0180)

The console decides who may see and do what. The CLI is not affected: someone with a shell on
the machine already has the files.

• Tenants. Each tenant has its own projects and its own members. A project or tenant you
  have no role in answers 404, the same as one that does not exist.

• Roles, granted per tenant or per project in the console's Admin page:

  Role	May
  viewer	read everything in a project, including runs and logs
  reviewer	also decide business semantics and conflicts
  operator	also run commands, manage schedules, edit the project config
  admin	everything in the tenant: projects, members, groups, API tokens, audit log

  A platform owner (named in EKOS_CONSOLE_OWNERS) can do everything, creates tenants, and sets
  which host directories each tenant's projects may read.

• Sign-in is OIDC. A role can be granted to an e-mail address before that person first signs
  in. A group can hold a role for its members, and can name a value of the provider's group
  claim so membership follows the provider.

• API tokens for agents and scripts (Authorization: Bearer ekos_…), created in the Admin
  page and shown once. A token can be viewer or operator. It can never review or administer:
  those need a signed-in person.

• Audit log per tenant: every request that changes something, and every refusal, with who
  and when.

# deploy/.env
EKOS_CONSOLE_OIDC_ISSUER=https://id.example.com/realms/main
EKOS_CONSOLE_OIDC_CLIENT_ID=ekos
EKOS_CONSOLE_OIDC_CLIENT_SECRET=...
[email protected]
EKOS_CONSOLE_SESSION_SECRET=...
EKOS_CONSOLE_PRODUCTION=true

With EKOS_CONSOLE_PRODUCTION=true the console refuses to start without OIDC, an owner and a
session secret, static tokens stop working, and access comes only from granted roles. Without it
the earlier behaviour remains for the default tenant: the two static tokens work (the write
token acts as a platform owner), and under OIDC every signed-in user can read default.

See RFC 0180.

MCP for agents, through the console (RFC 0182)

An agent connects to a project through the console, with an API token:

{"url": "https://ekos.example.com/mcp/default/shop",
 "headers": {"Authorization": "Bearer ekos_…"}}

/mcp/<tenant>/<project> speaks MCP Streamable HTTP, the same as ekos mcp serve --http. The
console sits in front of the project's own server and:

• lists only the tools the token's role allows. The two review tools need a signed-in reviewer,
  so no token ever sees them; ekos_session_note needs operator;
• attaches the token's read scope (RFC 0181) to every call. A scope the client sends itself is
  dropped;
• answers 404 for a project the token has no role in, the same as one that does not exist;
• records every call in the tenant's audit log: who, which tool, allowed or denied. Arguments
  are not recorded.

Projects' servers now start on first use. One unused for 15 minutes is stopped
(EKOS_CONSOLE_MCP_IDLE_SECONDS), and at most 8 run at once (EKOS_CONSOLE_MCP_MAX_SERVERS);
the first call after a stop waits for the server to start. Not built: OAuth for MCP clients and
rate limits. See RFC 0182.

Backup, restore and model credentials (RFC 0183)

ekos home backup                       # every project, and the console's database
ekos --tenant acme home backup         # one tenant's projects
ekos home restore ekos-backup-20261013T101500Z.tar.zst

• A backup is one zstd-compressed tar under <home>/backups/. It holds each project's config
  and compiled data, not the sources (they are not in the home).
• A project is copied under its ledger's write lock: a pipeline stage that is writing makes the
  backup stop with that project's name, and no file is left. Readers are not affected.
• Restore replaces nothing that exists, and changes nothing if anything would be replaced. With
  --force each existing project is first moved to the home's trash. An archive with a path
  that would leave the home, or with a link in it, is refused.

In the console, a platform owner names for each tenant the environment variables holding its
model keys and switches metered calls on. A tenant's pipeline and MCP server then receive those
variables and no other tenant's, and none of the console's own settings. With the switch off
they receive none; a local model server still works. See
RFC 0183.

Console settings, local mode and the keychain (RFC 0184)

A platform owner changes the console's own settings on its Settings page: sign-in, owners,
MCP limits, ports. They are kept in <EKOS_HOME>/control/console.toml. An environment variable
wins over the file and shows on the page as read-only, so a container configured by
deploy/.env behaves as before. Changes apply at the next start (Restart now when a launcher
or a restart policy starts the console again).

Secrets typed on that page, and model keys for projects, go to the operating system's keychain
(Windows Credential Manager, macOS Keychain, Secret Service on Linux), never to a file or a
response. A key stored under a variable name (ANTHROPIC_API_KEY) reaches the commands of a
tenant granted that name while its metered calls are on, as an environment variable would. A
container has no keychain (EKOS_CONSOLE_KEYCHAIN=off in the image); keep secrets in its
environment.

Local mode (mode = "local") is for one person on their own machine: no identity provider,
loopback only (a request from elsewhere, or naming another host, is refused), and the person who
can write the home's control directory signs in with a one-time code. That person is a platform
owner. The installer of RFC 0185 sets it up; see
RFC 0184.

Every command and every setting from the console (RFC 0186)

A project's Commands tab lists every ekos command, read from the binary itself
(ekos commands --json), with a form built from its arguments and the exact command line shown
before it runs. Who may run what follows each command's class: reading needs a viewer, changing
the project an operator, a review decision a signed-in reviewer (recorded under their name, never
a token), and destructive commands or ones that reach a live system outside EKOS an admin who
types the command to confirm. Servers, the installer and the console's own commands are not run
from the web. The run's output lands in the run log like the pipeline's.

The project's config page has a Settings form for every section of ekos.toml, built from
the schema of the running binary (ekos config schema --json): types, defaults and the
documentation of each setting. Changes are applied to the file's text with its comments kept,
then validated and saved as before.

Chat with a project (RFC 0187)

A project's Chat tab answers questions with a model that uses the project's EKOS tools: each
tool call is shown with its arguments and result, and the answer says which objects it rests
on. The model is set once by an owner (Settings → Chat, or in ekos setup): a local Ollama model,
any OpenAI-compatible server, or Anthropic. Its key is named by a variable whose value comes from
the environment or the OS keychain. A hosted model is paid per call, so it answers only in a
tenant whose metered calls an owner has turned on; a model on this machine always may.

The model is offered only tools that read, called with the person's own rights and read scope,
and every call is in the audit log. It never sees a review tool: review decisions stay with
people. Conversations are private to the person who started them.

Access inside a project (RFC 0181)

A project is one ledger over several sources, so limits inside a project are applied when the
ledger is read. A viewer can be limited to part of one project, or have part of it hidden:

• only a source, or a path pattern in a source (src/**), or one object;
• all except a source, a path pattern (secrets/**), or an object.

In the Admin page, grant viewer on one workspace and choose "only…" or "all except…".

What a limited person gets:

• The graph, search, object pages, neighbourhoods and impact, filtered. A hidden object is not
  found, the same as one that does not exist. A relationship shows only when both its ends do.
  Counts cover only what is visible.
• Nothing else in that project: the dashboard, run history and logs, the configuration,
  semantics and conflicts answer 403. Those show more than the ledger (file names in a log, a
  freshness report), and nothing there could apply the limit.
• Anything the compiler could not attribute to a source is hidden from them: commit authors,
  crates, rollups, business-semantics items, conflicts.

Operators, admins and platform owners cannot be limited: running the pipeline shows every
source. A rule hides objects; it does not rewrite the text of a visible document that mentions
a hidden one.

The same filter is available from the command line, for ekos mcp serve, ekos ekl and
ekos query:

cat > scope.json <<'JSON'
{"default": "allow", "rules": [{"effect": "deny", "source": "payments-api", "path": "secrets/**"}]}
JSON
ekos --project shop --read-scope scope.json mcp serve
ekos --project shop --read-scope scope.json ekl "FIND Object WHERE kind = 'Table'"

Under a restricting scope the MCP server offers the tools that read only through the ledger
(ekos_search, ekos_query, ekos_retrieve, ekos_ekl, ekos_neighborhood, ekos_state,
ekos_dependents, ekos_impact, ekos_diff, ekos_graph_export) and refuses the rest. Any
other command refuses --read-scope. A local ekos run without a scope is unrestricted: that
user already has the files. See RFC 0181.

From source

The Cargo workspace root is ekos/, not the repo root — there is no top-level Cargo.toml,
so cargo commands must be run from inside ekos/ (or with --manifest-path ekos/Cargo.toml).
The toolchain is pinned by rust-toolchain.toml; rustup installs the right version automatically.

Prerequisites on both platforms:

• Rust, stable channel, via rustup — 2024 edition needs rustc 1.85+;
  installing the latest stable is fine.
• A C/C++ toolchain — rusqlite's bundled SQLite and the zstd crate both compile native C
  source at build time, so a working cc is required even though the project itself is pure Rust.
• Git, to clone the repo.

macOS

# 1. C toolchain (skip if already installed)
xcode-select --install

# 2. Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

# 3. Clone and build (note: cd into ekos/, the actual workspace root)
git clone https://github.com/alexeyban/EKOS.git
cd EKOS/ekos
cargo build --release --workspace

# 4. Binary is at target/release/ekos — run it directly, or install onto PATH:
cargo install --path crates/cli

(Homebrew's rustup-init — brew install rustup-init && rustup-init — works the same way if you
prefer Homebrew-managed installs.)

Windows 11

Two supported paths — WSL2 is the path of least friction for a Unix-first Rust CLI project, since
it gives you a real Linux toolchain; native Windows works too and is fully supported by Rust.

Option A — WSL2 (recommended):

wsl --install                       # if WSL2 isn't already set up; reboot if prompted

Then open the Ubuntu shell it installs and follow the macOS/Linux steps above (xcode-select
isn't applicable — sudo apt install build-essential gives you the C toolchain instead — then the
same rustup.rs install and cargo build commands).

Option B — Native Windows:

# 1. C++ build tools (provides the MSVC linker rustc's default toolchain needs)
winget install Microsoft.VisualStudio.2022.BuildTools --override "--add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"

# 2. Rust (accept the default x86_64-pc-windows-msvc toolchain when prompted)
winget install Rustlang.Rustup

# 3. Git, if not already installed
winget install Git.Git

# 4. Clone and build (open a new terminal first, so the updated PATH takes effect)
git clone https://github.com/alexeyban/EKOS.git
cd EKOS\ekos
cargo build --release --workspace

# 5. Binary is at target\release\ekos.exe — run it directly, or install onto PATH:
cargo install --path crates\cli

Verify the install

ekos --help                # or: cargo run -p ekos -- --help, from ekos/
ekos init                  # creates .ekos/ in the current directory
cargo test --workspace     # optional: run the test suite (1,500+ tests)

See CLAUDE.md for the full command reference and the mandatory development
workflow if you're planning to contribute.

Legacy transformation recovery (RFC 0027/0028/0029)

A Pentaho step, a SQL SELECT, a VIEW, and a stored procedure are all the same underlying
concept — a transformation of data from sources to a sink through filter/join/aggregate/calculate
operations. ekos recover compiles all of them into one shared Transformation IR
(Source/Filter/Join/Aggregate/Calculate/Sink/Unmapped), so legacy ETL logic recovered
from a Pentaho .ktr/.kjb job can be diffed against a newly drafted SQL pipeline — no manual XML
reading required. Unmapped is deliberate, not a gap swept under the rug: anything that can't be
parsed is still recorded as evidenced fact ("something is here, not yet understood"), never
silently dropped.

PL/pgSQL stored procedures (RFC 0163)

A stored procedure is also imperative — ordered statements, branches, loops, exception handlers —
which a dataflow IR cannot hold without losing the order. For every .sql file resolved to the
postgres dialect (or that names plpgsql), ekos recover parses each routine with a
deterministic, in-process PL/pgSQL parser (no database, no LLM) and writes:

• one Procedure object per routine — signature, language, and a fidelity label computed from
  the parse: statements (every statement recovered), partial (with the exact counts and the
  lines of what was missed), or signature (a LANGUAGE sql/c routine whose body is not parsed);
• one ProcedureStatement per statement at every depth, whose evidence is its own source text and
  line, linked by Contains edges that record which branch each child sits in (then:0, else,
  handler:1, …).

ekos recover   # … PL/pgSQL routines: 556 (254 plpgsql: 251 complete, 3 partial), 1556 statements
ekos commit    # … Procedure links: 1852 new (409 reads, 308 writes, 989 depends-on, 146 calls …)
ekos ekl "FIND Object WHERE kind = 'Procedure' AND name = 'payment_post'"

On LedgerSMB every routine its LOADORDER installs recovers completely (212/212); the three
partial routines above are in a module LedgerSMB itself does not load, and are invalid PL/pgSQL.

Each statement's embedded SQL (and each condition, assignment and loop bound) is parsed as real
PostgreSQL, so every ProcedureStatement and Procedure records the tables it reads, the tables
it writes and the functions it calls — LANGUAGE sql routines included. ekos commit then
links them across the whole workspace: ReadsFrom/WritesTo per statement, DependsOn from the
routine to each table (with access: read | write | read_write), and Calls between routines — so
"what breaks if I change acc_trans" now names the 50 LedgerSMB routines that touch it. A name links
only when it names exactly one object; an overloaded or redefined routine or view, a temp table or
a system catalog links nothing rather than something guessed.

Triggers (RFC 0163)

Every CREATE [CONSTRAINT] TRIGGER becomes a Trigger object — timing, events (with UPDATE OF
columns), level, WHEN condition and function — linked to the table it fires on (DependsOn, so
impact analysis on a table finds its triggers) and to its function. Each is classified from what
its function actually does, read from the recovered IR, never from names: Validation (raises or
drops the row), DerivedColumn (sets NEW columns), Audit (only inserts elsewhere), Cascade
(updates/deletes elsewhere), Mixed (several of those, dynamic SQL, or calls whose effects are not
inlined — a human decides) or Unknown (the function body is not recovered). Every class carries its
reasons. A function redefined across migrations is classified per definition and only gets a class
when they all agree; RETURN NEW placeholders are set aside, and the reasons say so. Triggers are
never auto-translated.

ekos recover   # … Triggers: 42 (42 parsed, 0 read from tokens) — classified at compile

Views (RFC 0169)

Every CREATE [OR REPLACE] [MATERIALIZED] VIEW in an observed .sql file is a View object — its
declared columns, whether it is materialized, its exact source text and line, and what its query
reads and calls, from a real parse. Each definition is parsed on its own, so a view using syntax the
parser lacks is still recorded (footprint: unparsed, with the reason) and never takes its file down.
Views are linked like routines: a view DependsOn the tables and views its query reads, routines and
transformations that read a view link to it, and impact analysis on a table now reaches the views
built on it.

Routines and views are documented like code: docs generate --layout curated writes a page for
each (the author's COMMENT ON FUNCTION|VIEW text as its description, its lines, what it reads,
writes and calls, and who uses it), lists views among the Data Stores with the routines that use
them, and lists routines in API.md under their file. [llm-description] describes them from their
real source like any other symbol, and a backticked routine or view name in a doc links to it.

ekos recover   # … Views: 17 (17 parsed, 0 unparsed — still recorded)
ekos ekl "FIND Object WHERE kind = 'View'"

Business semantics → LinkML (RFC 0170, experimental, opt-in)

Business meaning that nobody wrote down survives as traces: WHERE NOT p.obsolete,
entity_class = 2, the lookup row (2, 'Customer'), COMMENT ON COLUMN account.category IS 'A=asset,L=liability,…', the commit that last changed the line. Recovery records every
column-vs-literal predicate in views, routines and CHECK constraints — normalized, so
status IN (1,3) and status = 1 OR 3 = status are one predicate — plus CHECK constraints,
NOT NULL/key columns and lookup-table seed rows — plus PL/pgSQL IF NEW.col … conditions
(resolved to the trigger's table), Pentaho FilterRows conditions, standalone analyst SELECTs,
and dbt schema.yml (descriptions, not_null/unique/accepted_values/relationships tests)
and model SQL (Jinja rendered: ref/source/var; model columns traced back to their source
columns) — plus SQL in Perl strings, application constants (Perl use constant EC_CUSTOMER => 2,
Python Enum classes; matched to columns by label agreement or name) and marked glossaries (local
docs, Confluence; unmatched terms become gaps). Filters through CTEs and derived tables are
restated on the base table. [semantics] ontology = "<file>" adds exact/close
mapping suggestions from your own vocabulary, exported as annotations only. With [semantics] enabled = true, ekos commit
synthesizes hypotheses from them, deterministically and without an LLM:

Kind	What
BusinessConcept	A filter that defines a view, or recurs in ≥ min-sites routines/views (PartsNotObsolete, EntityCreditAccountCustomer)
EnumMeaning	A coded value and its meaning, from a column comment legend, a lookup seed row via its foreign key, or a CASE label
ConstraintCandidate	A CHECK constraint, typed range/enum/pattern/not_null/other
SemanticGap	A coded value used in logic that no source explains; a concept nobody documented
(opt-in) AI summary	[semantics] llm-definitions = true: ≤2 sentences per undocumented concept, every sentence citing the concept's evidence (or the known code meanings) — uncited or hedging sentences are dropped; a reading aid, never the definition
RationaleLink	git blame -w on the evidence lines → the commit that last changed them

Every item starts as status: hypothesis with its evidence (path:line). Only a human promotes
one — ekos semantics confirm|reject|edit is CLI-only and no MCP tool can reach it. A review holds
while what the item asserts and its evidence are unchanged (line moves don't count); otherwise the
item becomes needs_review, and a confirmed item whose traces disappear is flagged, not dropped.
Same-column threshold disagreements (> 90 here, > 60 there) become ConceptConflicts.
The web console's Semantics tab is the review UI: a queue with evidence and bulk
confirm/reject, and a LinkML viewer and YAML editor (with a side-by-side diff against the current
export) whose edits round-trip through ekos import linkml (see Web console below).

Agents read them over MCP — ekos_semantics_lookup (what a term, table, column or code means)
and ekos_semantics_gaps (what is not known), listed when [semantics] is on. Every result
states its status in words ("CONFIRMED by a human reviewer…", "HYPOTHESIS — … say so if you use
it"); rejected definitions are left out; an empty lookup says no_semantics_found so an agent
does not improvise. No MCP tool can confirm anything. The export is a draft LinkML schema — EKOS feeds LinkML, LinkML
generates everything else (JSON Schema, Pydantic, RDF/OWL/SHACL):

ekos semantics list [--kind concept|enum|constraint|gap|rationale]
ekos semantics show PartsNotObsolete        # definition, evidence lines, links, commits
ekos semantics gaps                         # open questions, conflicts, and the needs_review queue
ekos semantics confirm TransactionsNotApproved --note "drafts"     # human-only (--as, default $USER)
ekos semantics edit PartsWithInventoryAccnoId --name "Inventory part" --description "…"
ekos semantics reject UserPreferenceWithoutUserId --note "anti-join plumbing"
ekos export linkml --out model.yaml          # confirmed only; --status hypothesis|all for the draft
ekos import linkml model.yaml --dry-run      # an expert's YAML edits → the review decisions they mean
ekos import linkml model.yaml --as ann       # …recorded, all or nothing (renames, descriptions,
                                             # code labels, ekos_status: confirmed|rejected)
ekos semantics confirm PartsAssembly PartsNotObsolete --as ann   # many at once, all or nothing
ekos semantics gold-template --out gold.yaml   # blank, structure-only: for an expert, before any output
ekos semantics eval --gold gold.yaml        # concept recall, label accuracy, gap recall, evidence validity

A runnable end-to-end example — clone LedgerSMB, compile, review, export, then lint, generate
and validate with LinkML's own tools — is demo/ledgersmb-linkml/.

On LedgerSMB (SQL + Perl + UI + git, no LLM): 649 predicate sites (458 resolved to a
table column), 47 concepts, 222 coded values (179 with a meaning), 0 identity conflicts in
resolve; an unchanged re-commit writes nothing. The LinkML file passes linkml-lint with 0 errors and
gen-json-schema/gen-pydantic run cleanly. Against a starter gold set (not expert-written —
see ekos/docs/rfcs/0170-ledgersmb-starter-gold.yaml): concept recall 0.68, enum label accuracy
0.95, evidence validity 1.0. Meaning defined only by comparisons between columns or against the
current date (overdue, unpaid) leaves no literal trace and is not recovered: no trace, no recovery.

dbt metadata extraction (RFC 0117)

dbt can point at any warehouse, so ekos recover extracts real Table objects from a dbt
project's own checked-in metadata rather than a live database connection — never manifest.json/
catalog.json either, since both are dbt/target/ build artifacts, gitignored in every real
project inspected while designing this. One Table per models/**/*.sql file (a model exists the
moment its .sql file does, regardless of whether any YAML documents it) and one per declared
sources[].tables[] entry (no .sql file backs a source — it's a pre-existing table dbt only
references). ref()/source() macro calls in each model's raw SQL become real DependsOn
relationships, resolved against the models/sources found in that same dbt project; an unresolvable
reference (e.g. into an installed, gitignored dbt_packages/) is honestly skipped, never
fabricated. Declared schema.yml columns are merged in as-is — explicitly partial, since dbt
projects typically only document tested/described columns, not every column a model produces.

The same real-world entity observed under different names across systems (Informix cust_mstr,
Postgres customers, Databricks gold.dim_customer) can be linked too: ekos identity scan
scores candidate cross-system matches (column overlap, naming-pattern similarity, type
compatibility) and writes them as unconfirmed relationships — never a silent auto-merge — for
review via the ekos_identity_review MCP tool.

Same-source duplicates (ekos resolve/ekos compile, e.g. two Table objects both literally named customers) auto-merge only when the
match is an exact normalized name; anything fuzzy goes through that same unconfirmed/review
flow instead of an irreversible merge (RFC 0063) — no confidence threshold on the underlying
scoring formula reliably separates real correct fuzzy merges from real incorrect ones, so an
irreversible auto-merge isn't a safe default for that case.

A same-name group spread over separate namespaces is not a conflict: a Perl package
LedgerSMB::GL beside a table gl, or a JS and a Perl initialize. resolve accepts a group
spanning the database (Table/View/Procedure/Trigger) and the code of Perl, JS, Python,
Rust or Elixir when each namespace's share is unremarkable on its own; a Table beside a
same-named View still conflicts (RFC 0147 amendment b — LedgerSMB 16 → 0 conflicts).

When sources disagree, both claims are kept (RFC 0172). A table created in two files with
different columns, a column type an exact-name merge would have dropped, two labels for one code
from different sources — each becomes a ConflictingEvidence item instead of the last one silently
winning: every claim with its file and line, what EKOS kept, and a Disputes link to the object.
Values are normalized first (INT/INTEGER/SERIAL are one type), a fact only one side states is
unknown rather than a disagreement, and prose is never compared.

ekos conflicts list
#   [open] account.category = 'A'.label  — 2 source(s) disagree: "asset"  vs  "L"
#   [open] user_preference.columns  — 2 source(s) disagree: ["dateformat","id",…]  vs  ["id","name","user_id","value"]
ekos conflicts show "account.category = 'A'.label"   # each claim's path:line and source
ekos conflicts resolve user_preference.columns --both-valid --note "1.9 replaces the table"  # human-only
ekos conflicts resolve "account.category = 'A'.label" --pick 1 --as ann

A decision holds while the claims are unchanged; when one changes, the conflict reopens with the old
decision kept. Agents read conflicts over MCP (ekos_conflicts, and open_conflicts on
ekos_state) and are told to present an open one as a disagreement, never one side as fact; no MCP
tool can resolve one. Answers say so too (RFC 0173): an ekos_query / ekos_retrieve claim about an
attribute with an open conflict carries disputed: true and the conflict ids, since its value is
only the side EKOS kept, and ekos ask tells the model the same in the evidence it reads. A label
conflict is also linked to the table its column belongs to, so a question about the table surfaces
it. The web console's Conflicts tab lists every conflict with each claim's file and line, lets a
write-role user pick the right claim or mark both valid, and the graph rings objects with an open
conflict (RFC 0174). On LedgerSMB: 3 conflicts — user_preference (re-created by a 1.9 migration)
and 2 order-class labels where an API query names the counterparty (customer) rather than the
order type. The detector first surfaced a CASE that swaps 'A'↔'L' misread as a label; the
extractor no longer reads a rewrite as a meaning. [conflicts] enabled = false restores the old behaviour.

Real schema and class structure from source (RFC 0091/0092)

File-based SQL schema needs a dialect. .sql files parse under the generic (ANSI) dialect
unless ekos.toml routes them somewhere real:

[recover.sql]
default-dialect = "generic"

[[recover.sql.dialect-rules]]           # first path-glob match wins
path-glob = "priv/repo/**"
dialect = "postgres"
[[recover.sql.dialect-rules]]
path-glob = "priv/ingest_repo/**"
dialect = "clickhouse"

This matters for any real schema dump — pg_dump output, a ClickHouse structure.sql, or
hand-written DDL past plain ANSI. The postgres / clickhouse dialects carry the preprocessing
(RFC 0057/0058/0059) that makes real dumps parse; generic does not.

A statement the dialect cannot parse costs that statement, not the file (RFC 0176). The rest of
the file is still recovered, and ekos recover says what was skipped:

• SQL005 — <file>: 2 of 61 statements were not parsed and skipped, with the line, the first
  words and the parser's error for each. A skipped CREATE TABLE is called out as a lost table.
• SQL001 — when a file states a CREATE TABLE and no table was recovered, the message names the
  dialect it was parsed under and the first error, and points at [[recover.sql.dialect-rules]].
  A file with no CREATE TABLE keeps the plain no tables found.

If ekos ekl "FIND Object WHERE kind = 'Table' COUNT" returns 0 for a repo that clearly has a
schema, read those two warnings first. Foreign keys declared after the tables, as pg_dump writes
them (ALTER TABLE … ADD CONSTRAINT … FOREIGN KEY), become ForeignKey edges too.

Hand-written schemas need more than the right dialect (RFC 0146). Everything above was tuned
against pg_dump, which emits a narrow, mechanical subset of PostgreSQL. A human-maintained schema
does not: measured on LedgerSMB, a correctly-configured dialect = "postgres" still recovered
0 of 158 tables, because the file's first COMMENT ON TABLE … IS $$…$$ failed the whole-file
parse and took every table with it. The postgres dialect now preprocesses COMMENT ON,
INHERITS, SECURITY DEFINER, RETURNS SETOF, := named arguments, DO blocks, CREATE RULE,
psql meta-commands and COPY … FROM stdin payloads out of the way; the same file yields 158 tables
and 214 foreign keys.

COMMENT ON TABLE / COMMENT ON COLUMN text is no longer discarded either — it becomes an
evidence-backed description on the Table object (and on the matching column), with the source
line recorded. Author-written text outranks the LLM's: description is the schema's own words,
the model's version is kept alongside as llm_description, and sql_comment marks the provenance.

[llm] max-tokens caps compiler-pass LLM calls — distinct from [ai] max-tokens, which governs the
read-side ekos ask runtime. Leave it unset and each file's budget is computed from its own schema
size; a new SQL004 diagnostic reports when enrichment named fewer tables than the file declares,
and says so explicitly when the response hit its ceiling.

Beyond raw SQL DDL, ekos recover's Python analyzer recognizes a real SQLAlchemy declarative
model (__tablename__ present on a class) and compiles it into the same Table object shape as a
CREATE TABLE statement — real column names, best-effort data-type hints, and ForeignKey edges
resolved against other models in the same file — so a project whose entire schema is ORM-declared
(the majority shape for a modern Python backend) still gets a real ## Data Architecture and
entity-relationship diagram instead of an empty section. The same analyzer also compiles real
class inheritance (class Document(Base):) into RelationshipKind::Extends edges between the
real PythonSymbol objects involved — visible on each class's own generated page and in its
Mermaid diagram — resolved only when the base class is defined in the same file, never fabricated
against an imported base EKOS can't see the definition of.

Document semantic memory (RFC 0025/0026)

Beyond structural parsing, an opt-in pass reads local documents through an LLM to extract real
entities (Concept objects) and the relationships between them — so the same concept mentioned
across different documents becomes one findable, linkable thing instead of isolated text hits.
Enable it in ekos.toml:

[llm]
provider = "ollama"   # or omit for Anthropic via ANTHROPIC_API_KEY; any LlmProvider works

[document-semantics]
enabled = true

Then ekos recover runs DocumentSemanticsAnalyzerPass alongside the structural document pass,
and the extracted Concept objects are queryable through the same MCP tools as everything else
— ekos_search, ekos_neighborhood, ekos_dependents, ekos ask. No new tool, no new query
surface — this is exactly the point: AI tools get real memory through the Runtime they already
talk to. ekos ask honors [llm] provider = "ollama" the same way ekos recover does — both
commands select the LLM provider through one shared function, so a workspace configured for local
Ollama works identically for recovery and for querying.

Markdown is split by heading (RFC 0144). Each heading becomes its own searchable Section named by
its heading path (docs/rfcs/0013-mcp-server.md § RFC 0013 — MCP Server › Motivation) with a real
line range, doc_type (rfc/devlog/readme/claude_md/doc) and, for RFCs, rfc_number/rfc_status.
compile also links docs deterministically: an RFC NNNN mention becomes a References edge to that RFC,
and a backticked identifier that names exactly one code symbol becomes an edge to it. ekos ask "what does RFC 0013 …" resolves the RFC by number. These edges show up in neighbourhoods but never count as
dependents in ekos_dependents/ekos_impact.

Using a hosted OpenAI-compatible model (RFC 0145). Any Chat Completions host works — OpenCode Zen,
OpenRouter, DeepSeek, Groq, vLLM:

[llm]
provider = "openai"
base-url = "https://opencode.ai/zen/v1"
model = "deepseek-v4-flash"
api-key-env = "OPENCODE_API_KEY"

[ai]
max-tokens = 8192            # reasoning models spend hidden tokens; 1024/2048 truncate or empty answers

A model per task (RFC 0197). provider = "zen" reaches every OpenCode Zen model on its own wire
format (Claude and Qwen Flash on Messages, GPT/Grok on Responses, the rest on Chat Completions), and
[llm.tasks.<task>] routes one kind of call to its own model — tasks: ask, doc-semantics,
description, semantics, sql-naming, session-capture, architecture, reconstruct,
nl-sql, docs-prose, marketing. Measured on this repository (RFC 0197's tables):

[llm]
provider = "zen"
model = "claude-haiku-5-5"   # ask: 98/101 for $0.10 a full eval run
api-key-env = "OPENCODE_API_KEY"
budget-usd = 1.00            # refuse a call whose worst case could cross this, per command

[llm.tasks.doc-semantics]
model = "gpt-6-luna"         # 40/40 sections parsed; deepseek-v4.1-flash parsed 4/33 at the old cap
max-tokens = 4096
fallback = ["claude-haiku-5-5"]

Every call that reaches a host is priced and logged (metadata only) to .ekos/llm-usage.jsonl:
ekos llm usage totals it, ekos llm routes shows each task's model and price, ekos llm probe
sends a few-token request per model to show which ones a key can reach (access is per account on
Zen). Models whose provider may keep requests are refused unless [llm] allow-data-collection = true;
Zen's free models are not reachable outside OpenCode.

For local Ollama, [llm] context-window = 8192 (the default; also OLLAMA_NUM_CTX) is sent as num_ctx —
without it Ollama silently truncated long prompts — and EKOS warns when a prompt fills the window.
ekos doctor shows the effective window or custom endpoint. Hosted models bill per token; a full
101-scenario ekos eval run on DeepSeek V4 Flash costs about $0.08. ekos eval run --evidence-only
calls no model: it reports Recall@10 and whether each scenario's expected facts reach the evidence. DeepSeek V4 Flash is a reasoning model whose
hidden reasoning counts against max-tokens — use [ai] max-tokens = 8192 or some answers come back empty.

Documentation generation (RFC 0035/0037/0042/0090/0094/0095)

ekos docs generate renders the compiled ledger straight into Markdown/HTML documentation —
zero LLM calls, every claim traceable to real compiled evidence. Three layouts:

ekos docs generate                              # --layout objects (default): one page per
                                                  # significant object, plus an index and ER diagram
ekos docs generate --layout curated --output doc # README.md/Architecture.md/API.md/
                                                  # SequenceDiagrams.md — the shape a developer
                                                  # actually expects, plus one detail page per
                                                  # crate/technology/pipeline/program-entity object
ekos docs generate --layout solution-architect --output doc-sa
                                                  # DependencyRiskReport.md/OnboardingGuide.md/
                                                  # FindingsMemo.md — a team-handoff bundle: real
                                                  # declared dependency versions and concentration
                                                  # risk, a first-day repository-layout guide, and
                                                  # an actionable findings list (unresolved
                                                  # dependencies, undeclared crate versions, missing
                                                  # doc-comment coverage). `--prose` layers an
                                                  # LLM-written executive summary on the findings
                                                  # list, never replacing the deterministic list
                                                  # underneath it.

--layout curated's Architecture.md includes a real crate/workspace dependency graph (parsed
Cargo.toml, not guessed, annotated with a C4 mapping — crate → Container, external dependency →
External System, RFC 0065), external technology dependencies, an ## Open Questions section
listing real knowledge gaps a deterministic pass couldn't resolve (e.g. an unresolvable workspace
dependency) rather than dropping them silently, CI/CD pipelines (parsed .github/workflows/*.yml),
and an entity-relationship diagram; API.md lists real functions/
structs/enums/traits (from RustSymbol/PythonSymbol objects, RFC 0038/0040/0041) grouped by
file, each linked to its own detail page; SequenceDiagrams.md covers both Transformation-IR
data-flow sequences and real function-call sequences (RFC 0041's Calls graph). Per-entity pages
nest under entities/<kind>/<2-char shard>/ so a large codebase's page count never blows past
GitHub's per-directory file-listing cap — running ekos docs generate --layout curated --output doc against this repo's own source is a ready example to try. Each entity page's Definition
section now shows the real human-written documentation from source when the analyzer found any
(/// doc comments, Python docstrings, @moduledoc/@doc, JSDoc — RFC 0087), honestly stating
"Not documented in source" rather than fabricating one when it didn't. --prose (opt-in) layers an LLM-written overview onto each
object page, reusing ekos ask's exact grounding+citation pipeline, with a token-cost estimate
shown before any call.

Architecture.md's Executive Summary now surfaces two more real, deterministic signals instead of
placeholder text: Major risks lists real "Observed Concentration Risk" objects — any object
with 3 or more real compiled DependsOn dependents, a structural single-point-of-failure
candidate, never an LLM-guessed severity score (RFC 0094) — and Architecture confidence shows
the same real completeness/evidence-coverage score ekos architecture investigate computes (RFC
0065 Phase 3), now also run from the plain docs generate path instead of only the investigation
loop (RFC 0095). Both say so honestly when there's no real signal to compute from yet, rather than
showing a misleading 100%.

Architecture reasoning + investigation loop (RFC 0065/0066/0067, opt-in)

Beyond deterministic extraction, ekos architecture investigate runs the RFC 0066 MVP agentic
loop: broad collection, deterministic crate-topology extraction, one batched LLM call classifying
each crate's architectural role (ArchitectureReasoningPass, RFC 0065 Phase 2), a deterministic
evaluator scoring completeness and evidence coverage (no LLM — RFC 0065 Phase 3), and — for any
crate the evaluator flags unclassified — a targeted second pass that reads that crate's own leading
doc comment for more context before trying again. Stops early once the quality threshold is met, or
after --max-iterations, always ending with a curated-docs docs generate run:

ekos architecture investigate                                 # RFC 0066 MVP defaults: 3 iterations,
                                                                # 0.90 quality threshold, --output doc
ekos architecture investigate --max-iterations 5 --quality-threshold 0.95 --output doc

Reuses the [llm] provider already configured in ekos.toml (local Ollama or a cloud provider) —
no separate --llm flag. See RFC 0067 for what's deliberately out of scope for this MVP
(persistent checkpointing/resume, concurrency-safety infrastructure, CI/CD exit codes, multi-format
output).

Because the LLM-classified crate role is a real judgment call, not a deterministic fact, two
follow-on commands treat it accordingly rather than silently trusting or silently re-deriving it:

ekos architecture diff --since <timestamp>   # real id-set comparison of technologies, crate role
                                              # classifications, risks, and open questions between
                                              # two points in time — not a fuzzy match (RFC 0108)
ekos architecture review                     # list/confirm/reject pending role classifications;
                                              # a confirmed-or-rejected review status survives the
                                              # next `ekos commit` even though the underlying claim
                                              # is content-signature-versioned and gets re-derived
                                              # on every run (RFC 0109)

ekos_architecture_diff and ekos_architecture_review expose both over MCP too — see the AI agent
access section below.

LLM-backed compile-time descriptions (RFC 0088, opt-in)

Unlike --prose above (render-time, re-spent on every docs generate call), [llm-description]
in ekos.toml runs at commit time and persists real, evidence-grounded ai_overview/ai_usage
properties straight into the ledger — queryable through ekos ekl/ekos ask/MCP the same as any
other compiled knowledge, not just rendered once. Covers every Module/Rollup/Crate, and every
Symbol with a compiled source_span (Rust and Elixir today), regardless of whether RFC 0087
already found a real doc comment — a doc comment is real input to the prompt, not a skip condition.
When one exists, a new ai_comment_check property (consistent/stale/incomplete) flags a real
discrepancy between what the comment claims and what the code actually does, rendered as a visible
callout right on the entity page's Definition section — never silently trusted, never overwritten.
A single project-level call also fills Architecture.md's Purpose/Architecture style fields
when real signal (a README, compiled subsystems, compiled technologies) exists to ground them.

[llm-description]
enabled = true
scope = "modules"   # "modules" (default, cheapest) | "symbols" | "all"

ekos commit          # shows a real call-count estimate, asks to confirm before any spend
ekos commit --yes    # skip the confirmation prompt

Opt-in and cost-gated like [architecture-reasoning] — a real, potentially large spend (~900 real
LLM calls at the default scope = "modules" against a real mid-size codebase, ~5x that at
scope = "all"), never defaulting to the more expensive tier just because it was turned on.

Compiled-knowledge query engine (RFC 0118 — SEARCH → QUERY → REASON)

Traditional RAG searches documents; EKOS queries compiled knowledge. The retrieval stack is
three operations, built as RFCs 0119–0126 (devlog_143–devlog_149):

• SEARCH — one KnowledgeStore::retrieve seam behind every consumer, fusing a BM25 arm, an
  exact-name arm, and (opt-in) a vector arm with Reciprocal Rank Fusion (Cormack RRF, k=60). BM25
  indexes object names and content with English stemming, so a singular mention ("the customer
  table") resolves the same object a plural exact mention ("the Customers table") already did.
  ekos query find "<text>" --mode lexical|vector|hybrid; --explain prints the compiled plan
  and per-arm timings (populated on partitioned and distributed stores too, not just a single
  FactLedger). The same fused path runs on the single FactLedger, the partitioned
  store, and the distributed gateway.
• QUERY — direct, zero-LLM answers over the compiled graph: fact(entity, attr) for one
  attribute of one object, named ops (dependents, callers, path, …) for the graph. Exposed
  as MCP ekos_query (a typed list of source-cited claims) and ekos_retrieve ("show your
  work": the plan + evidence set + how the question was understood).
• REASON — ekos ask "<question>" now compiles the question: a rules planner routes it to
  fact lookups and graph traversals, executes them into a typed EvidenceSet where every item
  carries its provenance, and the model's job shrinks to "explain this evidence, cite each item."
  --explain prints the plan and evidence; --classic selects the pre-0123 retrieve-and-dump
  path. A citation not backed by a source id in the evidence set is a reported finding, not a
  formatting detail. Claims cite a place in a file, not just a file (RFC 0140): every
  Rust/Python/Elixir symbol with a source_span carries real evidence naming its path, its line,
  and the source text itself, rendered as path:start-end — so asking what a function does shows
  the model the function body, not only its name. Each Rust/Python/Elixir function or method also
  carries a real signature (RFC 0141) — the discriminating term for "what builds an
  Arc<dyn LlmProvider>" often lives in a return type, not the function's own name — and a Rust
  Calls edge carries call_count, call_site_line, and caller_is_test, so impact analysis can
  weigh 40 production call sites differently from 40 mostly-test ones instead of returning an
  undifferentiated list. For the top few entities in each answer's evidence set, ekos ask also
  reads the entity's real, on-demand source text straight from the artifact store — never the live
  filesystem, so it never bypasses redaction (RFC 0140 §3) — giving the model the actual code
  behind a claim, not just a name and a short excerpt. When the artifact store holds no file text
  for the entity, the source recorded with the symbol at recover time (up to 40 lines) is shown
  instead (RFC 0177). An opt-in [retrieval] rerank = "llm"
  (off by default) asks the model to reorder the top candidate evidence items by real relevance
  before it answers (RFC 0140 §4) — never on the deterministic retrieval-ranking path RFC 0126's
  CI gate checks, only inside the REASON answer pipeline itself.

EKL gains FIND Object SEMANTIC 'text' [LIMIT k] — the retriever as a candidate-set strategy.
Retrieval quality is CI-gated: ekos_runtime::retrieval_eval holds a checked-in graded query set
and a reference estate, and a workspace test fails the build if Recall@10 / MRR / nDCG@10 or
intent-classification accuracy drops more than 2% below the recorded baseline;
cargo bench --bench retrieval_eval prints the current scoreboard (RFC 0126).

What search reads and how it ranks (RFC 0177):

• An object is indexed by its name, kind, excerpt, and for a symbol its doc comment, signature and
  fields. An index built by an older version is rebuilt on the next command that writes to the
  ledger; ekos ledger repair does it at once.
• A hit that holds every word of the query ranks first. Hits that hold some of the words follow,
  ordered by BM25 rank fused with how many words each holds.
• A file is found by its name without directory or extension (0043-secrets-and-pii-redaction).
• The excerpt shown with a hit is the passage that matched the query.
• A question that asserts something the evidence does not support is answered
  Premise not supported., followed by what the evidence does show. A question whose answer is
  not in the evidence is answered Insufficient evidence., with what the evidence does show.

The vector arm — semantic search (RFC 0125, opt-in)

A question phrased with none of the target object's words ("the thing that sends welcome emails" →
a function called dispatch_signup_notification) has no lexical hook. [embeddings] in
ekos.toml adds a vector retrieval arm: ekos commit embeds every compiled object (its name,
kind, and ai_overview prose if present) into a derived <ledger-dir>/vectors/ index, and
retrieval fuses cosine-nearest hits with the BM25 + exact-name arms (Reciprocal Rank Fusion, the
same fuser as RFC 0120).

[embeddings]
enabled = true
provider = "ollama"          # "ollama" | "openai" | "mock"; falls back to [llm] provider
model = "nomic-embed-text"   # optional override
api-key-env = "OPENAI_API_KEY"
cache = true                 # content-addressed .ekos/embed-cache/, on by default

ekos query find "sends welcome emails" --mode vector   # semantic only
ekos query find "welcome email" --mode hybrid          # semantic + BM25, fused

Opt-in and off by default — with no [embeddings] table nothing is embedded and retrieval is the
pure BM25 + exact-name path. Embeddings are cheap and disk-cached, so unlike [llm-description]
there is no spend prompt. Single-node only this phase (a no-op on a SQLite or partitioned
workspace); a vector/hybrid search with no index built yet degrades to lexical with a visible
note. The MCP ekos_search tool takes the same mode and reports arms_run. Per-arm wall-clock
timings (arm_timings, RFC 0126) ride along on the ekos_search / ekos_retrieve results, into
.ekos/query-log.jsonl, and in ekos query find --explain.

Eval harness (RFC 0138, on-demand)

ekos eval run grades whole ekos ask answers against a checked-in, 101-scenario suite spanning
seven categories — Architecture, Code, Dependencies, Lineage, History, Security, Adversarial
(evals/datasets/*.yaml) — not just retrieval ranking (that's RFC 0126's separate, narrower,
CI-gated ekos_runtime::retrieval_eval). Every score is deterministic keyword/id matching — no
LLM judge: did the answer state the expected facts, cite real (non-hallucinated) evidence, cover
what was asked, surface the right objects in the top-10, route to the right REASON planner query
type, and — for the Adversarial category — actually say "Insufficient evidence" for a question
with no grounded answer instead of inventing one. That last category is the harness's strictest:
every scenario in it expects a refusal, and an answer that invents a plausible-sounding detail
(a database name, a version number, a person) counts as a hallucination regardless of how
confidently it reads.

Recall@10 is graded against the exact keyword-only query reason::plan() actually searches with
(RFC 0139), not the raw natural-language question a reason/ask scenario asks — the two can rank
differently, so grading the wrong one could pass or fail a scenario for a reason unrelated to what
the model was shown. Lexical search itself also treats a bareword and/or as connector noise,
not a literal word a document must contain — the same two words this codebase already treats as
English stopwords everywhere else — so an "A or B" style query can match on either real term
without silently requiring the word "or" itself to appear.

cd ekos
cargo run -p ekos -- eval run --dataset ekos-full   # every category, "evals/" at the repo root
cargo run -p ekos -- eval run --dataset architecture --agent ollama
cargo run -p ekos -- eval run --dataset hallucination   # Category G only
cargo run -p ekos -- eval history                       # every past run, as a trend table

EKOS EVALUATION
─────────────────────────────

Dataset: architecture
Agent: ollama (llama3:latest)
Runtime: local

Scenarios:                    3
Passed:                       0
Failed:                       3

Answer correctness:       11.1%
Evidence groundedness:   100.0%
Completeness:             11.1%
Recall@10:                  n/a
Hallucination rate:        0.0%

Avg tokens:               1,221
P95 latency:              49.4s

Cache hits:                 0/3
Tokens saved:               n/a
Peak RSS:               69.7 MB
CPU time:                 14.7s

Status: FAIL

That example is a real run against this repo's own live self-analysis ledger with the local
llama3:latest model — not a simulated one. It is also the floor: a workstation-sized local model.

Full-suite results — ekos-full, 101 scenarios, graded deterministically. The answering model
went from a local llama3 8B on Ollama to DeepSeek V4 Flash through OpenCode Zen
(an OpenAI-compatible cloud endpoint), and the knowledge EKOS hands it was improved in two further steps. The same
checked-in suite, one change at a time:

Step	Passed	What changed
Baseline, local llama3:latest (8B) — report 20260914T154459Z	42/101	—
+ evidence text kept, citation parsing, refusal wording (devlog_183)	53/101	Grader and evidence-rendering bugs, not the model
+ DeepSeek V4 Flash (model swap only)	70/101	A larger model, and the local one had been reading a truncated prompt — EKOS never sent num_ctx, so Ollama dropped the start of long prompts, where the instructions live (RFC 0145)
+ document structure (RFC 0144)	79/101	Markdown cut by heading instead of blind 2,500-char chunks, fully indexed; docs link to code and to each other; output limit raised to 8,192
+ query-planner fixes (devlog_186)	87/101	A dependency phrase in a descriptive clause no longer triggers a graph walk; crate::path names resolve as one entity; generic words stop matching unrelated objects

Seventeen points came from the model and seventeen from engineering; 53 → 87 is +34. Per category, before → after:

Category	Before	After
Architecture	9/20	18/20
Code	8/15	13/15
Dependencies	10/12	10/12
Lineage	5/12	10/12
History	3/12	11/12
Security	7/12	10/12
Adversarial	11/18	15/18
Total	53/101	87/101

Metric	llama3 8B (53/101)	DeepSeek V4 Flash (87/101)
Answer correctness	50.1%	85.7%
Evidence groundedness	65.9%	95.6%
Hallucinated answers on adversarial questions	7	3
Recall@10	64.7%	55.9% — fell
Latency	up to 79 s p95, killed three times for lack of memory on a 15 GB workstation	median 5 s per fresh call
Cost of a full 101-question run	free (local)	about $0.05–0.08

Recall@10 fell because documents are now many heading-sections that outrank the crate or file a
retrieval-only scenario expects; answers improved anyway, and section-vs-document ranking is a tracked
follow-up. These numbers were re-verified against the per-category reports saved in
evals/reports/zen-*/ (zen-base 70, zen-final-8k 79, zen-planner2 87). The full account, including the model-swap
run's 2,048-token output limit that cut off 17 answers (DeepSeek V4 Flash reasons, and its hidden reasoning
counts against the limit — so the model's own contribution is, if anything, understated), is in
EKOS from 53 to 87 and
devlog_183 / 185 / 186.

What still fails — 14 of 101, named rather than averaged away:

Cause	Scenarios
The model correctly rejects a false premise but not in refusal wording the grader accepts	adv-004, adv-011, adv-015
The answer never reaches the top of the retrieved evidence	code-002, lin-008, dep-004
The fact isn't recorded yet (a workspace-inherited Rust edition, struct fields as claims, no caller reached the evidence)	code-004, lin-007, dep-005
Retrieval-only scenarios graded on recall@10 (see above)	arch-009, hist-007
Right in substance, wrong keyword	arch-020, sec-002, sec-010

Reproduce it, with your own model:

export OPENCODE_API_KEY=...        # or any OpenAI-compatible endpoint: set base-url/model in the config
cargo run -p ekos -- --config ../ekos.zen.toml eval run --dataset ekos-full --save-answers
cargo run -p ekos -- eval history  # trend table across every saved run

# ekos.zen.toml — hosted, or a self-hosted OpenAI-compatible server (vLLM, llama.cpp) inside the network
[llm]
provider    = "openai"
base-url    = "https://opencode.ai/zen/v1"
model       = "deepseek-v4-flash"
api-key-env = "OPENCODE_API_KEY"

[ai]
max-tokens  = 8192   # a reasoning model: hidden reasoning counts against this

# local Ollama: set the window explicitly — a silent default truncation cost 17 points
# [llm]
# provider = "ollama"
# context-window = 32768

Closed environments. Compiling needs no GPU, no network and no model; answer quality tracks the model you
can serve. A workstation-sized local model lands near the llama3 column; reaching the DeepSeek column without a
cloud API means self-hosting a much larger open-weights model on dedicated GPU servers, with a context window of
at least 32k tokens and enough throughput to answer in seconds. Prompts contain real source excerpts — EKOS
redacts secrets and PII before anything reaches the ledger, but a hosted model still sees your code (Zen's paid
models are zero-retention; its free models may use prompts for training).

A local model is real and free to run, but it is not the reference: ekos eval run refuses outright rather than
silently grading against the stub MockLlmProvider when the configured provider's API key isn't set (there is no
--agent mock option). Grading is unchanged across every step above — keyword and id matching at temperature 0,
unchanged prompts replayed from the LLM cache — so a score that moved is attributable to a real change.

Earlier history. With llama3 held fixed the suite went 39 → 43 (RFC 0140/0141), then 42 → 53 against the
20260914T154459Z baseline (devlog_183), mostly because the harness found something unflattering: an adversarial-question fabrication count that got worse (3 → 6) once
third-party noise was gone and real objects made a wrong answer read more convincingly; a bit-identical re-run
after RFC 0139/0140/0141 that confirmed temperature: 0 makes an unchanged prompt reproducible; and
recall@10 grading that had silently graded the wrong query (RFC 0139, devlog_182). Details: devlog_182, devlog_183.

Those numbers moved mostly because the harness found something unflattering: 94% of what this
repo was compiling wasn't its own code. A Python virtualenv and two .scannerwork/ directories
had never been excluded from the observation walk — .gitignore does not filter it. Excluding
them took the corpus from ~13,700 files to 834 and the model from 53,830 objects to 12,283, while
recover produced identical symbol counts (2,623 Rust symbols, 1,734 Calls edges) — all
signal kept, all noise gone. See devlog_174 and the
eval comparison report. Beyond the five headline scores, every run also captures real resource usage: tokens
saved is genuine cache-hit attribution (CachedLlmProvider tracks hits/misses; a scenario whose
answer came from the disk cache instead of a fresh network call is diffed and counted, not
estimated), and CPU time / peak RSS are best-effort, Linux-only, read straight from
/proc/self/{stat,status} — honestly n/a off-Linux or when unavailable, never fabricated.

Every run saves a timestamped JSON report to evals/reports/ and exits non-zero when the
aggregate gate (ekos_evals::report::GateThresholds) misses — deliberately not wired into CI
(real LLM calls against a real workspace, not free/fast/deterministic enough for every PR yet).
ekos eval history reads every saved report back and renders one line per run, newest last — the
"run history" a saved-JSON-per-run design gives for free. See evals/README.md for the scenario
schema and category breakdown, and ekos/docs/rfcs/0138-eval-harness.md for the full design.

Hierarchical rollups (RFC 0044)

Every other context-saving mechanism in EKOS (capped search results, hop-bounded graph walks) is
retrieval-limiting — fewer raw facts, never a synthesized higher-level one. ekos commit now
also synthesizes deterministic, zero-LLM Rollup objects: one per directory subtree (crate-level
by default) or, in a multi-project [observe] paths estate, one per project — each carrying real
member counts, a kind breakdown, and boundary-relationship counts (what crosses in/out of the
subsystem), linked to every member via the same Contains relationship everything else already
uses. This is exactly what closes the "huge project/many projects" context-window gap: an agent
asking about a whole subsystem gets one condensed, evidence-linked object instead of personally
synthesizing meaning from dozens of raw facts. Surfaced automatically in Architecture.md's new
## Subsystems section (see above) — running the same docs generate command against this
repo's own source produces one rollup per crate/plugin (46 at last count).

Hosted demo server (RFC 0045, experimental)

ekos/crates/demo-server is a small, read-only web server over a fixed two-repo catalog —
built to answer a strategic question, not a roadmap phase: pick EKOS's single most painful task
(making sense of a codebase without hitting an LLM's context-window ceiling) and put it in front of
peers in a 5–10 minute demo, without anyone installing the CLI. Two binaries:

cargo run -p ekos-demo-server --bin prerender -- <curated-markdown-dir> <output-html-dir>  # bake step
cargo run -p ekos-demo-server --bin demo-server -- catalog.toml                            # serve

prerender pre-renders ekos docs generate --layout curated's Markdown output to static HTML once,
offline (curated HTML isn't a general docs-gen feature yet — see the RFC). demo-server serves
that pre-rendered output plus a POST /ask endpoint that reuses AiRuntime::ask unmodified,
refusing to start rather than degrading silently if ANTHROPIC_API_KEY isn't set. Not general
self-serve ingestion — a fixed, pre-baked catalog only. Not yet demo-ready: implemented and
verified against a placeholder key (routing, boot check, rate limiting, static serving all confirmed
correct), but live-question answer quality is unverified pending a real API key and a rehearsed run
— see devlog_45.md and TODO.md.

AI agent access (MCP)

ekos mcp serve --workspace <dir> exposes the read-only Runtime as a Model Context Protocol
server over stdio (RFC 0013), a raw TCP socket (--tcp, RFC 0115), or Streamable HTTP
(--http, RFC 0143) — tools: ekos_search (limit param — RFC 0124; mode
lexical/vector/hybrid for semantic matching, plus arms_run in the response — RFC 0125),
ekos_query /
ekos_retrieve (compiled fact + graph answers and the inspectable query plan / evidence set, no
LLM — RFC 0124), ekos_ekl (EKL filters on built-in columns and on any property, WHERE status = 'confirmed' or a nested a.b; a field no candidate has is an unknown field error, not an empty result — RFC 0175; it supports point-in-time AS OF <timestamp> queries,
COUNT/GROUP BY aggregation — RFC 0096, and SEMANTIC 'text' retrieval candidate sets — RFC
0124), ekos_neighborhood (depth 0-3, stops at max_objects — default 500, at most
5,000 — and says so with truncated: true),
ekos_state, ekos_dependents (single-hop impact analysis), ekos_impact (directed,
kind-filtered, multi-hop impact tracing — RFC 0018), ekos_graph_export (the whole compiled
graph as nodes + edges in one call — filtered, optionally collapsed to super-nodes, truncation
reported; RFC 0127), ekos_diff (raw ledger-entry changes since
T), ekos_audit (the write history of one object/relationship with the pipeline run, stage, and
source artifact behind each version — RFC 0135; the source artifact is the object/relationship's
own recovered KnowledgeArtifact id(s) where compile tracked one, falling back to the run's CKM
content hash for compiler-synthesized objects like rollups and risks), ekos_status (with a
freshness block: does the ledger still match the source? RFC 0171), ekos_conflicts (where
sources disagree about one fact, every claim with its file and line — read-only, RFC 0172; an
ekos_query/ekos_retrieve claim about such an attribute is marked disputed: true, RFC 0173),
ekos_transformation_explain/ekos_transformation_diff (Transformation IR
explanation and migration diffing — RFC 0028), ekos_architecture_evaluate/
ekos_architecture_drift/ekos_architecture_diff (real completeness/evidence-coverage scoring,
documentation drift, and a real architecture-level diff between two points in time — technologies,
crate role classifications, risks, open questions — distinct from ekos_diff's raw entry report;
RFC 0065/0068 §55/RFC 0107-0108), ekos_semantics_lookup/ekos_semantics_gaps (RFC 0170 —
the recovered business meaning of a table, column or code value with its review status and
evidence, and the open semantic gaps; read-only — confirming or rejecting stays a human-only CLI /
web console action), and ekos_identity_review/ekos_architecture_review (confirm or
reject a cross-system identity match, or an LLM-classified crate role claim — RFC 0029/RFC 0109,
the two write-capable tools; every other tool reads only the local ledger, except the opt-in
ekos_session_note, RFC 0151 — it writes a redacted note to a local inbox file, never to the ledger). Long-lived server
sessions reuse one cached, read-only ledger handle across calls without ever blocking a concurrent
ekos build/commit in another process (RFC 0097). Every read tool (and ekos ekl run from the
CLI) appends one line to .ekos/query-log.jsonl — a real usage log the previous designs had no
equivalent of, groundwork for a future materialized-views pass (RFC 0114); a static heuristic
classifies each call cheap/expensive from its own arguments and opportunistically caches an
expensive one's result for an identical repeat while the workspace hasn't changed underneath it. A
gated ekos_clickhouse_query tool (RFC 0056) is also available, off by default — see the
ClickHouse connector section below. Connect Claude Code with:

claude mcp add ekos -- ekos --config /path/to/ekos.toml mcp serve --workspace /path/to/workspace

The server also honors EKOS_WORKSPACE and EKOS_CONFIG environment variables, so a
registration can be path-free: claude mcp add ekos --env EKOS_WORKSPACE=/path/to/workspace -- ekos mcp serve.

TCP transport — one server, multiple clients (RFC 0115)

Stdio mode spawns a fresh ekos mcp serve process (and a fresh cached ledger handle) per client,
which is fine for one tool but wasteful the moment a second one wants to talk to the same
workspace — a second Claude Code session, PyCharm's AI chat, or any other MCP-speaking tool.
--tcp <addr> starts a second, additive transport on the same command: a plain NDJSON-over-TCP
socket that any number of clients can connect to concurrently, each getting its own
std::thread::spawn'd connection and its own independent cached ledger handle (not shared across
connections — RFC 0115's Concurrency model section explains why). Stdio stays the default and is
completely unaffected when --tcp is omitted; passing it just adds the second transport alongside.

Local — multiple tools on the same machine. Bind loopback and point every local tool at it:

ekos mcp serve --workspace /path/to/workspace --tcp 127.0.0.1:7331

Any MCP client on that machine that supports connecting to a raw TCP socket (rather than spawning
its own subprocess) points at 127.0.0.1:7331 instead of a spawn command. Verify the server is
actually answering before wiring up a client:

printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26"}}\n' \
  | nc 127.0.0.1 7331

A one-line JSON-RPC response ("serverInfo":{"name":"ekos", ...}) confirms the server is up and
speaking the protocol correctly.

Optional bearer-token auth (RFC 0128). --token-file <path> (or, if that flag is absent,
the EKOS_MCP_TOKEN env var; --tcp-token-file is a back-compat alias) requires every TCP
connection's first message to be an initialize request carrying a matching
params._meta.token — anything else gets a single -32001 unauthorized and the socket closes
before any tool is reachable. The comparison is constant-time. Token-less --tcp is unchanged
(RFC 0115 back-compat); stdio is never gated. This is a plaintext-socket bearer token — defence
against a second local process connecting casually, not against a network attacker who can
read the wire; use the SSH tunnel below for that.

ekos mcp serve --workspace /path/to/workspace --tcp 127.0.0.1:7331 \
  --token-file /run/secrets/ekos-mcp-token

Remote — a client on a different machine. There is no TLS on this transport, and auth is
only the optional plaintext bearer token above (RFC 0115/0128's explicit v1 scope) — binding an
externally-reachable address exposes the same read surface stdio gives a spawning parent process,
plus the two write-capable tools, to anyone who can reach it. Two safe ways to do this:

• Trusted private network only, if the workspace machine and every client already sit on one
  (e.g. a home LAN, a VPN, a locked-down VPC): bind the interface facing that network instead of
  loopback, e.g. ekos mcp serve --workspace /path/to/workspace --tcp 0.0.0.0:7331, and firewall
  the port to that network explicitly — never expose it to the open internet.
• SSH tunnel (recommended for anything crossing an untrusted network), keeping the server itself
  bound to loopback on its own machine:

  # on the workspace machine
  ekos mcp serve --workspace /path/to/workspace --tcp 127.0.0.1:7331
  
  # on the client machine
  ssh -N -L 7331:127.0.0.1:7331 user@workspace-host
  

  The client then connects to its own 127.0.0.1:7331, tunneled over SSH's encrypted, authenticated
  channel — the EKOS server itself never has to bind or trust anything beyond loopback.

Both EKOS_WORKSPACE/EKOS_CONFIG env vars and --config still apply the same way they do for
stdio mode; --tcp only changes how clients connect, not which workspace is served.

Limits, on every transport. A message line is capped at 4 MB, and --tcp serves at most 64
connections at once; the first line is read under that cap before any token is checked. A request
that panics is answered with -32603 and the server keeps serving, where before a panic killed a
stdio server and permanently wedged --http's single worker. --http queues at most 64 waiting
requests and answers 503 with Retry-After past that. Traversal bounds (depth, max_hops,
max_objects) are refused with the limit named when out of range, never silently clamped. An
argument name a tool does not declare (e.g. max_depth for ekos_impact, whose parameter is
max_hops) is refused with the accepted names listed, instead of being ignored. Token-less
--tcp or --http on a non-loopback address logs a warning.

HTTP transport — for clients that only take a URL (RFC 0143)

Many MCP clients (VS Code / GitHub Copilot agent mode, Visual Studio 2022, ChatGPT Developer
Mode, mcp-remote) only offer stdio (spawn a command) or Streamable HTTP (a URL) — they
cannot speak the raw TCP socket above. --http <addr> serves MCP's HTTP transport at one
endpoint, POST /mcp:

ekos mcp serve --workspace /path/to/workspace --http 127.0.0.1:7331

--http and --tcp are mutually exclusive (each replaces stdio). The POST response is
application/json, or a single-shot text/event-stream when the client's Accept header asks
for it (ChatGPT's connector requires the latter); either way the response bytes are the same
JSON-RPC. There is no server push — GET /mcp returns 405, no sessions. Auth is the same
token (--token-file / EKOS_MCP_TOKEN), presented over HTTP as an Authorization: Bearer <token> header checked on every request. The Origin header, when present, must be loopback or
an explicit --http-allow-origin <origin> (DNS-rebinding defence); a request with no Origin
(the normal case for editors) is allowed. All HTTP requests are serialized through one worker
thread — a slow tools/call blocks the next request, matching the stdio loop.

VS Code (.vscode/mcp.json) or Visual Studio (.mcp.json):

{
  "servers": {
    "ekos": { "type": "http", "url": "http://127.0.0.1:7331/mcp" }
  }
}

Verify it is answering:

curl -s http://127.0.0.1:7331/mcp -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18"}}'

For a remote client, tunnel it the same way as the TCP transport (ssh -N -L 7331:127.0.0.1:7331 …);
there is no TLS on --http — terminate it at a reverse proxy if a deployment needs it.

ChatGPT (Developer Mode connector, Plus/Pro) is cloud-hosted and cannot reach 127.0.0.1, so
it needs a public HTTPS URL — put a tunnel in front (cloudflared tunnel --url http://127.0.0.1:7331, then use https://<name>.trycloudflare.com/mcp). The tunnel URL is
world-reachable and exposes the two write tools too, so protect it (Cloudflare Access / Tailscale,
or a short supervised session). ChatGPT's Deep Research connector won't work — it requires
tools named exactly search/fetch; only Developer Mode (full MCP) does.

Marketing agent (RFC 0030)

ekos marketing publish [devlog] turns a devlog_N.md into a human-approved X (Twitter) release
announcement: it classifies the devlog's importance (skipping docs/tests/refactor-only entries),
drafts a tweet through the same LlmProvider used elsewhere, validates it (length, EKOS mention,
GitHub link, hashtag count), asks for Y/N/E approval, and publishes via a real OAuth 1.0a-signed
POST /2/tweets — with marketing/posted/tweets.json preventing the same devlog from ever being
posted twice.

ekos marketing publish            # latest devlog_*.md, interactive approval
ekos marketing publish 28         # a specific devlog number
ekos marketing publish --dry-run  # preview only — never posts, never records

Configure via [marketing]/[marketing.twitter] in ekos.toml (see marketing/README.md);
publishing requires TWITTER_API_KEY/TWITTER_API_SECRET/TWITTER_ACCESS_TOKEN/
TWITTER_ACCESS_SECRET in the environment and stays off until [marketing.twitter] enabled = true is set explicitly.

World Engine simulation (RFC 0047-0055, experimental)

Auxiliary, opt-in tooling built on top of the same ledger, kept deliberately separate from the
compiler pipeline above: multi-agent scenarios with beliefs, goals, deterministic round-based
decision-making, seed-reproducible priority/resource conflict resolution, a VirtualForum
(channels, replies, likes, follows, shares), a durable, replayable event log, and world.sources
document ingestion (real files, via the actual localdocs connector) — layered additively over the
existing graph (see ekos/docs/rfcs/0047-0055 and devlog_47.md-devlog_55.md). Define a
scenario and its agents in YAML (source-document-style agent.yaml/scenario.yaml shapes), run
it, and read it back afterward:

ekos simulate scenario.yaml             # runs scenario.yaml's own simulation.rounds
ekos simulate scenario.yaml --rounds 5  # override the round count
ekos simulate scenario.yaml --seed 42   # override the round's priority/resource-conflict seed
ekos replay scenario.yaml               # read back every recorded round, read-only
ekos replay scenario.yaml --round 2     # narrow to one round

EKOS Migrate (PostgreSQL → ClickHouse, opt-in)

Evidence-backed migration of a PostgreSQL database to an analytical target — a migration proof
system rather than a SQL converter. Every scope decision, finding, mapping, validation result and
human approval is a ledger fact with provenance, and the report cites those facts or does not ship
(ekos/docs/rfcs/0154-0167).

Shipped: a real table was migrated PostgreSQL → ClickHouse and validated at V1/V2/V3, with a
planted one-row change caught by V3. The source side of logic migration is recovered: PL/pgSQL
routines statement by statement, views, and triggers classified by what they do, all linked to the
tables they touch (RFC 0163, 0169). Not yet: translating that logic to the target (RFC 0164:
lowering, constrained reconstruction, V5 differential execution), V0/V4 validation, signoff, a
second target, CDC and cutover.

ekos migrate init --name ledgersmb \
  --source postgres://pg/ledgersmb \
  --target clickhouse://ch/ledgersmb \
  --source-secret-env PG_PASSWORD      # the variable's NAME; the value is never stored
ekos migrate discover                   # live catalog → one unit per table + live-vs-repo drift
ekos migrate profile --tier p1          # p0 catalog only | p1 bounded sample | p2 exact, budgeted
ekos migrate assess                     # DQ + target-compatibility rules, affected rows measured
ekos migrate map --emit out.sql         # type map with lossiness, CH design, DDL with its reasoning
ekos migrate review --unit public.orders --env staging   # raise an approval request
ekos migrate approve <REQUEST> --as alice --show-evidence # human-only; no MCP equivalent exists
ekos migrate load --unit public.orders --dry-run         # gate + print every statement
ekos migrate load --unit public.orders                   # create, then copy chunk by chunk
ekos migrate validate --unit public.orders --tier v3     # V1 counts, V2 aggregates, V3 row hashes
ekos migrate report --out report.md     # compiled from ledger facts, every claim cited
ekos migrate status                     # units by state, and what is blocking
ekos migrate lower --database lsmb --sql   # views → ClickHouse, trigger proposals, routine eligibility
ekos migrate check-candidate --source open_items --candidate view.sql   # does it state anything new?
ekos migrate v5 --source-sandbox postgres://sbpg/scratch --target-sandbox clickhouse://sbch/scratch
                                        # run each lowered view against its source, on sandboxes
ekos migrate reconstruct --source close_period   # one routine → a model → kept only if it invents nothing

What each stage guarantees:

• Credentials never reach the ledger. A DSN names an alias, [migrate.connections.<alias>]
  resolves it to host/port/user, and secret-env names the password variable. A connection string
  carrying a password is refused rather than parsed and stripped — it has already leaked into shell
  history. Row values never reach the ledger either: profiles hold statistics, and a column
  classified as PII gets no bounds and no top-k at any tier.

• Logic is lowered by rule, or not at all (RFC 0164, Phase 1). ekos migrate lower reads the
  recovered views, triggers and routines from the ledger and their definitions from the source
  files. It opens no connection, calls no model and writes nothing.

  • A view becomes a ClickHouse CREATE VIEW, rewritten where the same text would return
    something else: sumOrNull for sum, join_use_nulls for outer joins, explicit NULLS FIRST,
    Nullable(T) casts, char_length, quoted names. Each rewrite is listed under the view.
  • Anything without a rule blocks the view by name (LOGIC.CH.DIVISION_TYPES,
    LOGIC.CH.FUNCTION: string_agg() has no lowering rule). A materialized view always blocks: a
    ClickHouse materialized view is an insert trigger, not a stored result.
  • The emitted statement must parse under the ClickHouse grammar and read back identically, and
    must state nothing its source does not.
  • A trigger gets a proposed replacement per class and the behaviour change it implies; Mixed
    and Unknown triggers are left to a person. No trigger is translated.
  • A routine is eligible for reconstruction only when every statement was recovered and none is
    built at run time.

  ekos migrate check-candidate applies the second check to a statement you supply: it is rejected
  if it names a table, column, literal, function or comparison the source does not.
  --record on lower saves each lowering and trigger proposal as a MigrationProposal with
  status: proposed.

• A lowered view is run against its source (V5). ekos migrate v5 builds a small fixture from
  the view's own comparisons — the literal and its neighbours for every column <op> literal,
  NULL in every column, compared columns crossed — loads it into a PostgreSQL sandbox and a
  ClickHouse sandbox, and compares what the source view and the lowered view return.

  • The verdict always carries branch coverage: how many of the view's comparisons the fixture took
    both ways. V5 passed alone is never printed.
  • A view that cannot be run (char(n) column, a LIMIT, a column the recovered tables lack)
    is not run, with the reason. Not run is not a pass.
  • Both connections must be marked sandbox = true under [migrate.connections.<alias>], and a
    migration source is refused even with the mark. PostgreSQL work happens in one transaction in
    its own schema and is rolled back; ClickHouse work in its own database, which is dropped.
  • Rows are compared as multisets. Row order is not checked.

• A model's rewrite of a routine is kept only if it invents nothing.
  ekos migrate reconstruct sends one PL/pgSQL routine to the configured model.

  • A routine with unrecovered statements or dynamic SQL is never sent.
  • A model that is not local needs --allow-metered on that run; there is no config switch.
  • The answer is rejected if it names a table, column, literal, function or comparison the
    routine does not. A survivor is a proposal, unexecuted: the check finds what was added, not
    what was left out, and V5 for routines is not built.

• Sessions are read-only against the source (default_transaction_read_only, statement/lock
  timeouts, low work_mem, a replica-lag guard that refuses a run above the configured threshold).

• Assessment measures, it does not guess. Each finding carries the SQL that measured it.
  Undeclared foreign keys are inferred from real joins in the compiled code and in
  pg_stat_statements, then checked by inclusion before anything is claimed.

• Documentation is checked against the data (DQ.CONSIST.DOC, RFC 0172 Phase 3). Where a
  column's repository or live comment — or a sentence in a Markdown or Confluence page that names
  exactly one `table.column` in code formatting — makes a checkable claim — "never null", "unique", "one of
  open, closed or void", an A=asset,L=liability legend, "between 0 and 100", "positive" — assess
  counts the rows that contradict it (counts only, no values). A contradicted claim becomes a
  doc_vs_data conflict carrying both sides: the comment with its path:line, and the measured
  count with its query. ekos conflicts list --type doc_vs_data; a re-assessment drops the ones the
  data no longer contradicts.

• Every executed statement is parsed and classified (never string-matched). An unparseable,
  unknown or credential-carrying statement is refused. Outside the sandbox, a statement runs only if
  an approval matches its artifact id, content hash and environment — there is no override.

• Approval risk is computed from statement class × environment × lossiness × blast radius (how
  many compiled objects depend on the table) × affected rows. An approval is pinned to an evidence
  snapshot, so it dies if the evidence changes after it was given. Self-approval is refused, and R4
  needs two distinct approvers plus a typed confirmation. Thresholds live in migrate.policy.toml,
  and its hash is recorded on every approval.

• Validation compares engines through one canonical serialization (RFC 0155), cross-checked by
  PostgreSQL, ClickHouse and Rust. A tier passes only with zero blocking divergences and every
  planted control fired, so a check that never ran cannot report green.

• The report is compiled, not generated. Every factual sentence carries fact ids that are
  resolved and structurally checked. Prose containing a number is refused. The same snapshot
  recompiles byte-identically.

Needs [migrate] enabled = true in ekos.toml. A PostgreSQL and a ClickHouse sandbox are in
docker-compose.migrate.yml; the live suites run with EKOS_MIGRATE_LIVE=1.

A scenario's world: { sources: [reports/report_01.md] } ingests real documents (PDF/DOCX/text/
Markdown/HTML/email) into its starting world; an agent's knowledge:/relationships: can
reference an ingested document by that same path string.

By default simulate writes to a dedicated .ekos/simulations/<scenario-id>/ledger.db, never
the real workspace ledger — simulated agents and events are fictional and regenerated on every run,
and because the ledger has no delete/tombstone mechanism (RFC 0043), they should never permanently
commingle with real, evidence-backed compiled knowledge. --ledger <path> opts back into a
different target explicitly, including the real workspace ledger, if a caller wants that.

This is a distinct capability from the "compiler for enterprise knowledge" positioning above, not
a replacement for it — kept intentionally separate rather than blended into one pitch. Whether it
grows into its own product surface remains a further, explicitly not-yet-committed idea, not a
decided roadmap direction — revisited RFC by RFC rather than assumed to keep expanding.

ClickHouse connector (RFC 0056)

Two independent pieces. Compiled metadata — ekos build/ekos recover observe a configured
ClickHouse database's system.tables/system.columns (via ClickHouse's stock HTTP interface, no
native driver) and compile every table into a real KirObject(ObjectKind::Table), searchable
through ekos_search/ekos ekl and cross-system identity-resolvable against same-named tables
elsewhere in the estate, the same way file-based SQL recovery already is:

export EKOS_CLICKHOUSE_URL=http://localhost:8123
export EKOS_CLICKHOUSE_DATABASE=analytics
export EKOS_CLICKHOUSE_USER=default        # optional
export EKOS_CLICKHOUSE_PASSWORD=            # optional
ekos build && ekos recover

File-based ClickHouse DDL (.sql files routed to the "clickhouse" dialect via RFC 0031's
[[recover.sql.dialect-rules]]) goes through the same sqlparser::dialect::ClickHouseDialect this
connector uses for its live SELECT-only gate. sqlparser never supported several real ClickHouse
CREATE TABLE clauses at all — CODEC(...) (RFC 0057), and INDEX ... TYPE ... GRANULARITY,
PARTITION BY, SAMPLE BY, SETTINGS, and whole CREATE DICTIONARY statements (RFC 0058) — found
and closed while using EKOS to document a real open-source repo's ClickHouse schema
(Plausible Analytics). ClickHouseDialectParser::preprocess strips each, well-formed occurrences
only, before the SQL reaches sqlparser; live-verified against that real repo's full,
unmodified structure.sql, which now compiles into real Table KIR objects with zero parse
warnings.

Live NL-to-SQL query — the one path in EKOS that intentionally crosses the Key Invariant above:
an LLM builds a ClickHouse SELECT from the compiled schema and the question, the generated SQL is
parsed and hard-rejected unless it's exactly one SELECT (no writes, no multi-statement batches),
then it's run live, redacted, and returned — every call is recorded as an Evidence/Event pair in the
ledger for audit, though the row data itself is never ledgered:

ekos clickhouse ask "how many orders were placed last week?"

This CLI command is always available. The matching ekos_clickhouse_query MCP tool is off by
default — ekos mcp serve only lists it once a workspace explicitly opts in:

[clickhouse]
enable-mcp-query = true

Deck, verified live against a real ClickHouse container: ClickHouse: Compiled Metadata + Live
NL-to-SQL.

A second deck covers the same connector's file-based DDL path, run cold against a real
open-source repo (Plausible Analytics) EKOS had never seen: EKOS Cold Against Plausible's
ClickHouse Layer
— git/CI/dependency knowledge compiled cleanly, and real gaps surfaced in sqlparser's
ClickHouseDialect (CODEC, INDEX, PARTITION BY, SAMPLE BY, SETTINGS, CREATE DICTIONARY), all since closed (RFC 0057, RFC 0058) — that same repo's full structure.sql now
compiles cleanly into real Table objects.

A third deck, ClickHouse Extraction, After the
Fix, re-analyzes
the same repo after both RFCs: 15/15 real tables now recover with zero parse warnings, but
re-analyzing surfaced a second, unrelated finding in a different pipeline stage — identity
resolution (crates/identity) over-merging 6 of those 15 real imported_* tables into one
identity at confidence 0.93, because they share both a name prefix and a common 8-column "spine."
Reported the same way the parser gap was, not silently fixed.

The same case study also produced real generated documentation and two live demos, all against
github.com/plausible/analytics, a real unmodified
open-source repo:

• ClickHouse Component — Generated Documentation
  — full schema, write/read paths, and data-migration framework for the event store, researched
  using EKOS's compiled ledger plus direct source verification.
• Top Referrers Dashboard
  — a real analytics dashboard reproduced from a screenshot; every number is a live query, built by
  a local Ollama model against EKOS's compiled schema and run against a real ClickHouse server.
• Why That Day Spiked —
  an open-ended "why" question answered by chaining real ekos_clickhouse_query MCP calls over
  stdio JSON-RPC (including a real failure and retry), plus a technical breakdown of how Claude,
  MCP, and EKOS's pipeline fit together.

A fourth deck, Proving the Core Loop, Cold, on a Real
Repo, goes past the
ClickHouse slice: a genuinely cold init → build → recover → resolve → compile → commit run over
the whole 2,045-file repo, timed stage by stage (~107s end to end), plus a real ekos ask + MCP
question set graded against ground truth read from the repo itself. It found three new gaps in one
sitting — a previously-unknown Postgres sqlparser failure (INCREMENT), identity resolution
over-merging real people and unrelated documents (not just ClickHouse tables — a real contributor's
own commit becomes unfindable under their own name), and a retrieval-brittleness bug in ekos ask
itself (full-sentence questions return no context even when the object is trivially findable by
keyword) — all reported the same honest way, not silently patched or hidden, and all fixed the same
day (RFC 0059, RFC 0060, RFC 0061 — see devlog_61.md), each with a live re-verification against
the same real repo, not just a passing unit test.

Compact storage (RFC 0015)

Workspaces created before RFC 0015 can be shrunk in place (both commands verify before
touching anything and leave backups):

ekos ledger status --storage   # per-component size report (or the shorter `ekos status --storage`)
ekos ledger migrate            # ledger v1 → v2: dictionary-zstd payloads (~2.5x smaller)
ekos ledger audit <id>         # write history of one object/relationship — which run, stage and
                               # source artifact produced each version (RFC 0135; --json).
                               # source artifact is the recovered KnowledgeArtifact id(s) for a
                               # tracked object, or the run's CKM hash for synthesized ones
ekos artifact repack           # loose JSON files → packed segments (~7x smaller on disk)

ekos status [--storage] (RFC 0116) is a top-level alias for ekos ledger status — same output,
shorter to type; both forms stay supported. --json (RFC 0127) emits one machine-readable object
instead of the text report — entry/object/relationship/evidence counts, the backend tag, a
per-component storage breakdown, and an mtime-proxy last_write; the text output is unchanged.
ekos doctor --json (RFC 0129) does the same for the environment checklist
({ok, checks:[{name, status, detail}]}), and ekos ledger timeline [--bucket day|week|month]
(RFC 0129) emits cumulative object/relationship counts bucketed by mint time — the growth series
behind the web-console dashboard. All the --json / export commands send their logs to stderr so
stdout is a clean document.

ekos config validate [--json] [--file <path>] (RFC 0130) parses ekos.toml and reports syntax
errors plus [observe] mistakes — most usefully, an ignore-patterns entry like *.lock or
src/fixtures that looks like a glob or path but is matched as a bare directory name, so it
matches nothing. ekos config preview-scan [--json] counts what ekos build would observe under
the current paths / ignore-patterns (files, by extension, and how many directories each
ignore pattern actually pruned) without reading or compiling anything.

ignore-patterns is the only filter — .gitignore excludes nothing from the observation walk.
A git-ignored virtualenv, build cache, or scanner working directory is still fully observed unless
you name it here. Measured on this repo (devlog_174): a .venv/ and two .scannerwork/ directories
made 94% of the observed corpus third-party code — over half the compiled knowledge model
described numpy and pytest internals, and every query ranked the project's own code against 27k
foreign objects. Run preview-scan before the first build on a new workspace, and prefer specific
directory names over generic ones like build/dist, which match a bare path component and can
prune real source.

Bulk graph export (RFC 0127)

ekos graph export writes the whole compiled graph — every object and relationship — as one JSON
(or NDJSON) document. It's the first non-per-object, non-LIMIT-capped read path in EKOS:

ekos graph export                                  # the whole graph, JSON on stdout
ekos graph export --kind Table --kind File         # filter to object kinds
ekos graph export --exclude-rel-kind CoupledWith --min-degree 1
ekos graph export --level aggregate --group-by kind   # one super-node per kind
ekos graph export --format ndjson --output graph.ndjson

Node ids are real object ids (feed them straight to ekos_state / ekos_neighborhood), degree is
computed over the post-filter edge set, and when the graph exceeds --max-nodes/--max-edges the
export keeps the most-connected core and says so in a truncated block rather than silently
returning a prefix. Output is deterministic modulo its generated_at timestamp. The
ekos_graph_export MCP tool exposes the same function to agents.

Web console (RFC 0127/0128/0129/0130/0131/0132/0133/0134/0136/0138/0170)

web/ is a browser surface over one or more compiled workspaces — a FastAPI app (web/api/) plus
a Vite + React app (web/ui/).

• Phase 1 (RFC 0129): a persisted workspace registry, a supervisor that spawns and restarts
  one ekos mcp serve --tcp per workspace on its own, and a statistics dashboard
  (entry/object/relationship/evidence counts, storage breakdown, objects by kind, a growth
  timeline, query-log stats, doctor).
• Phase 2 (RFC 0130): an ekos.toml editor with validate + preview-scan and the
  append-only warning when you narrow [observe].
• Phase 3 (RFC 0131): run EKOS pipeline commands from the browser and watch them stream. A
  hardcoded command allowlist (build/recover/resolve/compile/commit, pipeline,
  doctor, status, ekl, ledger repair, docs generate, …), a per-workspace job queue
  (SIGTERM→SIGKILL cancel, chained pipeline with per-stage status), and an SSE log tail. This is
  the first browser mutation, so it brings the read/write role split: auth is OIDC
  (Authorization Code + PKCE, a claim → the write role) or, when OIDC_ISSUER is unset, two
  static tokens — CONSOLE_TOKEN (read) and CONSOLE_WRITE_TOKEN (read + write).
• Phase 4 (RFC 0132): scheduled runs. A Schedule (workspace + command + cron or interval
  trigger + a required notify_url) fires the same job runner; APScheduler is rebuilt from the
  SQLite table on start. A non-succeeded run POSTs {schedule_id, run_id, status, …} to the
  notify_url; the UI shows each schedule's last-run status.
• Phase 5 (RFC 0133): the graph view (react-force-graph-2d, lazy-loaded). An overview as one
  super-node per object kind → click to expand a kind into its 500 most-connected real objects,
  kind / relationship-kind filter toggles (CoupledWith / FeedsInto off by default), a search
  that flies the camera to a node, and an object panel showing ekos_state — properties,
  relationships, and one evidence row per claim (path · analyzer · confidence · fragment).
• Phase 6 (RFC 0134, RFC 0136): a timelapse slider under the graph — drag it back and the
  diagram redraws as the knowledge existed at that instant; a clicked node's panel then shows the
  evidence it had then. Backed by ekos graph export --as-of <rfc3339> --first-seen (and the
  same as_of / include_first_seen on ekos_graph_export); the browser fetches the latest graph
  once with per-element first-seen stamps and filters it client-side against a frozen layout,
  so scrubbing never refetches and nodes never move. Ticks + the activity histogram come from
  ekos ledger timeline. The slider is only as deep as the ledger's retained history — rich on a
  workspace committed incrementally, near-flat on a freshly-rebuilt one. Graph v2 (RFC 0136)
  adds: neighbourhood isolation (an object's real BFS sub-graph, real edges included, at a
  depth of 1-3 — ekos_neighborhood, unmodified); impact mode (ekos_impact's hop-distance
  trace rendered as node coloring source-outward plus highlighted edges between two impacted
  nodes already on screen — the differentiating screen RFC 0127 called out as "the visual form of
  the claim that currently has none"); a server-side ForceAtlas2 layout (networkx +
  fa2_modified) for graphs past ~2,000 nodes, where the browser's own force simulation stops
  keeping up; and one-click PNG/glTF export of the current view, entirely client-side (whatever
  zoom, filters, and isolate/impact state are on screen, not a server-rendered reproduction).
• Eval history (RFC 0138): every saved ekos eval run report, browsable under a workspace's
  Evals tab — a history table (dataset, agent, PASS/FAIL, answer-correctness/hallucination-rate)
  and a detail view (all eleven metrics — the five headline scores plus tokens/latency/cache/RSS/
  CPU — and the full per-scenario pass/fail/hallucinated breakdown). Triggering a new run reuses
  the existing command runner (eval-run in the allowlist) rather than a bespoke UI — it shows up
  on the Run tab like any other command, with dataset/agent/category/limit params, and
  its progress streams through the same job log every other command already uses.
• Semantics (RFC 0170): human review of business-semantics hypotheses and the LinkML schema
  they export. Review — a queue (needs-review first, then hypotheses by confidence) filterable by
  kind/status/text; each item opens with its definition, evidence lines (path:line + fragment),
  links and rationale commits, and Confirm / Reject (note required) / Edit (name and
  description, or a code's label); tick several (or "select all shown") to confirm or reject
  them in one all-or-nothing decision. Gaps & conflicts — the open questions, conflicting
  definitions and the needs_review queue. LinkML schema — a viewer (concepts, enums and
  tables as a tree; attributes, constraints, permissible values with status chips; "review" on
  any recovered element) and a YAML editor: Validate is a dry-run ekos import linkml
  showing the decisions the edits stand for, Apply records them all-or-nothing; Diff vs
  current export shows the edits side by side (changed blocks with context); plus download and
  load-from-file. Decisions need the write role and run the human-only CLI (never MCP),
  attributed to the OIDC user — or, in token mode, to a typed name recorded as token:<name>.

cd ekos && cargo build --release -p ekos && cd ..
EKOS_BIN=$PWD/ekos/target/release/ekos \
EKOS_CONSOLE_CONSOLE_TOKEN=dev-read EKOS_CONSOLE_CONSOLE_WRITE_TOKEN=dev-write \
EKOS_CONSOLE_SESSION_SECRET=$(openssl rand -hex 16) \
uv --directory web/api run uvicorn --factory app.main:create_app --port 8000 &
cd web/ui && npm install && npm run dev        # http://localhost:5173 — sign in with a token

web/docker-compose.yml runs the same thing (api on :8000, ui on :5173), published on
127.0.0.1 only. Settings live in web/api/.env. With no EKOS_CONSOLE_SESSION_SECRET set, the
API generates a random secret per process (sessions do not survive a restart) rather than signing
cookies with a string anyone can read in this repository. It also logs a warning while the read
token is still the default dev-console-token. A hardening pass
(Phase 7 — performance, theming, packaging) is still to come, authored just-in-time.

Fact-segment engine (RFC 0016) — the default for new workspaces

A brand-new workspace (ekos init, nothing written yet) now runs on the fact-segment engine
(EAV facts, immutable segments, tantivy search, mmap'd reads) by default, as of 2026-08-21 — every
version is signature-verified, and it's real, not aspirational: the RFC's storage gate was amended
with measurements in hand (≤2× of the v2 ledger at equal-or-better read latency — it passes at
1.66× with 19× faster search), and the default switch itself waited on a real month-long soak
period on a live, actively-used multi-project estate before flipping (RFC 0016's own dated
section has the evidence). Any pre-existing SQLite-backed workspace is completely unaffected —
it keeps serving from SQLite forever unless explicitly migrated. ekos ledger migrate --v3
migrates an existing SQLite workspace onto the fact engine — the SQLite source is left untouched,
and deleting .ekos/ledger/facts/ rolls back.

Since that default switch, the fact engine has picked up three further hardening passes, all
opt-in-free and automatic on any fact-engine workspace:

• Concurrency safety (RFC 0104) — writes take a real cross-process file lock (fs4) instead of
  assuming a single writer, and multi-step writes run inside a transaction that rolls back cleanly
  on failure rather than leaving a half-written segment.
• Self-healing search + ledger repair (RFC 0103/0105) — a stale or corrupted tantivy schema
  is detected and rebuilt automatically on open; ekos ledger repair additionally re-verifies every
  sealed segment's signature and reports (or fixes) any that fail.
• Version-chain checkpoints (RFC 0106) — periodic checkpoints into checkpoints.jsonl bound how
  far back a version-chain read has to walk, keeping object_at/point-in-time reads fast as ledger
  history grows.

Partitioned storage (RFC 0111 Phase A) — opt-in

A brand-new workspace can opt into a partitioned store by setting [storage.partition] in
ekos.toml:

[storage.partition]
dimension = "entity-kind"   # partition by ObjectKind (Table, File, …); relationships by kind
time-bucket = "monthly"     # "daily" | "weekly" | "monthly"

Data then splits across many independent fact-segment ledgers keyed by kind + time bucket, with a
persisted catalog and a run-file index so a reopened store resolves any object/relationship with no
partition scan; aged partitions tier to cold (handle evicted, promoted back on read). It is a
drop-in for the single-ledger backend — every command (ekos status/ekos ledger status report
it as (partitioned, RFC 0111) with real counts), the MCP server, and docs generate work
unchanged. Retrieval fuses each partition's ranked hits plus a cross-partition exact-name arm
(RFC 0120), so an exact-name query still promotes the named object even when a strong lexical hit
lives in another partition. Existing SQLite or fact-engine workspaces are never switched
implicitly, same rule as the fact-engine default.

Multi-machine distribution (Phase B, RFC 0113) is feature-complete at v1 (2026-08-30) and
validated end-to-end: two autonomous fault-injection soak runs — the second against a real S3
endpoint (MinIO) and a 95-partition workspace — found and fixed 8 defects, each with a regression
test (devlog_144). What it provides:

• a SegmentBackend seam — LocalFsBackend (default) or ObjectStoreBackend (S3 / Azure /
  in-memory, behind a feature flag);

• a coordinator (ekos coordinator serve) that hands out fencing-tokened write leases and
  tracks per-partition commit watermarks over newline-delimited JSON-RPC;

• a compile worker (ekos compile-worker run) that runs the real
  build → recover → resolve → compile → commit pipeline under a coordinator lease, then
  registers the partitions it wrote and commits the new generation;

• self-describing object-storage partitions — [storage.partition] segment-backend-url = "s3://…" routes each partition's segments (sealed and active), manifest.json, dict.bin,
  HEAD, and search index to S3 / Azure / any S3-compatible store (MinIO included; provider
  credentials come from the standard AWS_* / AZURE_* env vars). A query worker can then serve
  the partition from its URL alone — including committed-but-unsealed rows, which under
  fine-grained partitioning is most of the data;

• query workers (ekos query-worker serve) that pull a partition into a local cache and serve
  reads for it, and a DistributedLedger gateway that implements the same KnowledgeStore
  trait every command already uses — fanning reads across the workers, merging, and failing over
  to another worker when one is unreachable — so pointing a workspace at a cluster is just
  [storage.distributed] in ekos.toml:

  [storage.distributed]
  coordinator   = "coordinator.internal:7333"
  query-workers = ["qw1.internal:7334", "qw2.internal:7334"]
  

• distributed search — the gateway fans each shard's BM25 top-k to a worker and merge-sorts
  the results (shard-local term statistics, the standard query-then-fetch approximation);

• a pooled, concurrent, pruned, fault-tolerant gateway — DistributedLedger reuses one
  connection per coordinator/worker instead of reconnecting per call, fans a multi-partition read
  out concurrently, prunes id-scoped reads (get_object and friends) to the few partitions the
  coordinator's index says actually hold that id, and rotates to the next worker on a connection
  failure;

• adaptive leases — ekos compile-worker derives its heartbeat from the lease's real TTL, so
  ekos coordinator serve --ttl-seconds 5 (fast failover) is safe; ekos compile-worker run --force is the Service-A equivalent of ekos resolve --force; ekos compile-worker run --retry-lease-seconds <N> keeps retrying (every 3s) if the shard's lease is already held by
  another worker instead of failing immediately — 0 (default) preserves the original fail-fast
  behavior, and only an "already leased" conflict is ever retried, never a genuinely failed run.

That completes Phase B at its v1 scope. Known follow-on: interrupt-of-in-flight-work on lease
loss (a fenced worker currently runs its pipeline to the end, then its commit is rejected) — needs
a cancellation signal threaded through the whole pipeline, materially bigger than the acquire-retry
loop above. None of this affects Local mode, which stays the default.

Development Process

All significant architectural decisions begin as RFCs in docs/rfcs/. No feature is implemented until its RFC is accepted. See CLAUDE.md for the full mandatory development workflow.

Presentations

Live decks at alexeyban.github.io/EKOS — every claim in them is reproduced live against real repos, not staged:

• Full-Stack Test Run — an autonomous 22-act pass over the distributed storage stack (RFC 0111/0113), the compiled-knowledge query engine (RFC 0118/0119–0126), and the MCP protocol (RFC 0013/0115): coordinator fencing, gateway failover, RRF fusion, REASON citation checks, the vector arm on a real embedding model, and the write-vs-read-only-gateway safety assertion. Three partitioned-store bugs it surfaced, fixed and re-verified in the same run.
• Claude Code + EKOS — how Claude Code searches and analyzes a codebase through EKOS's MCP server instead of raw grep/Read, with a measured with-vs-without comparison and real token/usage numbers.
• The AI-Native Enterprise Knowledge Compiler — the startup pitch, audited live by Claude Code using EKOS's own MCP server.
• ClickHouse: Compiled Metadata + Live NL-to-SQL — the one explicit, audited exception to "AI never touches raw enterprise systems directly," verified live against a real ClickHouse container, honest failures included.
• GitHub, Live, End to End — the GitHub connector's first live run, 1,600 real issues/PRs from a real repo: two known gaps fixed before the run, a third (96% of items collapsing into one identity) found only at real scale and fixed the same session, and the residual limitation reported honestly, not hidden.
• EKOS from 53 to 87 — the same 101-question eval, graded deterministically, moving from a local llama3 8B to DeepSeek V4 Flash (OpenCode Zen) and then to better document structure and query planning: 53 → 70 → 79 → 87, what each step bought, the 14 that still fail and why.
• Eval Comparison: Fixing a Self-Contaminated Ledger — how the eval harness found that 94% of what EKOS was compiling was not its own code, and three bugs that made fixes silently do nothing.
• TSD System Documentation and How TSD Works — a Windows CE barcode terminal and its desktop server documented from compiled .NET binaries alone (RFC 0148), every claim traced to a metadata token.
• Distributed Storage Under Fire — two end-to-end runs of the RFC 0111/0113 distributed engine and the eight defects they found.
• EKOS Web Console — the web console, RFC 0128–0133.

See alexeyban.github.io/EKOS/presentations.html for the full list.

Community

Questions and ideas: GitHub Discussions. Release
announcements post to X: @ekosproject (via ekos marketing publish,
RFC 0030). The project site is ekos.dev.

Versioning Roadmap

Version	Milestone
v0.1	Compiler Infrastructure
v0.2	Observation Layer
v0.3	Knowledge Recovery
v0.4	Identity Resolution
v0.5	Knowledge Ledger
v0.6	Runtime
v0.7	AI Layer
v1.0	Enterprise Knowledge Compiler

The latest release is v1.1.0 (2026-10-14: RFCs up to 0187, the setup wizard and full packages);
1.0.0 (2026-09-23) is the v1.0 row above. CHANGELOG.md lists what each release
added. The phase-by-phase record is TODO.md and
the devlogs, not this table.

Team

Alexey Banaev — Founder & Lead Developer

Alexey is a Data Engineer and Solution Architect with extensive experience designing data
platforms, analytics solutions, and distributed systems. He founded EKOS to address a fundamental
challenge in modern data environments: making complex, fragmented data ecosystems understandable,
traceable, and usable.

As EKOS's founder and main developer, Alexey leads the product architecture and engineering,
combining data engineering, software development, and AI-driven approaches to build a practical
solution for modern data teams.

Omid Ahmadi — Core Team, Ecosystem & Growth

AI Researcher & Agentic Systems Engineer

Researcher focused on sustainable AI applications across the oil, gas, and petrochemical
industries, with a focus on intelligent agents, autonomous systems, and reliable AI infrastructure
for industrial environments.

License

MIT — see LICENSE.