CORTEX

CORTEX is a local AI agent designed around the limits of 7B models. A FastAPI server plans each request, calls sandboxed tools, keeps long-term memory in SQLite and ChromaDB, and streams every step to a Next.js UI. Models run in Ollama on your own machine. The default, qwen2.5:7b, runs on a laptop GPU with 6 GB of VRAM.

With the default config, nothing you type, upload or store goes to a hosted service. The one tool with internet access, web_fetch, opens only URLs you typed yourself.

Recorded live with qwen2.5:7b on a 6 GB laptop GPU, with the model's thinking time sped up. In order: a direct answer, sandboxed Python, a saved file, memory across a page refresh, a refused "delete every file" injection, a question about the README, a web fetch, a blocked request to localhost, and a multi-step calculation.

Why the rules are in code

A 7B model doesn't reliably follow rules that exist only in its prompt. Running CORTEX through 187 live conversations with qwen2.5:7b made that concrete. The model said "I saved the file" without calling the tool. It pasted a program again instead of saying it couldn't run it, stated a number its code never printed, stored "your name is Ada" from a JSON example, and followed an instruction hidden in text it had been asked to summarize.

So the rules that matter don't live in the prompt. Which tools a turn may use, whether a file may be written, which URLs may be fetched and what memory may keep are decided in code around the model. The model picks tools, writes code and phrases the answer; it doesn't decide what it's allowed to do.

Each fix has a regression test, and CHANGELOG.md records the failure behind it.

How a request runs

1. Context. The session's last 6 turns are loaded from SQLite, and facts related to the message from ChromaDB.
2. Code the sandbox can't run. If you ask to run a pygame, tkinter or input() program, or one that needs a blocked package, you get an explanation and the local run command. The model isn't called.
3. Plan. The planner sees the tool list and, for near-duplicate past tasks, the tool sequence that worked before. Guards then adjust the plan using only your own words, never quoted or pasted text: a plan to delete data becomes a refusal, web steps you didn't ask for are dropped, and a file save you asked for is added if the plan missed it.
4. Execute. A direct-answer or refusal plan runs with no tool schemas. Otherwise the executor runs ReAct steps of up to 3 tool calls each, and every call is validated against the tool's JSON schema and time-limited (30 s, or 10 s for Python). Steps that fail, repeat a call or make no progress go to a reflector.
5. Recover. A tool that the plan and your request both needed, but the model skipped, is run or requested once. "Let me try running it again" after an error gets one real retry.
6. Stop. After 10 steps or 120 seconds, CORTEX writes a best-effort answer from what it has. A run that stalls with more than 15 s left is first handed to LATS (tree search) for up to 60 s.
7. Stream and learn. The plan, steps, tool calls, results and answer tokens stream over Server-Sent Events. After the answer, a background job stores facts you stated about yourself (see Memory).

Quickstart

You need Ollama, Python 3.12 or newer, and Node.js 20.9 or newer for the UI. The two default models take about 5 GB of disk. A GPU with 6 GB of VRAM is recommended; CPU-only works, slowly.

git clone https://github.com/roydonsequeira/CORTEX-Private-Intelligence-Framework.git
cd CORTEX-Private-Intelligence-Framework
ollama pull qwen2.5:7b && ollama pull nomic-embed-text

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -e .

cortex doctor                      # checks Python, config, Ollama, models, data folders, port, HTTPS
cortex serve                       # API on http://localhost:8000

cd ui && npm install && npm run dev   # UI on http://localhost:3000 (second terminal)

The agent can read and write files only in workspace/, next to cortex.yaml. Put the documents you want it to work with there.

Docker Compose (NVIDIA GPU)

This needs Docker with the NVIDIA Container Toolkit, plus make, bash and curl.

make up            # API, UI, Ollama and Jaeger
make pull-models   # qwen2.5:7b and nomic-embed-text, into the Ollama container

The UI is on http://localhost:3000, the API docs on http://localhost:8000/docs and the Jaeger traces on http://localhost:16686. Every port is published on 127.0.0.1 only.

make demo and make pull-models-demo start a CPU-only stack that uses llama3.2:1b for everything. It shows the moving parts (planning, streaming, tool calls, memory, traces) on a machine without a GPU, but a 1B model's answers are much weaker. The first request is slow while the model loads.

What connects to what

• The UI talks only to the API. Its Content-Security-Policy allows connections to itself and the API origin, and no remote images.
• The API talks to Ollama (ollama_base_url) and, when telemetry_enabled is on, to an OpenTelemetry collector (otel_endpoint). Both default to localhost.
• web_fetch is the only tool with network access. The default Python sandbox can import only pure-computation modules, and the container sandbox runs with networking disabled.
• Next.js sends anonymous usage statistics about its build tooling (not your prompts) to Vercel unless you turn it off. The UI's Docker image turns it off; for native development, run npx next telemetry disable once in ui/.

Testing and evaluation

pytest                                   # 375 unit, integration and security tests; the model is mocked
pytest -m ollama                         # live tests against a running Ollama
ruff check src tests && mypy src tests   # lint and strict type checking

CI runs these on Python 3.12, 3.13 and 3.14 on Linux, plus the UI's lint, type check and build and both Docker image builds, on every pull request and every push to main.

Live battery. evals/live_battery/ drives a running server over SSE the way the UI does: 187 cases, from plain questions to prompt-injection attacks, against an isolated database and workspace. It's how the failures above were found: 27 bugs the unit tests had missed, each now fixed in code and pinned by a regression test. The last full run was on v1.2.1 (29 Sep 2026, qwen2.5:7b, RTX 3060 Laptop GPU with 6 GB). It scored 39/39 on the test plan, 136/137 on the extra prompts and 11/11 on the ops and security checks. The miss was a correct answer that took 58 s against a 45 s limit, and it passed three re-runs. v1.2.2 re-ran the memory cases; the battery's README has the results for each release.

Outside review. After launch, Mustafa ERBAY did a security review and filed five issues: memory learning from data the user asked to process (#54), quoting forms that still read as the user's own words (#55), procedural memory learning from untrusted runs (#56), a race between the filesystem path check and the open (#57), and a URL check that was too loose about scheme and host (#58). All five are fixed, #54 in v1.2.2 and the rest in v1.2.3. The regression cases are in tests/security/ and tests/unit/.

Not evaluated live. LATS and the supervisor mode have unit tests with a mocked model, but no live battery case targets them.

Security model

CORTEX assumes one user, on a machine they control. It has no user accounts and listens only on loopback by default. Within that scope it defends against two things: text the model reads (files, pages, pasted data) steering what it does, and code the model writes doing damage.

Network. The API binds to 127.0.0.1, accepts browser calls only from local origins (CORS), and answers only to local host names, which blocks DNS rebinding from web pages. The Docker stack publishes every port on 127.0.0.1. To expose the API, set api_key (for example with CORTEX_API_KEY) and widen api_host, allowed_hosts and cors_origins deliberately; cortex serve warns if you bind beyond loopback without a key. The key is then required as a bearer token on every route except /health and the docs. The UI can't send it, so to use CORTEX remotely, put an authenticating reverse proxy in front of both the API and the UI. Direct tool execution (POST /tools/{name}/execute) is off unless debug_tool_endpoint is set.

Code execution. The default restricted backend compiles each snippet with RestrictedPython and runs it in a spawned process with a 10 s timeout. Imports are limited to pure-computation modules. The attribute guard blocks dunder access and any traversal that returns a module object, which closes chains such as json → codecs → sys → sys.modules['os']. open() is read-only, confined to the workspace, text only and capped at 1 MB. This is a best-effort sandbox, not a security boundary. For code you don't trust, use the container backend (pip install -e '.[container]', then code_sandbox: container). It runs each snippet in a throwaway container with no network, a read-only root filesystem, all capabilities dropped, no-new-privileges, the nobody user, a 64 MB noexec tmpfs, and limits of 256 MB of memory, half a CPU and 128 processes.

Files. The tools can reach only workspace/ (set by allowed_root). They refuse .., absolute and system paths, and never touch hidden paths such as .git, .env, .venv or .cortex. CORTEX's own package and plugins folder stay read-only even if you widen allowed_root over them, because a plugin written there would run unsandboxed on the next start. Writes are limited to .txt, .md, .json, .csv and .py files of up to 512 KB. They never replace an existing file unless the request asked for it (overwrite: true), even if the file appears after the check. Files are opened without following any link swapped in after the path check, and a file with a second hard link is neither read nor replaced. There is no delete.

Untrusted text. Content from files, documents and web pages is treated as data, and the defenses are the guards above rather than prompt wording. Direct answers and refusals get no tool schemas, a step runs at most 3 validated, time-limited tool calls, pasted text can't request a file write, and procedural memory won't learn a tool that a file or page asked for. The two ways read text could carry data out, URLs and images, are closed in code. The known limit: pasted text with no quotes and no request in front of it still reads as your own words.

Web fetch. It opens only addresses you wrote in the conversation. The host must match as written, http may become https but never the reverse, and a trailing slash may differ. Loopback, private, link-local (cloud metadata) and reserved addresses are refused after DNS resolution and again on every redirect. It honors robots.txt, verifies TLS against the operating system's trust store, stops reading at 2 MB and doesn't download binary content such as PDFs or images. Known gap: the HTTP client resolves the host again when it connects, so a DNS-rebinding server could swap in a private address between the check and the connection (#41).

API. Requests are validated with Pydantic (chat messages are capped at 8,192 characters) and rate limited per IP with a token bucket: 200 requests a minute, or 60 for chat. New chat requests are refused while the server drains on shutdown. The calculator plugin walks an expression tree instead of calling eval, and bounds exponents and factorials, so 9**9**9 can't stall the server.

Report vulnerabilities privately, as described in SECURITY.md.

Memory

Each request replays the session's last 6 turns and retrieves related facts. After the answer, a background job extracts facts only from turns where you say something about yourself ("my name is…", "I prefer…"), and keeps a fact only if your words back it, so a name inside pasted data is never stored as yours. Facts are keyed by their content, so learning one again updates it. A tool sequence is saved only if every tool used after reading a file, document or page was one your request asked for. It's offered to the planner only for near-duplicate tasks (relevance of 0.6 or more).

Everything stays on disk under .cortex/. Inspect it with GET /memory/sessions and GET /memory/search?q=… (add types=procedural for tool sequences), or search it in the UI's memory panel. The log events procedural_pattern_saved, procedural_hints_used and working_memory_cleared show learning as it happens. cortex reset-memory --yes deletes it all.

Observability

Logs are JSON, with a request ID on every line; a well-formed X-Request-ID from the client is kept. With telemetry_enabled, OpenTelemetry spans cover HTTP requests, the kernel, the planner, executor steps and tool calls, the reflector, Ollama calls, each memory tier, both sandboxes, LATS and the supervisor, plus metrics for model latency and agent steps. The Docker stack sends them to Jaeger. The shipped cortex.yaml turns export off, since no collector runs in a native setup.

Configuration

Settings live in cortex.yaml and can be overridden with CORTEX_* environment variables, for example CORTEX_API_PORT=8001. CORTEX finds cortex.yaml by walking up from the current folder, or uses the file CORTEX_CONFIG points to. Relative paths in the file are resolved against the file's own folder. The settings you're most likely to change are the model names, allowed_root, code_sandbox, api_key and telemetry_enabled.

Writing a tool

A tool is a BaseTool subclass with a JSON schema and an async execute():

from typing import ClassVar
from cortex.tools.base import BaseTool, ToolResult, ToolSchema

class EchoTool(BaseTool):
    schema: ClassVar[ToolSchema] = ToolSchema(
        name="echo",
        description="Echo input text.",
        parameters={
            "type": "object",
            "properties": {"text": {"type": "string"}},
            "required": ["text"],
            "additionalProperties": False,
        },
    )

    async def execute(self, **kwargs: object) -> ToolResult:
        return ToolResult(
            tool_name=self.schema.name,
            success=True,
            output=str(kwargs["text"]),
            execution_time_ms=0.0,
        )

Put the file in src/cortex/tools/plugins/ (plugins_dir) and restart. At startup the registry imports every .py file there and registers each BaseTool subclass it finds. Plugins run inside the API process without a sandbox, so only add code you trust.

API

curl -X POST http://localhost:8000/chat/message \
  -H "Content-Type: application/json" \
  -d '{"message":"Tell me the time","session_id":null}'

POST /chat/stream runs the same request and streams it as Server-Sent Events (plan, step_start, tool_call, tool_result, token, done). POST /tasks queues a background run, with orchestration set to single, lats or supervisor; the supervisor splits the task, runs up to 3 workers in parallel and merges their answers. GET /tasks/{task_id} returns the result. The OpenAPI docs are at /docs, and DEMO.md has more examples.

Limitations

• One user, on one machine. There are no accounts, and the UI can't send an API key.
• Answer quality is capped by the local model. Long multi-step tasks can hit the 10-step or 120-second budget and end with a best-effort answer.
• The restricted sandbox is best effort. Use container for code you don't trust.
• Pasted text with no quotes and no request in front of it reads as your own words, so it can still ask for a file write.
• web_fetch has a DNS-rebinding window between its address check and the connection (#41).
• LATS and the supervisor mode haven't been evaluated against a live model.
• CI runs on Linux, and the live battery on Windows 11. macOS is untested.

Roadmap

Open work is tracked in issues:

• Hybrid retrieval (BM25 and vectors) with reranking and citations (#40)
• Skip near-duplicate memory facts (#39)
• Connect web_fetch to the address it checked (#41)
• Show learned tool sequences in the UI's memory panel (#38)
• A "New chat" button (#37)
• Support for OpenAI-compatible local model servers (LM Studio, llama.cpp, vLLM)

Further reading

• Architecture decisions: why not LangChain, why ChromaDB, why SSE, why LATS.
• Live battery: the cases, how to run them on isolated data, and the results for each release.
• CHANGELOG.md: every fix, with the failure that prompted it.
• 187 live prompts, 27 bugs: what testing against a real 7B model showed.
• My local AI agent remembered things I never said: the memory bug an outside review found, and the fix.

Contributing

See CONTRIBUTING.md and the Code of Conduct. New tool plugins are the easiest place to start.

License

MIT