ScreenForge

中文 | English

Describe what to test. Watch it happen. Get a pytest script.

ScreenForge is an AI-driven UI automation engine that turns natural language into executable test scripts. Unlike record-and-replay tools, you don't perform the actions yourself — the AI does it for you.

Why ScreenForge?

Core architecture: Your AI Agent is the brain (understands requirements, makes decisions). ScreenForge is the hands (executes UI actions, generates code).

Quick Start

pip install screenforge

# See the magic without any API key:
screenforge --demo

# For real usage, set your LLM key:
export OPENAI_API_KEY=sk-...

# Inspect the current page (returns DOM tree for your Agent to analyze):
echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin

# Execute a single action:
screenforge --action click --platform web --locator-type text --locator-value "Login"

How It Works

You (or your AI Agent)          ScreenForge
        │                            │
        ├──── "Test the login" ─────►│
        │                            ├── inspect_ui (get DOM tree)
        │◄── DOM tree ──────────────┤
        │                            │
        ├──── click #email ─────────►│
        ├──── input "user@..." ─────►│
        ├──── click "Sign In" ──────►│
        │                            │
        │◄── pytest script ─────────┤
        │◄── Allure report ─────────┤

Each step: inspect → decide → act → verify. The AI decides, ScreenForge executes.

Features

• Cross-platform: Android (uiautomator2), iOS (wda), Web (Playwright)
• Self-healing engine: When tests break due to UI changes, the engine auto-repairs locators with confidence scoring and AST validation
• L1/L2 semantic cache: Same page + same instruction = instant response, no LLM call needed
• Visual fallback: When DOM can't locate elements (Canvas, games), VLM parses screenshots
• MCP server: Any MCP-compatible Agent can drive ScreenForge natively
• Structured output: JSON Lines events + report/runs/<id>/ artifacts for CI integration

Agent Integration (Claude Code / Cursor / Codex)

ScreenForge exposes itself as a tool for AI Agents. The standard loop:

# 1. Get page structure (your Agent analyzes it)
echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin

# 2. Your Agent decides what to do, sends precise actions
screenforge --action click --platform web --locator-type text --locator-value "Login"

# 3. Verify the result, repeat
echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin

For batch operations, use workflows:

screenforge --workflow ./workflows/login.yaml --platform web --json

Or start the MCP server for native Agent integration:

screenforge --mcp-server

See Agent Integration Guide for the complete protocol.

Installation (from source)

git clone https://github.com/jhinzzz/ScreenForge.git
cd ScreenForge
python -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"

Platform-specific extras:

• Web only: pip install -e . (default, includes Playwright)
• Android: pip install -e ".[android]"
• iOS: pip install -e ".[ios]"
• ML/cache: pip install -e ".[ml]" (sentence-transformers for semantic cache)

Configuration

# Required: LLM API key (OpenAI-compatible endpoint)
export OPENAI_API_KEY=sk-...

# Optional: custom endpoint (defaults to api.openai.com)
export OPENAI_BASE_URL=https://api.openai.com/v1

# Optional: model (defaults to gpt-4o)
export MODEL_NAME=gpt-4o

Or create a .env file (copy from .env_template).

Learn More

Contributing

See CONTRIBUTING.md for guidelines. Issues and PRs welcome!

License

MIT