screenshot-to-qml

An agent skill that turns a UI screenshot or HTML page into a production Qt Quick app,
then renders the result and proves it matches.

Install · Usage · How it works · Results · Evals · Changelog

_{A screenshot of a 12.3" instrument cluster, rebuilt as a Qt 5.15 C++ app by the skill. Every frame after the wipe is grabbed from the running QML (example).}

Most "image to UI" prompts stop at looks about right. screenshot-to-qml gives a coding
agent (Claude Code, Codex CLI and others) a measured workflow instead: it reads exact colors,
spacing, fonts and icons off the screenshot, builds a tokenized QML project, renders it
headlessly at the screenshot's size and DPR, and scores it against the source until it passes.
The result is native Qt: no browser, no WebView, from desktops down to a Raspberry Pi panel or
a car's instrument cluster on Qt 5.15.

Features

• Measured, not guessed. Colors, edges, borders and padding are probed from the pixels and
  stored in a design-spec.json, each value with its source and confidence. Given HTML (plain
  CSS or Tailwind), the page is rendered in Chromium and the exact values, text runs, icon SVGs
  and font files come from the DOM.
• Matched, not eyeballed. The font family and weight of every text run (63 open UI
  families), the icon set and name of every icon (11,869 icons from Lucide, Tabler, Font
  Awesome and Heroicons), CSS shadows and linear gradients are fitted from the image. A font
  that is not in the catalog is flagged instead of silently swapped.
• Production QML. One Theme singleton holds every token. 14 QtQuick.Templates controls
  cover hover, pressed, focus, checked, error and disabled states. Repeats come from models,
  every string is qsTr()-ready, and the code passes qmllint and qmlformat.
• Verified. check.py reports global and per-region SSIM, CIEDE2000 color error, geometry
  offsets and the position of every string, and runs 13 quality gates, including keyboard
  reachability and visible focus driven by real input to the running window.
• One QML tree, two hosts. The same qml/ runs from PySide6 or compiles into a C++ binary
  with CMake (Qt 6.8+, QML compiled ahead of time). Both renders are compared.
• Qt 5.15 for automotive and industrial boards. --min-qt 5.15 writes the Qt 5.15 dialect
  (versioned imports, QtGraphicalEffects, no Qt 6-only types) and a Qt 5 C++ host. The check
  builds it against Qt 5.15, runs it and compares its frame with the reference render.
• Down to the board. Profiles for desktop, GPU boards (Raspberry Pi, Jetson) and GPU-less
  panels (software renderer), with kiosk packaging: eglfs/linuxfb, systemd unit, installer.

Install

The skill is a plain Agent Skills folder:
skills/screenshot-to-qml.

Any agent: npx skills

npx skills add ozaysahin/screenshot-to-qml                                # pick agents interactively
npx skills add ozaysahin/screenshot-to-qml -g -a claude-code -a codex      # user-level, Claude Code + Codex

The skills CLI installs into the folder each agent reads (Claude Code,
Codex, Cursor, Gemini CLI, GitHub Copilot and many more). Without -g it installs into the
current project.

Claude Code plugin

/plugin marketplace add ozaysahin/screenshot-to-qml
/plugin install screenshot-to-qml@screenshot-to-qml

On Claude Code 2.1.275 or later, one line does both:
/plugin install screenshot-to-qml --marketplace ozaysahin/screenshot-to-qml.

Manual

git clone https://github.com/ozaysahin/screenshot-to-qml
cp -r screenshot-to-qml/skills/screenshot-to-qml ~/.claude/skills/       # Claude Code, all projects
cp -r screenshot-to-qml/skills/screenshot-to-qml ~/.agents/skills/       # Codex CLI

Any agent that can read files and run Python can also be pointed at the folder: "Read
skills/screenshot-to-qml/SKILL.md and follow it."

Dependencies

The agent installs these when it first runs the skill (SKILL.md has a Setup section). To do it
yourself:

pip install -r skills/screenshot-to-qml/scripts/requirements.txt   # PySide6 6.8+, numpy, opencv, scikit-image …
python skills/screenshot-to-qml/scripts/font_match.py fetch         # font catalog, ~40 MB from google/fonts
python skills/screenshot-to-qml/scripts/icon_match.py fetch         # icon index, ~170 MB (needs npm)

On Windows, set PYTHONUTF8=1 before running the scripts.

Usage

Give the agent a screenshot (or an HTML file) and say what you need:

Rebuild this screen as a PySide6 Qt Quick app for a 1920×1080 kiosk.

Turn dashboard.html into a C++ Qt app for a 7" Raspberry Pi 5 panel.

Rebuild settings.png (exported at 2×) as a desktop app with English and Turkish translations.

Rebuild cluster.png as a Qt 5.15 instrument cluster in C++ for our i.MX 8 board.

The agent analyzes the image, asks only what it cannot measure (for example a font that is not
in the catalog), builds the project, and iterates on the worst regions until the check passes.
It works to a budget: a simple screen takes about 60 tool calls and up to three checks, and the
checks hand it one contact sheet of the worst spots instead of letting it crop and probe pixels
by hand (v1.0.1 cut the tokens a settings screen costs by about two thirds).
You get:

project/
├── main.py                  PySide6 host (fonts, translations, frame grab)
├── CMakeLists.txt, src/     optional C++ host for the same QML
├── qml/App/
│   ├── Main.qml             Window root, screen composition
│   ├── *.qml                components (cards, charts, panels …)
│   ├── Style/Theme.qml      every color, font, radius and spacing token
│   └── Controls/            Template-based controls with all their states
├── assets/fonts, icons      bundled fonts and SVG icons with their licenses
├── i18n/                    translations
└── README.md                assumptions, run instructions, fidelity report

To check a project yourself:

python skills/screenshot-to-qml/scripts/check.py project --source input.png --spec design-spec.json

How it works

flowchart LR
  A[Screenshot or HTML] --> B[Analyze + match<br/>palette · surfaces · OCR<br/>fonts · icons · shadows · gradients]
  B --> C[design-spec.json]
  C --> D[Scaffold + Theme]
  D --> E[QML components]
  E --> F[check.py<br/>render · diff · gates]
  F -->|fail: fix the worst regions| E
  F -->|pass| G[Project + fidelity report]
  G --> H[deploy.py<br/>desktop bundle · board service]

Quality gates. A build fails on hard-coded colors, QML warnings, web engines, screenshots
pasted in as backgrounds, controls that Tab cannot reach or that show no focus or pressed
state, missing accessible names, untranslatable strings, effects the target cannot draw,
images or animations an embedded board cannot afford, and a C++ build that fails or draws a
different frame.

Reference results

An industrial HMI (examples/hmi-dolum), a
1920×1080 screenshot rebuilt in production mode:

The instrument cluster in the GIF (examples/auto-cluster),
1920×720, Qt 5.15 with the C++ host: global SSIM 0.999, worst region 0.983, ΔE 0.0, geometry
≤ 1.1 px, 61 of 61 strings, all 13 gates clean. The binary built against Qt 5.15.13 runs with no
QML warnings and draws the same frame as the reference render (max 32-px tile difference 14.7 of
24 allowed). Its Dial component (one Shape, generated ticks and labels, a needle bound to
value) drives both the speedometer and the tachometer.

Control kit

Every new project gets 14 Template-based controls styled only through Theme roles. When the
design does not show hover, pressed, focus or danger colors, they are derived from its own
tokens. The gallery ships in English with
a Turkish translation and passes all 13 gates.

Results

Twenty original eval screens (dashboards, HMIs, a kiosk, forms, phone screens, dense tables,
two HTML pages and two hard cases), each built once by an agent that sees only its input and
the skill, scored against ground truth extracted from the live DOM. (Case 21, the Qt 5.15
instrument cluster, was added in v1.1; agent runs on it come next.)
Full results.

Skill v1.0 with Claude Opus, all 20 cases:

16 of 20 builds meet every criterion at once. Under the stricter calibrated thresholds (global
SSIM ≥ 0.98, every string within 2 px, all 13 gates) 12 of 20 pass outright, and each miss is
one to three items: a dashed chip, a 28 px badge, a dash typed as a hyphen, a value 3–4 px off.

Against the same model without the skill (cases 01–08):

A strong model already copies the pixels well, and on these eight it passed the visual checks
more often without the skill (one build per case, so ±1 case is noise). What the skill adds is
everything a real codebase needs around the pixels: design tokens, keyboard and screen-reader
support, clean lint, translatable text and a verified fidelity report.

Other models and agents (shared cases):

The production rules carry over to every model and agent tested. Visual fidelity does not
yet: Haiku leaves more text 2–4 px off, and Codex CLI got the HTML case right but missed the
three screenshot cases on fonts and text position, on a machine whose font catalog was most
likely incomplete (details).

The analyzers and matchers were tuned on cases 01–08; cases 09–20 are unseen screens for them.

Case 20's font is outside the catalog by design and is flagged. The weak spot is shadows on busy
or dark backdrops: in the glassmorphism case only 4 of 13 are found. Per-case numbers are in
evals/matching-results.md and
evals/synthetic-results.md.

Eval set

All twenty-one screens are original designs with fictional brands, built from HTML so that every
color, box, string and font is known exactly. evals/ holds the cases, the
builder, the scorer, and a harness that hands any agent the same folders and prompt
(prepare_run.py, run_agent.py), so you can rerun the comparison with your own agent or
model.

Platform support

Limitations

• Qt Quick's Rectangle has no dashed border; dashed outlines are approximated and show in
  region scores.
• Shadows on busy or dark backdrops (glass, photos) are often missed by the fitter.
• Fonts outside the 63-family catalog are flagged, not identified: the agent asks for the font
  or names the closest substitute.
• One build per case per arm: differences of about one case between arms are within noise.

Repository layout

skills/screenshot-to-qml/   the installable skill: SKILL.md, references/, scripts/, assets/, examples/
.claude-plugin/             Claude Code plugin marketplace manifest
evals/                      21 cases, case builder, scorers, agent harness, results
tests/                      pytest suite: scripts, schema, control kit, gates, renders, C++ builds
docs/                       README images; docs/tools draws the eval and matcher sheets and the demo GIF

Roadmap

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md: run ruff check .
and pytest before a PR, and build new eval cases with evals/tools/build_case.py.

License

MIT for the code, docs and eval designs. Bundled fonts are under the SIL Open Font License
1.1; bundled icons come from Lucide (ISC), Tabler Icons (MIT) and Font Awesome Free (icons
CC BY 4.0, fonts OFL 1.1). License texts are in evals/assets/licenses.
The icon matcher also downloads Heroicons (MIT) into its cache; it is not bundled.