Laya Plays Google Snake

Real Google Snake. Real pixels. Laya at the controls.

Demo

This project plays the Google Snake browser game from screen captures and arrow-key presses. It does not implement a Snake clone, read the game's internal state or DOM, or use a vision-language model. A deterministic visual tracker turns pixels into a board state; Laya chooses among safe directions; the controller times physical keyboard input and checks whether the game accepted it.

visible Google Snake → screen pixels → board/head tracking → safe choices
                                                      ↓
                                                Laya probabilities
                                                      ↓
                                        timed keypress → visual acknowledgment

The optional floating overlay shows the model's four move probabilities, the selected turn, decision time, visual confidence, and keypress status. It runs separately from the capture loop.

Why this is different

• The environment is not ours. We control Google's existing browser game, not a simulator with a privileged get_state() call. The simulator in this repo is used only for training and repeatable evaluation.
• Pixels are the only observation. Checkerboard colors locate the grid; blue and red identify snake and food; white eye components identify the head. Consecutive frames reconstruct movement and body order. No YOLO, VLM, browser DOM, or hidden game variables are used.
• Laya actually decides. The adapted typed-decision model scores UP, RIGHT, DOWN, and LEFT. Unsafe moves are removed before the model is asked; narrow emergency cases can override or bypass it. The log and overlay distinguish Laya's proposal from the executed key.
• Timing is part of the problem. A correct direction can still fail if a browser input arrives after the cell's turn window. The controller buffers predicted turns, waits for visual acknowledgment, retries a missed turn only while safe, and fails closed when tracking is uncertain.

This is an experimental research/demo controller, not a promise of a particular score or a fully general computer-vision agent. It currently targets the classic green/blue/red Google Snake theme on Windows.

Try it

Requirements: Windows, Python 3.11 or 3.12, uv, and a normal visible browser window. The tested accelerated setup uses an NVIDIA GPU and the CUDA-enabled PyTorch build in uv.lock. CPU use is possible through --device cpu but has not been validated for live-game timing.

git clone https://github.com/Makankn/laya-plays-google-snake.git
cd laya-plays-google-snake
uv sync --extra laya

Download the Snake-adapted checkpoint from the v0.1.0 release (will be released soon :) ) and extract it so the model lives at models/snake-laya-balanced/. The Git repository intentionally does not contain the 1.3 GB weights. You can also train a local checkpoint using the command below.
The archive's SHA-256 is recorded in CHECKSUMS.txt.

First verify perception without sending keys:

uv run snake-state live --planner-only --duration 15

Each run opens a draggable, resizable capture box. Fit it around the game board and press Enter. Then bring the Snake browser window to the foreground; the whole board must stay visible. The timer starts only once a readable board is seen. JSONL telemetry is saved under captures/.

When the reported state matches the visible board, enable Laya and keyboard control:

uv run --extra laya snake-state live --model models/snake-laya-balanced --device cuda --duration 120 --execute --overlay

--execute is deliberately opt-in. Without it, the controller observes and decides but does not press keys. The overlay can be dragged or closed independently; it starts outside the selected capture region when screen space permits.

If you see waiting_for_board, check that the browser—not the terminal—is visible inside the selected box. If the game is in an in-app browser surface that Windows screen capture cannot see, open it in a normal browser window instead.

Model and evaluation

The checkpoint starts from the Apache-2.0 Laya multilingual model and adapts its decision layers on balanced Snake states generated by the included simulator. The encoder remains frozen. This is supervised adaptation, not end-to-end reinforcement learning from live Google Snake screenshots. See MODEL_CARD.md for provenance, limits, and release details.

Reproduce the local training/evaluation workflow:

uv run --extra laya snake-state train-laya --device cuda --output models/snake-laya-balanced
uv run --extra laya snake-state eval-laya --model models/snake-laya-balanced --device cuda --episodes 10 --max-steps 160 --seed 10000 --validation-samples 0

Simulation scores are not live Google Snake scores: the simulator does not reproduce browser focus loss, capture occlusion, sprite-animation ambiguity, or late keypresses perfectly. The included --input-delay-cells 1 evaluation option stress-tests one-cell-late turns, but live runs remain the real test.

Explore the pipeline

# Parse one screenshot into detected geometry, snake cells, food, and eye hint.
uv run snake-state parse path/to/frame.png --pretty

# Reconstruct ordered body and direction from consecutive frames.
uv run snake-state track frame-001.png frame-002.png frame-003.png --pretty

# Run deterministic tests without downloading model weights.
uv run --extra dev pytest -q

Core modules: parser → tracker → Laya policy and safety layer → live controller. The overlay only tails JSONL telemetry and never participates in move selection.

Attribution and license

Project code is Apache-2.0; see LICENSE. Laya and the upstream model are by Convai Innovations and are also published under Apache-2.0; see NOTICE and the upstream model card. Google Snake belongs to Google; this independent project is not affiliated with or endorsed by Google.