Snake: the benchmark for AI decision models

100 games per model on the same seeded boards, in real time at a step every 275 ms: the mean number of apples eaten.

=1 means tied: those means are within each other's margin of error (two standard errors). Run by @opper-ai means we played the games through Opper. Late steps are the share of steps the model didn't answer in time, so the fallback moved the snake. Self-reported models were benchmarked by whoever submitted them, named under Run by: CI replayed every game they sent in to check its score, and the timing (late steps, latency) and cost are as they reported. Baselines are simple scripts, shown for reference and never ranked.

Your apples when you race an AI. The server replays every game from the board it dealt before it takes a score.

The benchmark is open source, and any model behind an HTTP endpoint can play it: a hosted model, a fine-tune or one running on your laptop.

1. 01

   Put your model behind an endpoint

   At every step the game sends the board as JSON and asks which way the head goes next: up, right, down or left (never back the way it came). The repo has a short example endpoint to start from.

2. 02

   Run the benchmark

   One command plays the games under the leaderboard's rules: the same seeded boards on the same 275 ms clock, and a simple fallback move for every step your model is late on.

3. 03

   Send a pull request

   CI replays every game you submit and checks its score. Your model then joins the leaderboard as self-reported.

Board

16 by 16 cells with walls around it. The snake starts 3 long in the middle, heading right, with one apple on the board. Apples come from a seeded random sequence, so the same moves always make the same game.

Decisions

Every step the model gets the board as text and picks the direction the head moves next: up, right, down or left, never back the way it came. Every direction is offered, a deadly one too. Each option states facts: what is in the next cell, the distance to the apple after the move, and how much room is left from there; a move into the wall or the snake says the game ends. None is labelled safe or best.

Clock

The game runs in real time: the snake steps every 275 ms (about 3.6 steps a second), the same speed for every model. Each step gives the model 275 ms, plus any time left from the previous step: once its answer for the coming step is in, the next question goes out at once. An answer that misses its step's tick, or isn't one of the options, makes a late step: a fallback moves straight unless that is deadly, else left, else right. It never steers toward the apple. So the score measures good decisions made in time.

End

Into a wall or the snake's own body, 150 steps without an apple, or 1000 steps. The score is the apples eaten.

Scores

The ranking uses the mean over 100 fixed seeds, with a 95% margin of error (two standard errors); models within each other's margin are tied. Every game is recorded and replays exactly, so any result can be checked.