JevMan: the Pac-Man benchmark for AI decision models

100 games per model against the classic ghosts.

=1 means tied: those scores are within each other's margin of error. Run by @opper-ai means we played the games through Opper. Community models were benchmarked by whoever submitted them, named under Run by: CI replayed every game they sent in to check its score, and the latency and cost are as they reported. Only decision models are ranked: they answer with odds for every option in one pass, without writing text. Chat models, regular LLMs that write their answer, and rules that use no model at all show at their score for reference, marked Chat or Base (baseline) instead of a rank. Claude Haiku 5.5, which we ran ourselves, was too slow to show: two thirds of its answers came too late, so backup moves played instead.

Games

Each model played 100 games against the classic scripted ghosts, each until it lost all three lives. Games are capped at 5 minutes so every run ends, but none came close.

Decisions

At every junction the game asks the model one question, with the maze, pellets and ghosts as state. The model returns a probability per direction, and Pac-Man takes its pick.

Deadline

An answer that takes longer than 2 seconds is replaced by a simple backup rule, counted under backup moves.

Scores

The ranking uses the mean score with a 95% margin of error (±2 standard errors), and models within each other's margin are tied. The high score is a model's best single game.

Checks

Every game is recorded and replays exactly, so any result can be checked.