Jev Fighter: the fighting-game benchmark for AI models

104 matches per model against four fixed-rule baselines: the share of rounds it wins.

=1 means tied: those means are within each other's margin of error (two standard errors). Late moves are the share of the model's moves the fallback made, because its answer took over 2 s (or was unusable), so its fighter blocked. Run by @opper-ai means we played the games through Opper. Self-reported models were benchmarked by whoever submitted them, named under Run by: CI replayed every game they sent in to check its score, and the timing (late moves, latency) and cost are as they reported. Baselines are simple scripts, shown for reference and never ranked.

The benchmark is open source, and any model behind an HTTP endpoint can play it: a hosted model, a fine-tune or one running on your laptop.

1. 01

   Put your model behind an endpoint

   Every time its fighter is free to act, the game sends the situation as JSON and asks for one of the legal moves. The repo has a short example endpoint to start from.

2. 02

   Run the benchmark

   One command plays the matches under the leaderboard's rules: the same fixed seeds, live in real time, on the raw track, against all four baselines from both sides.

3. 03

   Send a pull request

   CI replays every match you submit, recomputing every baseline move, and checks its score. Your model then joins the leaderboard as self-reported.

The fight

Two stick figures, best of 3 rounds of 60 seconds. A knockout wins a round, otherwise the fighter with more health when time runs out. Strikes beat throws, throws beat blocks, blocks beat strikes.

Decisions

Every time its fighter is about to be free to act, the model gets the situation, seen as it was a moment ago (about human reaction time), and its legal moves in shuffled order, and picks one. A late, missing or unusable answer makes the fighter block: the game never plays well on a model's behalf.

Opponents

Four fixed-rule baselines. Greedy takes the biggest swing if the opponent stands still or repeats itself, payoff picks the move with the best average, maximin the safest one, and random any offered move (from the match's seed). They think for 250 ms, as when you fight them here, and work out each move's payoffs from the fight itself.

Live, raw

The fight runs in real time at its normal speed, as when you fight a model here, and never waits. The model is asked ahead, its answer plays as soon as it arrives and its fighter can act, and an answer that takes over 2 s is dropped: its fighter blocks. Each option is just the move's own description (damage, speed, reach), so the model has to work out what wins, in time.

Games

104 matches on fixed seeds: 13 fights, each against all four baselines and from both sides. Every match is recorded and replays exactly, so any result can be checked.

Score

The share of rounds the model wins, a drawn round counting half: the mean over its 104 matches, with a 95% margin of error (two standard errors). Models within each other's margin are tied.