People against Jev: what 7,500 games of a picture quiz show
By Jose Sabater -
Jev Cards is a picture quiz: you look at a card, answer a question about it, and Jev answers the same card next to you. We posted it as a Show HN, it went to #1, and in the first 20 hours people finished 7,571 games: about 69,000 answers from people and 91,000 from Jev.
That is a lot of people answering the same questions as a model, on the same pictures, with the same buttons. So here is what it shows.
What we found, in short:
- Jev won 77% of games. It got 74% of cards right against 64% for people, and answered in a median 0.33 s. People took 4.1 s.
- Recognising is solved, reasoning is not. Jev is right on 95% to 100% of nature, objects, food and animal cards. On music it is right 25% of the time, on chess 31%, on logic 47%. People beat it on all three.
- The rules decide who wins. In a speed race people won 4.5% of games. In a game of right answers in a row, a third of the time they went further than Jev.
- It cost under $9 for the 102,000 calls of the launch, at an average of 284 ms per call.
What Jev is
Jev, from TypeSafe, is a decision model. It does not write text. You give it a state and a question with a fixed set of options, and it returns one of them, with a probability for every option. Jev 1.14 can also take a picture as part of the state, which is what this game uses.
Each card is one call through Opper, with the model string typesafe/jev-1.14.0: the picture, one line on how the card is laid out ("four fruits on a white background in a two by two grid"), the question, and the options, shuffled and named option-1, option-2 and so on so their names cannot give the answer away. Jev never sees which option is right.
There are 490 cards in 21 themes: animals, flags, landmarks, maps, charts, art, colours, music, chess, illusions, counting, logic, science and more. Some are recognition ("tap the true berry"), some need reading or arithmetic on the picture ("what do these three darts score?"), and some are built to fool people ("which line is longer?").
Three games, three things to measure
A model can be better than people in more than one way. It can be faster, it can be right more often under pressure, or it can simply be right more often. We made one game for each:
- Sprint: speed. Ten cards, your total time, plus 15 seconds for every miss. Lowest total wins.
- Versus: right answers on a clock. Ten cards, 10 seconds each. Most right answers wins, and points (more for answering early) break a tie.
- Streak: right answers, nothing else. Answer until you miss. Longest run wins.
Live matches put Jev in a room with friends, as one more player.
The same model, against the same crowd, wins 97% of Sprints and fewer than half of Streaks. Only the scoring changed.
Sprint: you cannot beat it on speed
People took a median 50 seconds for ten cards. Jev took 3.6. Not one person in more than 3,200 Sprint games was faster than Jev before misses. Both missed about 3 cards a game, so a person only wins a Sprint when Jev misses noticeably more than they do, which happened in 4.5% of games.
The miss penalty started at 5 seconds and went to 15 during the launch, when it became clear that tapping at random beat thinking. With 15 seconds, misses make up 47% of a person's Sprint score. With 5, it was 23%, and nobody beat Jev at all.
Versus: closer, but Jev is right more often
On a 10-second clock, Jev's most common game was 8 right out of 10, a person's 7. People got more cards right than Jev in 22% of games and the same number in 15%. The clock ran out on 4.7% of people's cards.
Taking time does not save people here. They were right on 76% of the cards they answered within 2 seconds and 55% of those they answered near the limit. Partly that is the card, not the clock: the hard cards are the ones people stall on.
Streak: people own the long runs
Streak is the fairest test of who is right more often, and it is the one where people did best. Their average run was 2.5 cards, Jev's 2.7, and a person went further than Jev in a third of games.
The tail is all human. Jev's longest run was 18. People had 21 runs longer than that, 19 of them 20 or more. A single wrong answer ends a run, and Jev has whole kinds of card it reliably gets wrong, so it can never go very far. People make mistakes more evenly.
Where Jev is good, and where it is not
The split is clean. Jev is close to perfect when the answer is "what is this?": nature 100%, objects 97%, food 95%, animals 95%, landmarks and charts 91%. It falls apart when the picture has to be reasoned about: music 25% (people 55%), chess 31% (people 57%), logic 47% (people 66%).
Some single cards show it well:
- Four angles, one of them a real 90°. The real one is tilted; the others have an arm lying flat or upright and only look square. 99% of people picked the tilted one. Jev never did.
- Magnets. "Which pair pulls together?" People 89%, Jev 0%.
- Circuits. "Which bulb lights up?" People 87%, Jev 0%.
- The upside-down T. The classic illusion, where the upright line looks longer but the two are the same. On this card they are not: the flat one really is longer. People knew the trick, said "same length", and were right 6% of the time. Jev was right every time.
Tap the right angle. It is the tilted one; the upright ones only look square. People 99%, Jev 0%.
The cards that fool people most are illusions, and Jev gets most of them. The cards that fool Jev are the ones a person solves by counting, adding or following a rule.
Running it on Opper
The whole launch is the spike on the right: 102,272 calls in a week, 31 of them failed (25 at the provider), an average of 284 ms per call, and $8.98 in total. That is about $0.09 per 1,000 answers, or a tenth of a cent per game. Almost all of the 219 million tokens are input, the picture: about 2,100 tokens per call in and 50 out.
Caveats
- Players' times are measured in their browser and could be faked. We left out 122 answers from people that took under 0.3 seconds.
Jev Cards is now on Opper
After the launch, Jev Cards moved to Opper. It is at opper.ai/decision-benchmarks/jevcards, next to the other games where decision models play, and jevcards.com takes you there. The leaderboards and Jev's tally of wins came with it.
Version 2 adds:
- 300 new cards, about 14 more in every theme, so the deck now has 800. Every answer is checked in code where it can be (counts, routes, chess positions, notes), and they are global rather than US-centric.
- Top 100 leaderboards instead of top 10.
- Challenge a friend: share your result, and the link asks them to beat it.
Try Streak first: it is the one game where you have a real chance. The original version is open source at github.com/Jose-Sabater/jevcards.
Jev and the other decision models are on Opper behind one key. More of them at play on our decision benchmarks: JevMan, Snake and Jev Fighter.