#benchmarks - natmc

Wednesday, October 28, 20261 item

X DEBRIEF

WEDNESDAY, OCTOBER 28 - 1 post · tagged: #picpoker #AI #benchmarks

ARGON IS THE CLEAR WINNER

I posted the Pic.Poker benchmark scoreboard: 17 identical photos, nine cards each - 153 cards read per model, against human-confirmed answers. My summary line: Argon is the clear winner.

The table backs it. Gemini 4 Argon read 95.4% of cards correctly, got all nine cards right in 64.7% of photos, and ran a 21-second median at about 4 cents a photo. Second place, GPT-6.1 Sol, read 90.8% of cards but grouped all nine correctly only 47.1% of the time, at 76 seconds. The all-nine-right column is the separator: no other model clears half. Cost spreads wide too - GPT-6 Astra ran about 26 cents a photo, while GPT-6 Luna was the cheapest at three-tenths of a cent and got zero photos fully right. At the bottom, Gemini 3.8 Flash read 5.9% of cards and failed 16 of 17 photos.

Akul replied that empirically he finds Gemini does really well with most vision tasks, including computer use. One reply, no likes yet at collection.

Monday, October 26, 20261 item

X DEBRIEF

MONDAY, OCTOBER 26 - 4 posts · tagged: #picpoker #poker #AI #benchmarks

THE DIFFERENCES ARE WILD

At 8:52pm I posted the first public description of Pic.Poker, an app I am building to analyze poker hands from a photo. The first challenge is reading the cards at all: overlapping hands, awkward angles, cards scattered across a real table. So I built a benchmark - identical photos, human-confirmed answers, scored on accuracy, time and cost.

The results across the top 10 models are spread wide. Card accuracy runs from 64% to 98%. Getting all 9 cards correctly grouped ranges from 0% to 85%. Median time is 21 to 76 seconds per photo, and estimated cost runs 0.29 cents to 26.28 cents per photo. Same 13 photos, shared prompts and validation, failed reads scored as zero. Small sample, huge differences.

Earlier, Jason L. Baptiste posted that he had not felt a chill like this from a product he worked on since the Onswipe demo days with Andres Barreto. I asked which product. His whole answer: "You know." I answered with a GIF. The product stayed unnamed.