THE DIFFERENCES ARE WILD
At 8:52pm I posted the first public description of Pic.Poker, an app I am building to analyze poker hands from a photo. The first challenge is reading the cards at all: overlapping hands, awkward angles, cards scattered across a real table. So I built a benchmark - identical photos, human-confirmed answers, scored on accuracy, time and cost.
The results across the top 10 models are spread wide. Card accuracy runs from 64% to 98%. Getting all 9 cards correctly grouped ranges from 0% to 85%. Median time is 21 to 76 seconds per photo, and estimated cost runs 0.29 cents to 26.28 cents per photo. Same 13 photos, shared prompts and validation, failed reads scored as zero. Small sample, huge differences.
Earlier, Jason L. Baptiste posted that he had not felt a chill like this from a product he worked on since the Onswipe demo days with Andres Barreto. I asked which product. His whole answer: "You know." I answered with a GIF. The product stayed unnamed.