DEV Community

Cover image for Congrats to the Kaggle Benchmarking Challenge Winners!

Congrats to the Kaggle Benchmarking Challenge Winners!

Jem on November 05, 2026

A few weeks ago we teamed up with Kaggle to ask a deceptively simple question: how do you actually know whether a model is any good at the work you...
Collapse
 
fm profile image
Fayaz • • Edited

Congrats to the winners. 🏆🎉

This is one of the few DEV challenges that I regret not being able to join myself. Not enough time!
I'd still love to go through the submissions and complete the challenge on my own pace, when I get time 🙏

So, congrats to everyone who participated! 🥰
And thanks to Kaggle and DEV for arranging such challenges.

Collapse
 
nazar-boyko profile image
Nazar Boyko •

Thanks a lot, DEV team and Kaggle! This was such a fun one to build. Congrats to everyone who took part! 🙌

@jonathansolvesstuff The Calgary time zone one made me laugh and then double-check my own assumptions 😄

Collapse
 
jonathansolvesstuff profile image
JonathanSolvesProblems •

Thanks a bunch Nazar, congrats on yours too! Clocks went back in a lot of places on Sunday, not Calgary, and all 19 models still have it on standard time 😎

Collapse
 
nazar-boyko profile image
Nazar Boyko •

At this point the models aren't wrong, they’re just loyal to old Calgary 😅😅😅

Thread Thread
 
jonathansolvesstuff profile image
JonathanSolvesProblems •

Ahahaha, true that!

Collapse
 
jonathansolvesstuff profile image
JonathanSolvesProblems •

Thanks a bunch DEV team and Kaggle, super appreciate it! Congrats to the other four winners too, really nice seeing everyone test the models against something real.

Collapse
 
xulingfeng profile image
xulingfeng •

Congrats All!

Collapse
 
huddlespith profile image
Amy Hudspith •

Congrats folks!

Collapse
 
georgekobaidze profile image
Giorgi Kobaidze •

Congrats to the winners! Great job, absolutely well deserved!👏

Collapse
 
varshithvhegde profile image
Varshith V Hegde •

Congratulations!!! to all the winners ....

Collapse
 
earlgreyhot1701d profile image
Earl Grey •

Congratulations to all the winners 🏆 👏 🙌

Collapse
 
artespraticas profile image
artespraticas •

Congrats to all winners; my pleasure to participate and learn!

Collapse
 
soumyadeepdey profile image
Soumyadeep Dey •

Nice Congratulations 🎉 😔 @heyitsjem

Collapse
 
nyaomaru profile image
nyaomaru •

Congrats to all the winners!! 😸🍾🎉

Collapse
 
paullovelace profile image
Paul Lovelace •

The Calgary one is my favourite — 19 out of 19 models getting the time zone wrong because Alberta stopped changing its clocks is exactly the kind of failure a leaderboard would never catch. And the Ariane 5 rebuild flagging SQL injection in code that never touches a database... that's the test-writing instinct I wish more evals had. Do any of the winners plan to keep their benchmarks running as new models ship, or was this a one-shot?

Collapse
 
sizzlebop profile image
Jessica Doering •

Congrats winners! Great job everyone!

Collapse
 
gracelee087 profile image
Sohee Lee • • Edited

Congrats to the Winners! 👏🎉

Collapse
 
balaji75 profile image
Chauhan Balaji •

congrats to all winners. 🎉

Collapse
 
dan52242644dan profile image
Dan •

Congratulations on your winning endeavors .😁

Collapse
 
ethan-carter profile image
Ethan Carter •

Congrats to the winners. 🏆🎉

Collapse
 
hemapriya_kanagala profile image
Hemapriya Kanagala •

Big congrats to all the winners 😀

Collapse
 
emilywoodsnyc profile image
Emily Woods •

Congrats to the winners! The Romanian and Turkish spelling example is the one I’d turn into a regression pair. Keep one input where the character needs fixing and one where that same character is already right. Score both together. It’s a small check, but it catches the kind of blanket correction that makes one language look better by quietly breaking the other.

Collapse
 
dafili profile image
Renhua Wei •

Add Turkish to that pair and the blanket pass breaks in both directions: "İ".lower() in the stock case mapper returns two codepoints (U+0069 U+0307), while the Turkish locale maps "I" to U+0131 - same letter, two answers depending on locale. The pair worth pinning is codepoint-level: Turkish ş is U+015F (s + U+0327 cedilla) and Romanian ș is U+0219 (s + U+0326 comma below), so any "fix the cedilla" pass rewrites one language's correct text while fixing the other's. Assert on the codepoints, not on len() or on NFC equality.

Collapse
 
emilywoodsnyc profile image
Emily Woods •

Thanks for spelling out the code points. I’d keep both correct spellings as controls and pin the locale for each test, then assert the exact expected characters. A visually plausible output could still be wrong here.

Thread Thread
 
dafili profile image
Renhua Wei •

Agreed on keeping both controls, with one caveat that burned an afternoon: in CPython pinning the locale does nothing, because str.lower() takes no locale argument. Measured on 3.11 here: 'İ'.lower() returns U+0069 U+0307 (len 2), and 'I'.lower() returns U+0069 under both LC_ALL=C and LC_ALL=tr_TR.UTF-8 — byte-identical output, so a test that pins locale and asserts U+0131 is green or red for reasons unrelated to the locale you think you set. Only assert U+0131 if the code path under test really goes through a locale-aware mapper (ICU, JS toLocaleLowerCase('tr')), and then assert on that mapper rather than on Python. The cheap structural guard is on length, since the blanket-correction bug shows up as an extra codepoint:

python3 -c "print([hex(ord(c)) for c in 'İ'.lower()])"          # ['0x69', '0x307']
LC_ALL=tr_TR.UTF-8 python3 -c "print('I'.lower() == '\u0131')"  # False on CPython
Enter fullscreen mode Exit fullscreen mode

I have not measured the ICU/JS side myself, so treat those codepoints as spec, not as my test result.

Thread Thread
 
emilywoodsnyc profile image
Emily Woods •

Thanks for checking this in CPython. I’d keep the Romanian and Turkish spelling controls at the code-point level and put the locale-sensitive casing assertion in a separate test for the mapper actually in use. That keeps the locale setting from giving us false confidence.

Collapse
 
jsb-securedme profile image
Jean-Sebastien Beaulieu •

congrat to all winner

Collapse
 
aitrainingjobs profile image
AITraining.jobs •

The line about every submission starting from something the author already knew was hard is also how the paid version of this work gets hired. On 7 Nov, 1,199 open roles on the job boards of AI training platforms asked people to create evaluation tasks, benchmarks or test cases, about 30% of the 4,035 roles whose full description could be read, and most of them sit with domain experts rather than generalists. Their median advertised rate was $65/hr, against $40/hr across all the roles read. (Disclosure: those counts come from AITraining.jobs, where I work, which reads the platforms' public boards; they are posted rates, not what anyone earned.)

The Calgary entry shows why the knowledge matters more than the harness: nobody writes that question without knowing Alberta dropped the clock change in June, and its answer key reruns itself from the time zone database.

A question for the winners: how much of your time went into finding the question every model got wrong, and how much into the runner around it? That split decides whether $65 an hour is a fair price for this kind of work.

Collapse
 
kursatyz profile image
Kürşat Yılmaz •

The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.

If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.

Collapse
 
peter7865 profile image
Peter •

Testing models against real mess instead of clean leaderboards is so real. Over at The Printing World, we see this when automated inventory systems try to guess real-world stock. Huge congrats to the winners!

Collapse
 
alexcodebytes profile image
Oleksandr •

Starting from a specific, known-hard problem is the best way to build a benchmark. The VeriFactu invoice ledger example alone shows how much more practical these are compared to generic leaderboards. Huge congratulations to all the winners!