The UX is awful - I bailed out at 25/100 JUST IN LEVEL ONE (BASICS)
Might I suggest adaptive difficulty? After getting 10, 15, 20 correct in a row it should scale up the difficulty immediately, rather than waiting for 100 in the basic level 1...
We noticed this years ago when looking at -- IIRC -- ikaros antibodies. They were clearly faked. Lacking any sort of platform to gain attention we moved on to Abcam and our lab just sort of maintained a mental map of who not to purchase ANYTHING immuno- from.
Is this substantially different, cognitively or skills-wise, than moving into management and directing a team to write code, but no longer writing code oneself?
I disagree and think GP poster is right on the money.
There are many ways in which inflation numbers are cooked; just one of them is the hedonic adjustment [1].
Others include an un-representative basket of goods.
The basket of goods is adjusted every 2 years, but not necessarily in a way that mirrors the way real households adapt their spending patterns to increased prices.
Owner equivalent rent (LOL) massively lags behind home prices.
Honestly, when 10s or hundreds of millions of people's perception does not match *Official Government Numbers*, then it's reason to suspect that the official numbers are a poor metric.
I love that you are doing this test. However, as it purports to be a test of "English-to-SQL", your hardest question (Q9) seems ungrammatical:
> Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory,
gross profit, and margin percentage for each product subcategory.
In particular, the clause "in the subcategory, gross profit, and margin percentage for each product subcategory" is ambiguous, and I wonder if more models would pass if the English were reformulated to be correct.
(it's also notable that Claude Opus 4.6 and Sonnet 4.6 both "missed" this one)
When I was a young(er) postdoc and had to overhaul my bicycle -- my main transportation to work-- this site was invaluable. Forever grateful to Sheldon.
I've found the same. I've moved from a homebrew tinkerer's delight home network with pfSense running in a VM, Supermicro FreeBSD ZFS fileserver, hodgepodge of cameras etc. to a top-to-bottom Unifi Stack.
My time is worth a lot more now, my family appreciates very much that it "Just Works" (tm), and -- most importantly -- the cognitive load for me is a fraction of before.
Might I suggest adaptive difficulty? After getting 10, 15, 20 correct in a row it should scale up the difficulty immediately, rather than waiting for 100 in the basic level 1...