Really? If a company advertises a new red version of their widget and I excitedly upgrade because I love red, but when it comes it's gray just like the old widget, don't I have a case? Surely I don't need to demonstrate that red makes me more productive.
That makes sense when tools are as dumb as static notes and TI-84s.
But in the (hypothetical) limit where AI tools outperform all humans, what does this updated test look like? Are we even testing the humans at that point?