Reading another article in this series, "Can You Boil an Egg Too Long?" [1] really made me smile. Apparently no one knows exactly what happens if you boil an egg for multiple months or years. This seems such a trivial thing compared to all the other stuff humans have discovered. On the other hand this also means almost anyone can expand the limits of human knowledge: you just need an egg, a reliable source of heat and water, and lots of patience. Granted, the knowledge gained may not change the world, but you will still be the first who is in possession of that knowledge!
In a linked blog post [0] and on the Rust performance page [1] the performance metric "instructions" is used. What exactly is meant by that? Number of instructions executed?
> But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers.
Hmm, too me it feels like I can explain the reasons why I came to a conclusion in many (but certainly not all) cases. You "just" need to clearly identify your feelings and emotions and separate them from your rational arguments.
Anyway, these are our own shortcomings and of course don't have to be adopted by any artificially built black-box model.
I was more thinking about higher level reasoning. But yes, a lot of the lower level stuff just appears as thoughts in my mind seemingly out of nowhere.
> You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.
Well, I guess it depends on how self-aware a person is. I think the biggest danger is trying to rationally explain your decision when in fact it was based mostly on your feelings, in which case I agree that the explanation is "basically a lie". One needs to be honest when something is not based on a fact but on a feeling to prevent pointless discussions. (If I hold an opinion based on a feeling then you cannot convince me that I am wrong by giving me facts.)
> You can already do that. Just change that number in the input and see how the output changes.
Makes sense. But I guess transparent models would still be generally preferable because you can fully understand how the output is produced, whereas in black-box models you might have to ask quite a lot of questions to get a feeling for it, but even then you can't be sure that you have a full understanding of it.
> I think it's important to note that human pattern recognition is basically black-box as well.
Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)
So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".
I'm curious, could you elaborate?