Coding, Fast and Slow: Developers and the Psychology of Overconfidence

“We should just be more careful at the specification stage”

Coding, Fast and Slow: Developers and the Psychology of Overconfidence

"… If you do “fully understand” something, you’ve got a library or existing piece of software that does that thing, and you’re not writing anything. Otherwise, there is uncertainty, and it will often blow up. And those blow ups can take anywhere from one day to one year to beyond the heat death of the universe to resolve.
…
When you first hit this pain, you think “We should just be more careful at the specification stage”. But this turns out to fail, badly. Why? The core reason is that, as you can see from the examples above, if you were to write a specification in such detail that it would capture those issues, you’d be writing the software. And there is really just no way around this. (if, as you read this, you’re trying to bargain this one away, I have to tell you — there is really really really no way around this. Full specifications are a terrible economic idea. Some ways below I’m going to lay out better economic choices)
…

In Thinking Fast and Slow, Kahneman explains a great deal of psychology as the interplay between two “systems” which govern our thoughts: System I and System II. My far-too-brief summary would be “System II does careful, rational, analytical thinking, and System I does quick, heuristic, pattern matching thinking”.

And, crucially, it’s as if evolution designed the whole thing with a key goal of keeping System II from having to do too much. Which makes plenty of sense from an evolutionary perspective — System II is slow as molasses, and incredibly costly, it should only be deployed in very, very rare situations. But you see the problem, no doubt: without thinking, how does your mind know when to invoke System II? From this perspective, many of the various “cognitive biases” of psychology make sense as elegant engineering solutions to a brutal real-world problem: how to apportion attention in real time.

…

The real trouble here is the interplay between the two sources of estimation error: the human bias towards overconfidence, and the inherent uncertainty involved in any real software project. That uncertainty is severe enough that even the careful, rational SystemII is unable to come up with accurate predictions.
…"