For software engineers, this scenario is all-too familiar…
Partner: Hey can we build this feature?
Dev: Yeah that should be easy.
…3 weeks later…
Partner: Hey where’s that feature?
Dev: Turns out there was a bit more to it than we thought.
Why is this so common? When it comes to software estimation, why do we consistently underestimate how long something will take, and more importantly, why do we continue to do so even when we should’ve learned our lesson? Well, it’s more complicated than you think. *winks insufferably*
I’m far from the first person to observe this, or even write about it, but it bears bringing up again because it continues to happen regardless. Perhaps I am particularly sensitive to this kind of thing because growing up, my dad would repeat the mantra ad nauseam: “Everything takes longer than you think”. For a long time I assumed this was something he had made up—until one day at work, when I repeated “his” phrase, a colleague asked me, “is your dad Hofstadter?”. Hofstadter’s Law states:
“It always takes longer than you expect, even when you take into account Hofstadter’s law.”
Turns out Doug is a fellow software guy (read: computer scientist), but this phenomenon extends well beyond the software world. Projects across industries (and day-to-day life) are notoriously perpetually behind schedule: construction, academics, government, public works, cooking weeknight dinner, etc… It’s a universal truth.
There are a few psychological phenomena at play here. Inherent in Hofstadter’s law is the planning fallacy, which underlines that humans get it wrong despite knowledge that previous tasks have generally taken longer than planned. Inherent in the planning fallacy is the optimism bias, which leads us to that optimistically wrong conclusion. But why? Because humans feel the need to present ourselves as competent at all times, and because we are just plain bad at estimating.
We strive to be perceived favorably in the eyes of others. This is especially true when there’s more at stake, such as keeping clients happy, or perhaps winning a bid against a competing firm. Obviously a business is more likely to get the gig if they say they can do it faster and cheaper than the other guy. Basically, lowering the bar for permission now because it’s easier to ask for forgiveness later. Perhaps part of the problem is that we all inherently know this is how things work. We’re not all that surprised when deadlines aren’t met. It’s almost as if we subconsciously expect a 25% “optimism tax”. So if for once we come up with a realistic estimate, the person on the receiving end is mentally including their own optimism tax and thinking “wow it’s really going to take that long?” So we decide it’s better to maintain the status quo by underestimating upfront to save face, and then, surprising nobody, it ends up taking longer than initially thought.
But what if there isn’t a bid on the line? What if it’s just a casual conversation in Slack? I’d argue that the knee-jerk impulse to say “that’ll be easy” is stronger than ever in these situations! It really boils down to the same thing: we want to look smart and capable and on top of our shit. What will the Product Manager think if the answer to every bug that comes up is “I don’t know, we’ll have to look into it”? What will our Partner think if our answer to build a seemingly-simple “interdimensional user portal” is “that could be months of work, maybe more.” We think we’re doing ourselves a favor by answering with what we think they want to hear. And maybe, just maybe, our gut impulse is right. Maybe it does end up being a “quick fix”. But the reality is that 9 times out of 10 that is not the case. The reality is that every time we give in and say “it’ll be easy”, we are digging a deeper hole for ourselves. Not only does it set us up for failure in that one instance, but it creates a snowball effect when combined with the last 9 times we said it’d be “easy” only to find out that was not the case.
Aside from wanting to “look good”, humans are bad at estimating, especially when it comes to time. After all, entire books have been written on the subject! Of course organizations are well aware of this fault, which is why systems exist to combat it. One example is “Best Value Procurement” or BVP. BVP is often used by government agencies as a means to determine the best bid by looking at many different factors in addition to the cost and time estimates, including a bidder’s track record of hitting deadlines or staying under budget. An example in the software world would include the Lean Agile practice of “playing poker”. This technique intentionally avoids time-based estimations in favor of gauging general “complexity”. For example, If a feature request comes in, the request is first documented in a clearly defined ticket with specific “Acceptance Criteria”, meaning step-by-step requirements for the request to be complete. Based on this ticket, the team comes together to “play poker” by discussing the details, and then secretly submitting a complexity score of 1, 2, 3, 5, 8 or 13, from easiest to hardest (the first 6 numbers of the Fibonacci Sequence). The scores are only revealed after everyone has submitted their number, which produces more “pure” estimates that are untainted by others’ opinions. Then the team discusses their reasoning for their respective scores, before ultimately coming to an agreement and assigning a final score to the ticket. This allows the engineer that is ultimately assigned the ticket to have some idea of how complicated it’s going to be at the outset, without the added pressure of getting it done in a prescribed amount of time.
In software development we are uniquely susceptible to Hoftsadder’s Law for several reasons, one of which is the knowledge gap between technical and non-technical people. Not to sound like a “Big Fancy Nerd” (to borrow a phrase from a coworker), but software is complicated, and most people don’t even have a frame of reference to gauge that complexity. Imagine two friends, Bob and Alice, are standing on a sidewalk in a residential neighborhood looking at a house. Alice asks Bob “how long do you think it took to build that house?” Now, Bob may know nothing about construction, but he’s seen houses before. He’s likely seen houses under construction before. Hell, he may have even lived in a house! He thinks, “Well, it looks pretty big. It’s two stories, it’s got a bunch of rooms, each one of those rooms needs electrical, plumbing, insulation… yeah that probably took a long time!”. Now imagine the same two friends browsing The Web.
Wow, over-the-shouder web browsing, what an antiquated concept!
Bob asks Alice, “How long do you think it took to build this website?”. All Alice sees is a page with some text, buttons, images, and maybe a video. “I don’t know, an afternoon?” she answers. The reality is that the simple interface Alice sees is a façade hiding multiple layers consisting of many different pieces of software, written by many different engineers (or LLMs!), all interacting and cooperating so Alice can “like” a post or download an audiobook. Ironically, the emphasis on developing intuitive user interfaces only exacerbates this misunderstanding by further abstracting away all the complexity. All this to say, it’s hard for non-technical folk to understand why on earth adding a single button might take an entire week. Because of that, engineers often feel the pressure to match our own estimations to others’ expectations, even when deep down we know it’s wrong! At the same time, this knowledge should come with a balance. We don’t do ourselves any favors by fixating on how difficult something will be, as it can have a paralyzing effect. Not to mention coming across like “we software engineers are so smart and you couldn’t possibly understand the inner machinations of our code.” (Big Fancy Nerd Syndrome).
But when it comes to underestimating time, there’s also another, less obvious drawback. Oftentimes when we’ve already cursed ourselves with a promise for a “quick fix”, we are tempted to take shortcuts. Maybe taking a shortcut will let us get it done today, but it comes at a cost. In software, this falls into the “Technical Debt” category, meaning: some code might be “good enough” for now because we want to move fast, but eventually we’ll have to come back and deal with it because it’s not sustainable. In other words, if we take out too many “loans” there will come a time when the code is so in debt that we can’t afford to build anything new. The thing about software is that every feature we build does not exist in a vacuum. Every piece of code has the inherent added complexity of considering how someone (or something) might come along and build on top of it. If we build a thing with rubber bands and Scotch tape, it might get done quickly but it’s not going to support much weight.
So how do we break this habit? We have to be honest with ourselves that estimating time accurately is hard. If I haven’t convinced you at this point, I have a challenge: next time you nonchalantly offer an ETA, pay close attention! How long does it actually end up taking? How much longer did it take than you expected? Maybe you were pretty close, but how many shortcuts did you take? It’s not something we’re likely to get right on a guess. It’s a skill in and of itself that takes intentional effort to hone. We need to stick to the processes we have established for this exact purpose such as “playing poker” or Best Value Procurement. It’s those one-off side conversations, like when a partner DMs you to ask a “quick favor”, in which we most often dig holes for ourselves. I still catch myself saying “yeah, that should be easy, no problem”, even when nobody asked!