Why estimations are doomed to fail

So you've been part of a software development team and want them to estimate? You gotta have some dates to tell business when they can expect new functionality?

The problem with estimations is not to come up with a number, the problem is that the number is not a proper reflection of how the reality turns out usually. Hence we base important business decisions on numbers we cannot trust. Unfortunately not many business people realize this, and it causes a lot of stress in organizations, even for very experienced developers and managers. The team leader gets frustrated because the developers could not get a proper estimate. The product manager gets frustrated because the team leader gave an incorrect number. And so the list goes on.

But why can't you give an accurate estimate? You have 10 years of experience for god sake!

A shopping mall example

Imagine you tell your partner that you'll go to the mall and shop some clothes. How long does this typically take? Some people will smile and say they don't like shopping, so that's a clear 1 hour max inside the mall and then transportation on top of that. Let's say 2 hours max in total.

Another person enjoys shopping, to look at all the people, go into different stores, have a chat here and there. Grab a coffee and enjoy the day. For this person the time inside the mall takes 5 hours and then transportation on top of that so 6 hours in total.

It doesn't matter which person goes to the mall this day, but one of them does. Also, they have a family event booked in the evening that they'll need to attend.

All of a sudden inside the mall there is a robbery. People are locked in and it's a terrible situation. Fortunately, the police can handle the situation quickly and people are only kept hostages for 4 hours which is not very long let's say.

However, we have a problem, we have now missed the beginning of our family event in the evening. Whoever went to the mall this day made it with the assumption that it would take 2 or 6 hours, and we did not have enough slack time before the event on the evening started.

The important part: even if you have been at the shopping mall 1 000 times during your lifetime, and you are absolutely 100% confident that on average it takes around 5 hours, you cannot possibly take into account that some stranger is going to rob the mall just this stupid day.

This is a relatively trivial example that most people can imagine, and it's also a good one. In the same way, even if you have 10 years of experience, how the hell are you going to take into account that a production error will not disturb you in the middle of everything? On top of that, how are you going to take into account that even after the production error is fixed, a colleague turns sick and you'll have to build it on your own? Or what about when you find another detail and realize that doubles the effort required to build the functionality?

So why continue estimate even if we know the estimate is not accurate?

Estimations can be very useful sometimes. Let's say a team spent 15 minutes to refine a story, and afterwards the team leader asks the team:

"How long do you think this will take to build?"

One responds with 4 hours, another responds with 2 days.

This is very useful, the two persons obviously have a very different perspective on what is required by us as a team to leverage business functionality X. So far, an estimate can be fine in my opinion, as long as the number never leaves that room.

When the problem starts

As I said, as long as the number never leaves that room, you're probably fine, since the discussion regarding 4 hours vs 2 days will probably only give you some input for a useful discussion to learn what to build. If you continue to refine and then bury the number to the ground then the estimation phase is fine. But that's not usually the case, because the leader will give this number to the manager with a bold understatement that this number is only an estimate so you cannot take it for granted. The manager hears it but have the same psychological issue that we all do: when we see something we cannot unsee it. It's pretty much the same thing as the example "don't think about a pink elephant". We as humans cannot ignore the fact that we have seen a number previously. So this is really not a leadership problem either per se. It's just human psychology.

So the manager puts it into an excel file and then render the below picture:

Here goes the planning that we all now know is not a very suitable approach towards software development. It looks neat even though most business people and software development teams knows the reality never turns out to be so simple. Usually story 1, 2, 3 and 4 are not finished on time. Bad software development teams! Give us accurate estimations!!

But you gotta improve, right?

Most organization honor improvements and continuous education or so. We learn new programming languages, leaders take leadership courses etc. We all improve. Slowly slowly, but at least we improve. So why can't estimations improve?

I have heard countless colleagues say something like this:

But if we made a better analysis phase, then we would estimate better

And you know what? They're probably right. If we analysed the problem more in-depth we would probably know the problem better, and know what to build more in detail, giving us a good foundation to make better estimations. But what if we do all this, and our estimations still turned out to be wrong? Certainly better estimations will make you feel better. But feelings are not necessarily a successful metrics when deciding how an organization should work.

The problem with estimations is not that people lack the necessary skills to make good estimations, the problem is that software development is a way too complex task as a whole. No matter how good we are, we'll always be wrong from time to time. And this is where I think it starts to hurt: those times when you are wrong, will business accept it or not? I.e. will they accept the fact that we were wrong, or will they say that:

Well, we have already signed contracts with dealers, so we must go-live on this date. Of course, we can postpone the release, but it will cost of a great amount of money. Can you try make it to this day?

Developers only pay attention to the word "try" and then they say:

I don't think this will work out well, but sure, I'll try

The business person only hear what he/she wants to hear, and silently notes:

Alright, that's all I needed to hear, we'll keep the go-live date then

The developers continue to work, take shortcuts (this is where technical debts increases and the vicious cycle of unmaintainable legacy code starts), and delivers, once again, on time. Victory! From the outside - the program managers perspective, we succeeded once again. In the short run, we did manage it on time. But in the long run, we failed badly. The reason why we failed badly is because these kinds of shortcuts we as developers take is what causes messy code to grow over time up util either the company can evolve too slowly and other companies instead will take market share, or we will have to rewrite the systems so they're more adaptable to the current situation. (For some reason here I still find it a bit odd that to completely rewrite a system is generally seen as reasonable and prioritized by business, but to slowly with a steady pace continuously refactor and improve the system on a week-by-week basis is unacceptable by business, because we have to much to do).

Are projects appropriate for software development?

So why bother estimate even if we know it'll fail? From my experience all organizations that loves "projects" and "project management" heavily also always love planning and dates. For all projects you gotta have dates for your charts (the agile waterfall chart). So this is inherently a problem built into the organization as it currently is. Usually, most people on leadership positions in these organization from my experience does not have a proper software development background (nor interest in improving in the area), but instead have a background in "project management" or some heavy financial background where they honor control over flexibility (numbers matter more than value delivered). So essentially, these organizations suffer from leadership problems. And the reason is simple: we all know that estimations will forever fail, and if you know something will fail then it is a question about solving the core problem, and this is not a developer problem, it's always a leadership problem.

A solution to the problem

If we by now agree that leadership is the problem in the organization, what can we do? Should we continue to blame estimations and planning but not provide an alternative? No of course not, that would not be very useful nor fun. We as developers must try inspire the business people to adapt to a workflow that is more suitable for software development.

Some (successful?) organizations instead seem to be using workflows such as OKR (Objective-Key-Result) for example.

The big difference from traditional planning methods? OKRs are frequently set, tracked, and re-evaluated – usually quarterly. OKR is a simple, fast-cadence process that engages each team’s perspective and creativity.

In a nutshell, it's a way to stop working such deadline-driven and talk less about the fact that story X must be delivered on week 6 and talk more about that "this is our organization goal for this quarter", "try to come up with good solutions so we can reach this goal, it's all up to you in the teams to figure out what we gotta build to reach our goals". If we finish things on week 3 or 11 is less important, as long as we've reach our business goal the first quarter. And this is absolutely fantastic for teams, because less pressure is put on accurate estimations (which will never be accurate) and more focus is put on delivering customer value.

Creating alignment in the organization is one of the main OKR benefits. The goal is to ensure everyone is going in the same direction, with clear priorities, in a constant rhythm.

Sounds absolutely lovely, right? I've worked some with OKR's and I must say I absolutely do love it. More responsibility is put it the hands of the team, people feel what they say matters, you can experiment way easier etc. All as long as you reach the goal, or strive to reach the goals. This is how software development should definitely be done, instead of the old waterfall planning process with heavy load put onto estimations which is an absolutely perfect way to create a cascade of stress throughout the entire organization. You can read more about what OKR are here: What is OKR?

Summary

We now know that estimations as a concept is not bad per se, the problem is that they are far too often inaccurate. And this is problematic, because companies make very important business decisions based on these estimations.

Try stop estimate if you can, but keep delivering value to the organization and the customers, because that's what everyone cares about in the end.