A reasonably sized software project is rarely delivered in time and under budget. This is a known fact in the software industry. While I can’t remember the exact figure, I read somewhere that only about 10% of all large software projects were done within planned time, and an even smaller percentage of projects were done under budget. Anyway, my point is that why it is so hard to give good estimates?
Software projects are drastically different than many other projects (e.g. construction project) in that software development is a very abstract and complex process. While good software architects can foresee many aspects in different phases of software development, any seemingly small issues that are ignored can later on come back and become major problems.
Estimating certain projects can be done fairly easy and with great confidence. For example, how long would it take you to get from here to say a place 40 miles away? Well, suppose you drive and it is a mix of city roads and highway, it is reasonable to suggest that it might take around an hour. And depending on the traffic, the end result might be slightly different in most cases, but should be very close to our original estimation of an hour.
To estimate even a relatively simple software project however, can be tricky. How long would it take you to create a simple application, say notepad? Well, the application itself might not be all that difficult, but the requirement is not very clear just from this statement. For example, what is the largest file this notepad can read? Does it read everything into memory at once or does it read only chunks of data at once? While it is reading the file, does it show what it has read or does it show the contents only till the whole file has been read?… Even just after the first couple of questions, you can see that depending on what is required, the development time for this simple notepad application can take from a couple of hours to maybe a couple of weeks even months.
In the real world, business applications are far more complex than the simple notepad example I gave. The requirement documents can be huge and most likely will miss some key information that can only be discovered in later development cycle. These missing requirements might turn out to contradict some of the existing ones and thus require changes in design… So even if the original estimates are relevant, after adding in all the missing requirements and making all the necessary adjustments we all of sudden realize that there’s a larger portion of the iceberg hidden underwater. And most business applications require at least a couple of such iterations, no matter how carefully it was planed out first.
Human factor is another crucial one. Most people at workplace tend to underestimate the problem they face. Why? No one wants to be seen as being pessimistic. Everyone wants to show his or her boss that he or she is capable. How long do you think it will take you to code this spec? “At most a couple of days” is often the answer. While one might be able to get the basic functionalities done within a couple of days, getting the whole thing done might take weeks. As a real world example, in one of the places I worked, a guy was asked to give an estimate on a “small” project. He estimated two weeks. The boss was pleased. Sure enough, after two weeks, he showed the skeleton of the “almost” functioning “product”. And the result was…? It took him another four months to finish the product, with certain functionalities taken out… Alas, had he estimated a couple of months rather than a couple of weeks, that project might never get funded to carry out, because the gains were so little compared to the cost. And the reason why he was allowed to take on that project was the rather insignificant estimate of … just two weeks. But somehow, the whole company got sucked in. Because at the end of the two weeks he promised, the application looked so “close” to be finished. And it was “reasonable” to ask for some “small extensions”…
In the real world, this kind of story happens way too often unfortunately. For smaller projects, delays often mean more resources, and more money. For larger projects, delays can be deadly. Precious market timing might turn a delayed project into a competitor’s trophy…
It seems that everyone knows how easy it is to underestimate software development effort, so why it is still such a huge problem everywhere? Maybe managers need to think harder, and look at those estimated hours again. And maybe being too optimistic is a bad thing in software development…
