Optimism - Curse of Big IT Projects and How to Manage It

Most of us who have worked on IT projects feel the resonance of Hofstadter's Law:

"It always takes longer than you expect, even if you take Hofstadter's Law into account." Douglas Hofstadter

When we estimate we don't have a clear view of all of the nuances that will have to be handled. And the nature of IT projects is that every unforeseen nuance will have to be handled eventually and will cost time. Optimism can be useful in inspiring motivation and getting a project going. But optimistic simplification can be dangerous as it blinds us to risks of under-estimation and can lead us to pursue ideas that just don't work when fleshed out.

Let's study the kinds of projects that display the dangers of optimistic simplification most clearly. When we understand the logic of optimistic simplification then we'll be able to make recommendations on how to better manage our relationship with optimism. To get there we'll first have to look at how optimism can work against us.

Optimistic simplification can be especially dangerous on big cost-saving projects. Many people can tell you about a big IT project that was meant to save a lot of money and instead wasted a vast budget. There are some very famous examples out there:

-  In the UK there was the National Health Service systems unification project. It was meant to bring all the patient records across surgeries and hospitals under a common system. It ran for 10 years with a budget of £11bn before being abandoned.

- In Canada there was the government payroll unification project (Phoenix). Its purpose was to bring public sector pay under a single system. When it went live many workers were not paid and it became a national scandal. Costs to fix the system were much higher than hoped with the project eventually costing around $3bn.

- In the US the LA Unified School District project aimed to provide every student and teacher with an iPad that would give everyone access to the same education content. But the software was painful to use and the content it presented was inadequate. The project cost $1.3bn and most of the schools under the program abandoned trying to use it.

Failed government projects run by private contractors get the most public attention (and costs in the billions) but there are also examples at private companies. Big Digital Transformation projects have similar aims of shifting a whole department or company to a new and more efficient way of working. These have been failing in a big way. It has been estimated that in 2019 $1.3 trillion was spent on digital transformation and that as much as $900bn went on failed projects. That includes respected companies like Ford, GE and Proctor & Gamble.

These mega-projects have especially high failure rates, much higher than smaller projects (Standish CHAOS Report 2015):

It's not just massive organisations either. From my experience many much less famous companies also have stories of a "let's just make it all one thing" project that went badly wrong. Huge cost-saving IT unification projects have a special risk of going badly wrong. We can learn from these failures as I think they embody extreme cases of a logic of optimistic simplification that we see in other IT projects too.

Let’s consider a simplified example to bring out the logic of optimistic simplification. Imagine that a consulting firm approaches the manager of a vineyard and says:

(Image from https://www.mdpi.com/2073-4395/10/1/102)

Anyone who knows wine is likely to be sceptical right away. Different parts of the vineyard have different growing conditions and different types of grape are better suited to them. Different parts of the vineyard are different environments. Unless the vineyard is very uniform you can’t use the same grape variety everywhere and expect good results across the board. The consulting firm’s suggestion is therefore likely to be rejected and never get off the ground.

Now imagine that instead of approaching the vineyard manager, the consulting firm approaches an investment firm with a controlling stake in the vineyard. If the investment firm aren’t knowledgeable about wine then they might not see the nuances and might instead focus on the money to be ‘saved’ by the unification project, which for them would translate into profits and share value.

If this doesn’t seem plausible to you then substitute the vineyard for a farm.

(Picture from Images From Finchley.)

The consulting firm might look at the types of crop being grown and propose to ‘rationalise’ and bring under a single system for planting, growing and picking. The farmer would point out that different crops suit different conditions and have different growing seasons. But if the farm is run as an agri-business with multiple levels of management then it’s plausible that the grand unification project pitch might get a more receptive audience as it is pitched higher in the management chain. The more complex the organisation is, the more levels it will have and the more difficult it will be to command a detailed picture of how it operates.

The key point here is that it takes skill and effort to see differences — it can be difficult even for specialists to see all the nuances. Non-specialists are more likely to be taken in by a picture that suggests various grape or crop types are all basically the same.

Non-specialist susceptibility to simplification applies for crops and grapes or documents and record-keeping processes and ways of doing payroll. If there are non-specialist managers who are incentivised to find cost savings, then they will be even more likely to see all the grapes as basically the same. We humans have a tendency to see what we want to see. And the risk of seeing what we want to see can get amplified by a tendency to report up the management chain what we think our superiors want to hear.

The many famous examples of failed big unification projects were not so doomed as the simplified crops or wine examples. What is common to these kinds of projects though is a logic of optimistic simplification. Nuances are passed over in initial planning. Problems are then amplified because the project is unprepared for them. The project tries to pass over the nuances rather than properly address how the new system will handle them. Then you get resistance from stakeholders, politics and increasingly entrenched differences.

A dangerous kind of optimism can come into play even when the aim isn't cost-saving or rationalisation. Big tech projects involve lots of moving parts and are hard to estimate. Stakeholders who are less familiar with the details and/or more invested in the high-level aims can struggle to appreciate the risks involved. The pattern of thinking has been called been given various names, including 'the technological sublime', 'techno-idealism' and 'techno-optimism'. It is most prominent in mega-projects but can emerge in many types of project. If we better understand how it operates then we can better compensate for it.

The driver to focus on is our tendency to assimilate different cases and treat them as ‘basically the same’. Sometimes grand unification projects can succeed because the different cases really can be assimilated or ‘rationalised’. Or business processes can themselves be changed to fit the new approach (which often has advantages and disadvantages).

(Image from reddit.)

Sometimes differences can be assimilated, sometimes they cannot and often they can be assimilated but only by accepting trade-offs. It is hard to determine in advance whether rationalisation will succeed. There is a tendency in large organisations for non-specialists to have most influence over this kind of decision-making. Often senior managers who aren’t afforded much time to pick through the nuances. Starting from a position with too much wishful thinking will mean the trade-offs and costs involved will effectively be unplanned.

This logic of optimistic simplification can lead to projects that are doomed from the beginning. What makes it worse is that optimistic reporting up the chain and a desire to realise the promised benefits can mean that good money continues to be thrown after bad. This can create situations where different sub-leads compete to report more optimistically and expand their own mini-empires within the project. Even when it becomes recognised that there are serious problems, a culture of excessive optimism can lead to chasing false saviours and desperate rounds of restructuring. Then good money is thrown after bad and costs spiral.

There are of course other factors in play in the failure of big IT unification projects. The role of private consulting firms can get attention as these firms can have a vested commercial interest in either underestimating or prolonging the project. But note that the core simplification logic can apply when the pitch and delivery is in-house. Government management chains can be a factor but the temptation towards optimistic simplification is not specific to government. The temptation to oversimplify is deep in us and our cultures, especially in how we run large organisations.

Senior management can't possibly know everything that is going on in the organisation. They need to make decisions based on key considerations, which results in a simplified narrative. Centralised governance lends itself to optimistic simplification - which is why these big failures keep happening.

Big IT projects in general have a significant failure rate and can fail in various ways. Some of the simplification logic discussed can operate on projects that don’t specifically aim at ‘unification’ or ‘rationalisation’ — there is a spectrum. Big unification projects are a special case with a special danger due to their size, complexity and apparent promise of such huge cost savings. Too much wishful thinking in chasing cost savings can sometimes lead to enormous calamity.

The problem of optimistic simplification may be structural but that doesn't mean we can't do anything to fix it.

Much hangs on the project's attitude to change. If plans are all made upfront and budgets are allocated for specific purposes then it can be hard to change. Using pilot projects to explore approaches before further committing can be an effective way to be more flexible. But you really do have to be prepared for some approaches to fail. A lesson of Agile is that lots of small failures that you recover from are much better than one very big failure.

A key element in how the project approaches change is the messaging of senior management. It matters that communications of project direction are presented in an open-minded way. A tone like "here's the central focus now" communicates that there may be other considerations not explained fully and that direction can change if problems emerge. A tone like "we've got a great plan, let's do this" may sound motivational but it also sends a very different message about openness to change. This affects how middle management behaves, what risks will get raised and how they'll be phrased.

It can also be important for the signalling of tone that it's not only outright successes that get talked about. If a project hits problems it's good to talk about those. If a project has to abandon one of its initiatives because it wasn't working out then this can be an opportunity to highlight what was learnt. That encourages future problems to get the attention they need.

We've seen that specialists are less susceptible to simplification than non-specialists. So one technique for limiting the dangerous side of optimism is to ensure that specialists have a strong voice in the central governance. One might also aim to decentralise some of the governance by giving sub-units some independence (provided that doesn't lead to conflict).

There are also techniques that can bring risks to the fore and encourage collaborative discussion. Pre-mortems, for example, are exercises where a team imagines that a project has failed and works backwards to imagine what would have been the most likely cause. This helps to break groupthink and encourage a different attitude towards risk.

These recommendations are of course just pointers. But the sum of the various practices like these can have a big impact on the culture of a project. How optimism figures in a project is principally a matter of direction and culture.

Big IT projects may carry large risks but they also have the potential for transformative benefits. The Apollo Program, for example, has been estimated as returning $7 in economic benefits for every dollar spent on it. Great things can be accomplished with the right combinations of skill, patience, determination and inspiration.

(Title image by Trainiac on flickr.)

If you liked this you can vote for me for Hacker Noon Contributor of the Year - Project Management