Before you go, check out these stories!

0
Hackernoon logoWhy Big IT Unification Projects Famously Fail by@ryandawsonuk

Why Big IT Unification Projects Famously Fail

Author profile picture

@ryandawsonukRyan Dawson

Engineer at seldon.io. Trying to keep getting better at making software.

Many people can tell you about a big IT project that was meant to save a lot of money and instead wasted a vast budget. There are some very famous examples out there:

-  in the UK there was the National Health Service systems unification project. It was meant to bring all the patient records across surgeries and hospitals under a common system. It ran for 10 years with a budget of £11bn before being abandoned.

- In Canada there was the government payroll unification project (Phoenix). Its purpose was to bring public sector pay under a single system. When it went live many workers were not paid and it became a national scandal. Costs to fix the system were much higher than hoped with the project eventually costing around $3bn.

- In the US the LA Unified School District project aimed to provide every student and teacher with an iPad that would give everyone access to the same education content. But the software was painful to use and the content it presented was inadequate. The project cost $1.3bn and most of the schools under the program abandoned trying to use it.

Failed government projects run by private contractors get the most public attention (and costs in the billions) but there are also examples at private companies. Big Digital Transformation projects have similar aims of shifting a whole department or company to a new and more efficient way of working. These have been failing in a big way. It has been estimated that in 2019 $1.3 trillion was spent on digital transformation and about $900bn was wasted. That includes respected companies like Ford, GE and Proctor & Gamble.

These mega-projects have especially high failure rates, much higher than smaller projects (Standish CHAOS Report 2015):

It's not just massive organisations either. From my experience many much less famous companies also have a stories of a "let's just make it all one thing" project that went badly wrong. So what is it about these grand cost-saving IT unification projects that makes them so prone to go wrong so spectacularly?

An Example

Let’s consider a simplified example. Imagine that a consulting firm approaches the manager of a vineyard and says:

“We’ve looked at all the types of vines you’re buying and the different ways you’re planting and tending them. We can save you X by rationalising the processes, using a smaller number of vine types and bringing it all under a computer system that will tell workers when to do what.”

(Image from https://www.mdpi.com/2073-4395/10/1/102)

Anyone who knows wine is likely to be sceptical right away. Different parts of the vineyard have different growing conditions and different types of grape are better suited to them. Different parts of the vineyard are different environments. Unless the vineyard is very uniform you can’t use the same grape variety everywhere and expect good results across the board. The consulting firm’s suggestion is therefore likely to be rejected and never get off the ground.

Now imagine that instead of approaching the vineyard manager, the consulting firm approaches an investment firm with a controlling stake in the vineyard. If the investment firm aren’t knowledgeable about wine then they might not see the nuances and might instead focus on the money to be ‘saved’ by the unification project, which for them would translate into profits and share value.

If this doesn’t seem plausible to you then substitute the vineyard for a farm.

(Picture from Images From Finchley.)

The consulting firm might look at the types of crop being grown and propose to ‘rationalise’ and bring under a single system for planting, growing and picking. The farmer would point out that different crops suit different conditions and have different growing seasons. But if the farm is run as an agri-business with multiple levels of management then it’s plausible that the grand unification project pitch might get a more receptive audience as it is pitched higher in the management chain. The more complex the organisation is, the more levels it will have and the more difficult it will be to command a detailed picture of how it operates.

The key point here is that it takes skill and effort to see differences — it can require specialists to see all the nuances. Non-specialists are more likely to be taken in by a picture that suggests various grape or crop types are all basically the same.

Non-specialist susceptibility to simplification applies for crops and grapes or documents and record-keeping processes and ways of doing payroll. If there are non-specialist managers who are incentivised to find cost savings, then they will be even more likely to see all the grapes as basically the same. We humans have a tendency to see what we want to see. And the tendency to see what we want to see can get amplified by a tendency to want to report up the management chain what we think our superiors want to hear.

The many famous examples of failed big unification projects were not so doomed as the simplified crops or wine examples. What is common to these kinds of projects though is a logic of optimistic simplification. Nuances are passed over in initial planning. Problems of rationalisation are then amplified because the project is unprepared for them. The project tries to pass over the nuances rather than properly address how the new system will handle them. Then you get resistance from stakeholders, politics and increasingly entrenched differences.

Optimistic Simplification

The driver to focus on is our tendency to assimilate different cases and treat them as ‘basically the same’. Sometimes grand unification projects can succeed because the different cases really can be assimilated or ‘rationalised’. Or business processes can themselves be changed to fit the new approach (which often has advantages and disadvantages).

(Image from reddit.)

Sometimes differences can be assimilated, sometimes they cannot and often they can be assimilated but only by accepting some trade-offs. It is hard to determine in advance whether rationalisation will succeed. There is a tendency in large organisations for non-specialists to have most influence over this kind of decision-making. Often senior managers who aren’t afforded much time to pick through the nuances. Starting from a position with too much wishful thinking will mean the trade-offs and costs involved will effectively be unplanned.

This logic of optimistic simplification can lead to projects that are doomed from the beginning. What makes it worse is that optimistic reporting up the chain and a desire to realise the promised benefits can mean that good money continues to be thrown after bad. This can create situations where different sub-leads compete to report more optimistically and expand their own mini-empires within the project. Even when it becomes recognised that there are serious problems, a culture of excessive optimism can lead to chasing false saviours and desperate rounds of restructuring. Then good money is thrown after bad and costs spiral.

Complexity is Part of Life

There are of course other factors in play in the failure of big IT unification projects. The role of private consulting firms as having a vested interest in prolonging the project can get attention. But note that the core simplification logic can apply when the pitch and delivery is in-house. Government management chains can be a factor but the temptation towards optimistic simplification is not specific to government. The temptation to oversimplify is deep in us and our cultures, especially in how we run large organisations.

Big IT projects in general have a significant failure rate and can fail in various ways. Some of the simplification logic discussed can operate on projects that don’t specifically aim at ‘unification’ or ‘rationalisation’ — there is a spectrum. Big unification projects in particular are a special case with a special danger due to their size, complexity and apparent promise of such huge cost savings. Too much wishful thinking in chasing cost savings can sometimes lead to enormous calamity.

(Title image by Trainiac on flickr.)

Tags

The Noonification banner

Subscribe to get your daily round-up of top tech stories!