Roger Sessions has published a white paper, “The IT Complexity Crisis: Danger and Opportunity” (PDF). It’s created a bit of a stir in tech circles, largely because Sessions estimates that “worldwide, we are already losing over USD 500 billion per month on IT failure, and the problem is getting worse” (page 1; emphasis in original). He feels that the consequence is a “coming IT meltdown”, then goes on to offer his own solution, namely designing simpler IT systems.

This naturally intrigued me, since for the last 15 years, I have been writing, consulting, lecturing, and testifying about troubled and failed IT projects. While there are indeed tremendous financial losses due to late and failed IT projects, the figures Sessions gives seem much too large to me, and so I decided to do this critique of his analysis.

Sessions is good enough to provide the basis of his estimates and calculations, including footnotes. But that’s where some of the problems start. For example,  on page 3, Sessions cites (his footnote ‘02′) to the US Budget, Fiscal Year 2009, Analytical Perspective (PDF), p. 169, for information on “at-risk” or failed IT projects, specifically:

• “According to the 2009 U.S. Budget [02], the failure rate is increasing at the rate of around 15% per year. If this trend continues, within another five years or so a total IT meltdown may be unavoidable.” (p. 3)
• “According to the 2009 U.S. Budget [02], 66% of all Federal IT dollars are invested in projects that are ‘at risk’. I assume this number is representative of the rest of the world.” (p. 3, in “Calculating the Cost of IT Failure” box)
• A large number of these ['at risk' projects] will eventually fail. I assume the failure of an ‘at risk’ project is between 50% and 80%. For this analysis, I’ll use the average: 65%.”

These three statements run into immediate problems. First, and relatively minor, Sessions gets his page number wrong: he’s citing “page 169″ of the Analytical Perspective document, but there is no discussion whatsoever on page 169 of that document about IT projects. However, page 157 of that document (which happens to be page 169 of the PDF document) does start a section titled “INTEGRATING SERVICES WITH INFORMATION TECHNOLOGY”, so I presume that Sessions made the simple mistake of using the PDF page count rather than the document’s actual page numbering.

Even so, serious problems remain with Sessions’ citations and analysis.

Page 157 of the Analytical Perspective document does not say what Sessions claimed in the two comments above. I have not been able to figure out where Sessions gets his figure for “the failure rate increasing around 15% per year” from the cited US Budget Analytical Perspective document, much less his conclusion that “if this trend continues, within another five years or so a total IT meltdown may be unavoidable.” As far as I can tell, the Analytical Perspective document does not talk about failed IT projects at all, much less the increase in failure rates.

Furthermore, the phrase “the failure rate increasing around 15% per year” is itself ambiguous and may not be that significant. To start with an arbitrary number, assume that 100 projects “fail” in a given year. If “the failure rate [is] increasing around 15% per year”, then that means that 115 projects would fail the next year, and 132 projects would fail the year after that. But unless we know both the actual number of failed IT projects and the total number of IT projects in that same year, Sessions’ figure tells us nothing. If there’s only 150 IT projects total, then the 15% failure rate increase becomes very significant; if there’s 1000 IT projects total, then we’re many years away from Sessions’ threatened “meltdown”.

Sessions also ignores or confuses the failure rate for new projects vs. the systems already deployed. In other words, the failure rate for new systems development says very little about the continued functionality of existing, deployed systems now in use. While there are occasions (most notably Y2k, now a decade behind us) where existing IT systems just won’t function or function properly if they aren’t fixed or replaced, by and large both governments and private concerns have gotten along remarkably well for years or even decades with antiquated systems

As for Sessions’ second statement, there is a table on page 158 that may represent the basis for it:

As can be seen in the FY 2009 column, 66% (535 out of 810) of the FY 2009 “Major IT Investments” are projects that are “Not Well Planned and Managed”. Note that this table does not (as Sessions infers) indicate Federal dollars but rather actual projects; that is, in FY 2009, there are 810 projects listed as “Major IT investments”, of which 535 are designated as “Not Well Planned and Managed”. The previous page appears to indicate that these projects represent $27 billion, which is roughly 38% of the proposed Federal IT budget — not a great figure, but still almost half of the 66% that Sessions claims.

What’s more, supplementary data (PDF) for the FY 2009 Analytical Perspective makes it clear that the US Government’s designation of such projects — which puts them on a “Management Watch List” (WML) — has reduced the risk of such projects during each fiscal year:

Note that in FY 2007 and 2008, the number of IT projects designated as “Not Well Planned and Managed” shrunk significantly during the year (from Q1 to Q4) without a proportional shrinkage of the overall number of major IT projects. In other word, it appears that the government’s efforts to remove such projects from the “Not Well Planned and Managed” category is relatively successful. And the actual US IT budget dollars at risk at the end of each of those fiscal years ($4.2 billion for FY 07, $8.6 billion for FY 08)  is a much smaller percentage (6.5% and 13%, respectively) of the Federal IT budget for each of those years ($64.2 billion for FY 07 (XLS), $66.4 billion for FY 08 (XLS)).

Sessions then states that “I assume this number [66% of all Federal IT dollars being at risk] is representative of the rest of the world.” There are numerous problems with this assumption, starting with the fact that the 66% figure is wrong; in fact, the actual “at risk” (his term, not the US Government’s) percentage of the IT budget at the end of FY 07 and FY 08 were, as noted above, 6.5% and 13%, respectively.

Sessions’ error here is significant, since he goes on in several places (cf. page 4) to cite his use of the % of the total IT budget as being significant, when he’s not talking about the total IT budget at all.

Furthermore, it is unclear whether his phrase “the rest of the world” means all other national governments, or all other entities doing IT project development. It seems to be the latter, though it’s hard to tell from his statements. On the other hand, I have spent years consulting with corporations on troubled projects, and I can tell you that they do not have 66% of their IT budgets devoted to “at risk” projects. In fact, the majority of corporate IT budgets are devoted to maintenance of existing systems, not new and risky projects (cf. here, here, here, and here, as simple examples).

As noted, Sessions then assumes that the failure rate for “at risk” IT projects is 65%, which means that (as he says) “I am calculating that 43% (.65 x .66) of the total IT budget” is devoted to failed projects. At this point, his figures become nonsensical, as they are derived both from misreadings and lack of complete information about the Federal IT budget and projects. To wit:

• The 535 “not well planned and managed” IT projects in the US FY 09 budget only represent 38% of the total IT budget, not 66% as Sessions mistakenly states.
• In the two previous years (FY 07 and FY 08), the number of IT projects labeled as “not well planned and managed” dropped during the course of each year (see the 2nd table above). In FY 07, it dropped from 263 projects in Q1 to just 84 in Q4, which means that 69% were moved off of the “not well planned and managed” list during the year. Likewise, in FY 08, it dropped from 346 projects in Q1 to 134 projects in Q4, a drop of 61%. This directly contradicts Sessions’ assumption of a 65% failure rate for projects in the “not well planned and managed” category.
• The FY ‘09 Analytical Perspective says nothing about actual failed projects, as far as I can tell.

Sessions then goes on to make further out-of-his-hat assumptions regarding “direct and indirect costs”. He cites an example of the IRS (an agency long troubled by IT woes) and notes a lost opportunity based on fraudulent tax returns due to the system not being operational. He projects a loss over two years ($1.788 billion), compares it to the cost of the failed modernization ($185 million over a ten-year period), and calculates an indirect costs ratio of 9.6 to 1. He then decides — with no other documentation or analysis whatsoever — that the universal ratio of indirect to direct costs for a failed IT project ranges from 5:1 to 10:1, and uses the “average” of 7.5:1.

There are so many problems here that I scarce know where to start. For starters, the term “average” assumes an even distribution of ratios from 5:1 to 10:1 and does not recognize any ratios lower than 5:1. I’ve seen many failed projects that had much lower ratios of “indirect” to “direct” costs, since the firm simply continued to operate using the existing systems, and the “lost opportunity” for not having the new system in place was relatively small.

More importantly, the IRS gets to collect taxes from the entire US: $2.5 trillion in tax collections each year. Using the IRS as a baseline makes little sense for most other government agencies, and even less sense for most corporations and non-government organizations (NGOs), because most IT systems in most organizations (government or private) do not have the ability to generate such magnitudes of revenue, period.

Indeed, there is a long-standing controversy within IT management circles as to whether a new computer system can be relied upon to provide any significant return on investment (ROI), or whether it exists merely to “keep up with the competition”.

Sessions concludes his section on calculations thusly (p. 5, emphasis his):

Of course, these calculations are estimates. I recommend you don’t get overly focused on the exact amounts. I could be off by ten or twenty percent in either directions. The real point is not the exact numbers, but the magnitude of the numbers and the fact that the numbers are getting worse.

Unfortunately, Sessions is fundamentally wrong in his numerical analysis, and his numbers are off by far more than “ten or twenty percent”. For the Federal Government alone, they are off by almost  a full order of magnitude (10x), due to his critical errors both on the percentage of the Federal IT ‘09 budget “at risk” (it’s 38%, not 66%) and the number of “at risk” projects that fail (he says 65%; the US government numbers for FY 07 and 08 show that only 35% of the projects — representing just 6.5% to 13% percent of the Federal IT budget — were still “at risk” at the end of each fiscal year, and it gives no figures that I can find for actual failed IT projects).

Furthermore, his projection of the (erroneous) 66%-of-IT-budget-at-risk figure on the rest of the world is just wrong, especially in corporations and business (which spend vastly more on IT than the US government). In those organizations, maintenance costs dominates, and the percentage of the IT budget devoted to new projects tends to be small (20% or less), with an even smaller fraction of that representing “at risk” projects.

I may comment more on Sessions’ paper, but my conclusion here is that his estimate of $500 billion/month in lost direct and indirect costs due to IT systems failure just does not hold up, in my opinion.  ..bruce..

INDIANAPOLIS – Problems with Indiana’s landmark automation of welfare eligibility have cost some disabled residents food stamps and other benefits they need to survive, the American Civil Liberties Union alleges in a lawsuit that seeks class-action status.

In one case, a woman with hearing problems and other disabilities lost her Medicaid and food stamps after being told she could not meet in person with a state case worker, the lawsuit alleges. In another, a mother of two lost her food stamps and subsidized health care for her children when the tax return she provided didn’t include one attachment. . . .

Secretary Mitch Roob of the Indiana Family and Social Services Administration, the lead defendant in the case, said the state was complying with the law in denying or eliminating benefits to clients whose applications were lacking necessary information. He said the state would seek to have the lawsuit thrown out.

Roob also said the state’s rollout of the automated system under a $1.16 billion, 10-year contract with IBM Corp., Affiliated Computer Services Inc. and other companies reached 20 additional counties in northeastern and southwestern Indiana on Monday.

The case was amended Friday in Marion Superior Court in Indianapolis to include members of six households, all residents of the 12-county region where FSSA inaugurated the privatized welfare system by which clients can use the Internet, call centers and fax machines to apply for and renew benefits.

Rose said that automation creates obstacles for people with mental disorders and other disabilities, or those with limited schooling, further complicating an eligibility process with dozens of hurdles for each of Indiana’s 1.1 million welfare recipients.

Read the whole article. I will be interested in following this case.  ..bruce..

…the computer-operated baggage system has crashed and luggage is now being sorted manually before being loaded on to planes.

Twelve return flights to short-haul destinations have been cancelled. . . .

In just over a week the £4.3 billion investment has seen an unimaginable series of PR catastrophes.

Around 19,000 lost bags had to be transported to Milan to be sorted, hundreds of flights have been cancelled, and thousands of furious passengers have sworn they will never fly BA again.

A lack of staff parking and faulty IT systems started the rapid descent into terminal chaos which has seen BA suffer a 5% fall in business class passengers and left it with a £16 million bill.

This, of course, will remind those of us who track such things on this side of the Atlantic of the 10-year failed effort by United Airlines to automate the baggage system at Denver International Airport. One can only hope that our British cousins get their problems solved and solved a bit quicker. ..bruce..

This news just came over the wire:

BOSTON/NEW YORK (Reuters) – Waste Management Inc said it spent more than $100 million on a computer system that was supposed to help it save money, but instead turned out to be a “complete failure.”

Waste Management spokeswoman Lynn Brown said on Wednesday that her company sued SAP AG, the German-based company that sold it the system, seeking all its expenses plus punitive damages.

Installations of enterprise resource planning (“ERP”) systems, such as SAP, are prone to end up in disputes just because of the sheer complexity of the installation, as well as the heavy business process changes required. ..bruce..

Officials at HM Courts Service (HMCS) say they have turned around a failing £447m project to provide a national case management system for magistrates courts – a scheme that is 16 years late and will cost nearly three times more than expected.

HMCS change managers say they have been standardising business practices in courts and “not just delivering an IT system”, since they took direct control of the project in January 2007.

Officials have told Computer Weekly that independent Gateway reviews of the Libra project have progressed from a red warning light in 2006 to amber last March and last month to green.

Libra, which is in 83 courts, is due to be rolled out to all 370 magistrates courts in England and Wales by the end of December.

HM Courts Service has changed direction by attempting to shape new business processes around the specific needs of those who work in the courts, rather than imposing any central diktat.

Executives on the change management team say they are rolling out new IT and business practices based on what works best in the courts, and validating it in live operation.

Those last two paragraphs are the most intriguing, and they point out the critical nature of change management (from a process and organizational sense, not in the ’source code control’ sense) in any large-scale IT project that will impact business processes. Be sure to read the entire article; it not only gives more details, it also has links to earlier articles about the same project.

Ten years after it was first announced and almost $100 million later, Sydney is no closer to a cashless public transport ticketing system after the NSW Government was forced to terminate its contract for the troubled Tcard.

Transport Minister John Watkins announced the contract with Integrated Transit Solutions Ltd (ITSL) to develop the Tcard was cancelled at 1pm (AEDT) because of repeated delays.

He said the Government would now pursue a damages claim through the courts to try to recoup as much of the $95 million of taxpayers money spent on the failed project as possible.

If you read the whole article (and you should), you’ll note that this project was announced in 1997 and was originally intended to be completed in time for the 2000 Olympics in Sydney. The current contractor, ISTL, took over in 2003, but (according to the NSW Goverment) repeatedly failed to meet milestones.

UPDATE: Here are a few more articles on the same canceled project:

• $95m down the drain, and transport card is years off (refers to ERG, the parent company of ISTL)
  
• Public transport card plan dropped
• NSW scraps Tcard project (refers to other troubled IT projects contracted by the gov’t)

Who is the chief architect? Fred Brooks famously raised the issue and role of the chief (software) architect over 30 years ago in The Mythical Man-Month, and yet many IT organizations still fail to see the critical need for conceptual unity in a given software project. On one project I reviewed, I asked this question of the 40 or so people involved and got several different answers, the most common of which was “no-one”.

Who is the project manager? This is usually more easy to answer — but if it’s the same person as the chief architect, watch out. I know from personal experience that it is nearly impossible to do a good job both as a project manager and a chief architect for at least two reasons. First, for a software project of any significant size, each job is a full-time job, and someone who tries to do both will do neither well. Second, there are inherent conflicts between the two roles — the architect wants to do things perfectly, and the project manager wants to do things quickly. One person carrying out both roles will inevitably (and privately) lean one way or the other.

Who is the chief quality engineer? Remarkably, many IT organizations will not appoint a quality assurance lead for a given project. I was stunned when I reviewed a multi-hundred-million-dollar project and found that there was no separate, formal QA department and no head of QA. It also explained why this project was more than 100% over schedule and over budget. The chief quality engineer is the gatekeeper — s/he stands between the development group and production/shipping. Without that key role, a large IT project can go into a unstable state that never goes into production.

How do you define “quality assurance”? If the answer is simply or mostly “testing”, then you have big, big problems. Testing, while critical, is only a small part of the full cycle of QA activities, including: defining (and enforcing) standards and guidelines; finding project personnel with appropriate expertise; reusing relevant and tested deliverables; setting up and enforcing appropriate configuration management; setting up and using a defect management system, along with an appropriate change control process; gathering useful metrics; conducting appropriate reviews, inspections, and walkthroughs; and having a formal software release process.

Where is your current baseline specification for the project? Years ago, I was brought in a (consulting) chief architect for a subcontractor in a global project coordinated by Motorola. The same day I showed up at the subcontractor’s offices, Bob Millar also showed up as the new program director from Motorola for the scheduled weekly meeting with the subcontractor. Almost the first question Millar asked was: “Where’s your current baseline specification?” The people around the conference table looked a bit confused, so Millar repeated: “Where is the current baseline specification for your subsystem in this global project?” No one could answer him, so Millar continued: “If you don’t have a current change-controlled baseline specification for your subsystem, then how do you know when you’re done?” Millar than made it clear that he was going to raise this issue each and every week until that baseline specification was sitting in the middle of that conference table. Within about four weeks, the baseline was there, in binders, on the table.

What metrics are you tracking, and how are you tracking them? Accurately predicting the progress of a large-scale IT project is almost as difficult and error-prone as estimating it in the first place. There’s an old saw in IT project management that “the first 90% of a project takes 90% of the time, and the remaining 10% of the project takes the other 90% of the time.” The most common metric — # of lines of source code — is only vaguely informative at best and wildly misleading at worst (particularly in object-oriented projects, where rearchitecting and refactoring is the norm). But most other metrics used tend to be subjective metrics — “I [the developer] think we’re about 70% done with this module” — and so largely useless. To be useful, a metric has to be: objective (anyone collecting it would arrive at the same value); automated (to help with “objective” and to allow it to be collected at any time); relevant (actually related to the completion of the project); and predictive (able to predict with some degree of accuracy when the project will be completed).

What is driving the planned completion date, and how is it being managed? A completion date given by fiat from above will almost certainly not be met unless there is a corresponding ruthlessness in cutting features and reducing scope. On the other hand, the lack of any planned completion date almost always result in an open-ended project that never comes to completion. Tying into the QA question above, you need to see if there is a regular, effective, and (when necessary) ruthless change-control process in place, both for prioritizing bug fixes and for controlling or even reducing scope (in the way of implemented or abandoned features).

What is the history of the schedule and budget? Some large-scale software projects get into a pattern that I’ve previously described as “the never-ending story” — a series of schedule slips and budget overruns, typically occurring as the project nears its latest deadline, with the project never really getting any closer to going into production. It may seem impossible to you that a firm could spend years and millions (or even hundreds of millions) of dollars and yet never produce something that can actually be used, but it happens all the time.

What do the engineers in the trenches say? My project reviews almost always include a series of confidential, not-for-attribution interviews with the engineers down in the trenches — and while they may or may not have a global picture of the project, they usually have a pretty good idea of how things are going in their specific area, and they’re usually honest about it when asked. Bad news tends to get filtered as it moves up the food chain. Often the result is that at the highest levels the project appears to be doing just fine — until three weeks before completion, it suddenly slips another 3-6 months.

No big secrets or fancy analysis. These few questions, honestly answered, will quickly flush out most of the trouble areas (if any exist) in a large-scale IT project.

Summary: A client starts a large IT project that involves one or more outside firms (consultants, vendors, developers, manufacturers). Within the client organization, a champion arises (or is appointed) for this project. The project moves ahead with the typical bumps and setbacks, but the champion keeps it moving forward. Then the champion disappears from that role: s/he leaves the client, gets transferred, gets reassigned, gets fired, or even (as in one case we worked on) dies. With the champion gone, the client starts rethinking the project. Whoever inherits the project (within client management) may have other priorities, goals, and/or projects and has no particular investment in this one. The client decides to scale back or cancel the project rather than push it to completion.

Causes: Extensive literature exists on the champion’s critical role in project success (e.g., this). The simple fact of the departure/demotion of the original project champion can have a significant impact on the project’s success. However, what I have seen time and again is more than the mere absence of the original project champion. The underlying dynamic is that the individual who inherits the project typically has little invested — professionally, politically, or personally — in seeing the project to completion and often has conflicting priorities, often (though not always) budget-related (hence the “buyer’s remorse”).

In such cases, the new “project owner” starts looking around for a way to bring the project to an end. This often takes the form of blaming the developer — legitimately or otherwise — for alleged or real problems with the project. The customer may suddenly withhold all further payments, in spite of contractual agreements; may suddenly expand the scope and/or contract the schedule of the project; or may simply halt the project altogether. From the developer’s point of view, this sudden shift is completely unexpected and may even be in direct contradiction to what the customer was saying just weeks or even days beforehand.

In fairness, the customer may well take these steps for a truly troubled IT project, and with good reason, up to and including removing the existing project champion. But that’s not the situation I’m talking about. Here are a few real-world examples:

CASE #1: One national office of a global firm had agreed on a multi-year, fixed-price plan to phase in modified off-the-shelf software and phase out their existing home-grown systems. The vendor had first taken the customer through two short-term pilot projects to ensure feasibility and to more accurately judge the scope of the main project. The CEO of the national office was the project champion. A project review (several months into the main project) by the firm’s global CIO and CTO resulted in approval for the approach and progress to date. Yet a few weeks later, the same CIO came back, claiming the project was unacceptable, that feature delivery would have to be accelerated, and that no further funds would be paid until the project was completed. The vendor was stunned, and the firm’s national CEO was doing his best to mediate things — until he died, unexpectedly, due to undiagnosed health problems. The dispute ended up in mediation, with the vendor claiming various contractual payments that were still due (the vendor won).

What we found out (via discovery) was that during those few weeks between the global CIO’s glowing review and the sudden change of demands on the customer’s part is that the firm had appointed a new regional director over the part of the world where this project was going on, and that regional directory had held a meeting with the global CIO to review all IT projects in that region. After that meeting, the global CIO suddenly had a completely different viewpoint about the project.

CASE #2: A professional services firm retained a mid-sized consulting organization to come in and replace its existing, home-grown, UNIX-based mission-critical customer management and accounting system with commercial off-the-shelf accounting software and a custom-developed customer management system. The consultants went through a very detailed requirements gathering process and produced a detailed (200+ pages) specification along with a fixed price bid. The customer did not want to pay that much, and so features were removed until the fixed price bid was acceptable to the customer. The consulting firm started work and successfully delivered the first three (out of four) milestone deliveries.

However, by this time the original project champion had left the customer’s firm, and the firm’s CEO had taken over project oversight. When the consultants delivered the fourth (and final) milestone delivery, the customer did not formally accept or reject it within 7 days (as was called for in the contract); instead, the customer came back and started asking for features to be added back in that had previously been removed to get the fixed price bid down. The consultants accommodated up to a point, but when it became clear after several weeks that the customer was trying to go back to the original full specification without paying any additional money, they ceased work on the project (which the contract explicitly allowed them to do in the event of the customer’s failure to formally accept or reject the delivery). The dispute ended up in mediation (the consultants won).

CASE #3: A major utility firm hired a major utility-oriented development group to come in and replace their existing, mostly home-grown systems with a mixture of custom and off-the-shelf software and hardware. This was a multi-year, multi-million-dollar project. The developing firm went through a rigorous methodology to not only specify the deliverable in the project but to specify and have both parties agree upon the conditions under which the deliverables would be accepted. The project moved ahead for many months, with all deliverables being formally accepted by the customer.

However, nearly two years into the project, the utility firm was acquired by a multi-state utility holding company that already owned several other utility firms. The holding company brought in its own private consultants to review the project. Discovery later indicated that these consultants were specifically tasked to find reasons to halt the project, since the systems under development were customized to the utility firm, and the holding company wanted to use standardized systems across all its holdings. The utility firm halted delivery of the software literally days before the first major production software delivery by the development group, then put the entire project into limbo without making the contractual payments required either by a suspension or a cancellation of the project. The dispute ended up in mediation, but eventually settled.

Recommendations: Champions matter, and they especially matter when they disappear or otherwise lose influence. I saw this pattern often enough in IT systems failure lawsuits that I contacted my former boss, Tony Gibson (of OSG Incorporated) and told him that this was a major risk factor that he should monitor for all of OSG’s projects with its customers. Any vendor/developer/consulting firm should carefully track the political dynamics within its customer organizations; you may not be able to prevent an abrupt course change by your customer, but you may be able to prepare for and soften the blow.

Having reviewed these cases and the patterns they exhibit, some practical suggestions come to mind.

Lesson 1: Get expert legal and IT guidance before signing anything.

Most of the legal pitfalls in IT business deals are well documented. Too often though, clients, vendors, and manufacturers sign contracts and agreements without having them reviewed by lawyers who understand IT-specific pitfalls. This may seem obvious, yet case after case in the survey focuses on the terms of signed agreements and the efforts by parties on one or both sides to find promises and protections that the courts find were never put in writing.

It’s also a good idea to run the agreement past technical people in relevant IT departments. For vendors and manufacturers, such people are likely to point out “overly enthusiastic” promises that the vendor might have difficulty keeping. For clients, such people can help spot flaws in technical commitments that relate to what is to be delivered and when.

Lesson 2: Specify critical terms and articulate protections before agreeing to a sale of goods or services.

In many of the cases we surveyed, one side or the other ended up losing because the parties failed to define important technical terms with sufficient specificity. When reaching agreement on what will be delivered and when, parties should endeavor to define clearly and in detail key terms. These include the various types of quality assurance (QA) that will be performed, the IT requirements that will need to be met for the client accepting the system, and the process for having the vendor/manufacturer deal with defects that show up after acceptance.

IT quality assurance aspects and activities that you could define and agree upon might include: expertise; guidelines and standards; metrics (measurements of progress); quality reviews; the various forms of testing; defect management; configuration management; and product release cycles, including technical support, maintenance, and upgrades.

Likewise, a good starting point for defining acceptance criteria is the list of quality attributes given earlier: reliability, performance, functionality, compatibility, lifespan, deployment, support, and cost. If both sides have a clear, written understanding in advance of what the system will and won’t do in each of those areas, the project has a higher chance of success.

Lesson 3: Act quickly when problems arise.

The foundation of both the “Faulty Towers” and “Never-ending Story” patterns is the client’s willingness to let the situation drag out for months or even years in spite of apparent repeated failures on the part of the vendor/manufacture to deliver or perform.

Why does this so often happen? Three factors, individually or combined, underlie this unwillingness to pull the plug on a project. First, the client may have a substantial monetary investment in the project and sees going forward as a better option than pulling out. Second, the executives and managers within the client firm who made the decision to acquire or develop the system in the first place often have a strong professional and personal interest in seeing it go forward and not having their original support for the project held against them. Third, migration off old systems that has already occurred may make going back difficult.

Even so, letting such projects go ahead as they are is almost always the worst decision. Case after case, both in our survey and in software engineering literature, shows that without direct and dramatic intervention, the client will end up spending more time and money in a project that in the end fails anyway. Instead, the client should document clearly the problems and attendant risks and consequences, then review them internally to determine the best course: proceed ahead, redirect the project (such as by scaling it back or changing the requirements), or cancel the project altogether. The client should then work with the vendor/manufacturer to achieve the desired goal.

Lesson 4: Remember that new technology entails risks.

The phrase “new technology” here refers to either relatively new commercial IT system offerings from a commercial vendor or manufacturer, or custom IT systems being built specifically for the client. While many IT system project failures stem from mismanagement or simple human failings, some founder on genuine technical issues: that which is being attempted may just not be feasible. Adele Goldberg, an IT software pioneer, has said, “Only optimists build complex systems.” One might amend that to read “build or buy”, but the corollary is that it is often optimists who end with project failures. Some healthy skepticism and caution, as well as a sober realization that large, complex IT projects have a high rate of late delivery or failure, should be taken into account when planning and negotiating the acquisition of such systems.

Conclusions

Most of the observations and recommendations made in this paper are common sense. That said, we have to ask ourselves why such common sense is so often set aside or ignored. The court documents and other literature on these cases don’t spell it out, but professional experience suggests that it largely boils down to human failings, ranging from naïveté to dishonesty.

The patterns of litigation involving information technology are readily identified. Judicious thought, consultation, and agreement on detailed terms, especially before entering into a legal agreement, will go a long ways to avoiding those patterns, reducing the risk of or need for a lawsuit. And that, in the end, is better for all parties involved.

Summary: The manufacturer makes some change in the functionality or configuration of the system, which is already in use. The change results in unpleasant or unintended consequences for one or more clients.

Causes: Someone at the vendor/manufacturer mandates or proposes a change, and it gets made without careful consideration of its impact on existing systems or proper testing. The resulting consequences for one or more clients lead to legal action. Note that the only class-action lawsuits represented in our survey all fall into this category and all involve commercial products or services.

Recommendations: Think carefully about the consequence of changes. Test modified products thoroughly. Roll them out in limited numbers with trusted, friendly clients to flush out problems. Act quickly to fix problems that show up.