CATEGORIES: political, architectural, managerial

While discussing the challenges of software development with Taligent trainer Tom Affinito back in the mid-1990s, I mentioned — citing Fred Brooks — the need for a chief architect. Tom immediately responded, “Yes, and ultimately architecture is a political act.”

That observation has stuck with me ever since. Unless you are both sole architect and sole implementor — and maybe even then — architecture is a political act, and it is especially true in development groups of any size. Developers like to think themselves above politics. Not so; they are as human as anyone else. Developers can be quite political and will use a variety of techniques, good and bad, to achieve their ends: persuasion, hyperbole, fear-mongering, caucusing, lobbying, consensus building, rumors, backbiting, leadership, intimidation, sacrifice, tantrums, threats, and subversion, to name a few. The fact that developers are generally very bright just increases the effectiveness of whatever techniques they use.

Architecture is the lodestar of developer politics because it is ultimately the most prestigious role in software development. Brooks goes to great lengths (and rightfully so) to assert the intellectual and creative challenges of implementation, noting that architecture is not the be-all and end-all. Still, it is Frank Lloyd Wright’s name that remains attached to the structures he designed, not the name of the contractors who built them. Or, to update an old programming joke, the verb “to develop” is conjugated this way: “I architect, you implement, he tests.”

Yet we may fail to recognize or acknowledge this situation and its implications. Why? First, we want to believe that all developers have (with regard to the project) the same intentions and goals. Second, we want to believe that the best course is the one that is obvious to us and that everyone else will agree with that. Third, we don’t want to have to deal with politics ourselves, particularly politics that can lead to confrontations.

Symptoms

Assuming — without verification — that all other members of the development group have the same goals, ideas, and talents. Not wanting to assert architectural issues. Finger-pointing as design problems arise.

Consequences

Hidden or overt team dissension, leading to poor morale and lack of cooperation as well as a weakened architecture.

Detection

Ask yourself these questions:

• Who is the chief architect of this project?
• How well do the other developers support the chief architect?
• How effective is the chief architect in her role?
• What are the obstacles she faces?
• What are all the political issues involved?

Answering these questions should go a long way toward determining whether you have underestimated the politics of architecture.

Extraction

The first issue to address is that of authority commensurate with responsibility: if there is a chief architect on the team, does she really have the authority to compel adherence to the architecture? This kind of authority is different than being able to say, “Do this or you’re off the project,” which often leads to subtle or overt rebellion by team members. This kind of authority accrues from several factors:

• a proven track record
• respect from teammates
• support from technical and upper management
• a mutual pact with the developers that the chief architect will give serious consideration to all their ideas and suggestions but that they will abide by her decisions
• a willingness to give in on the small things so that she can hold her ground on the big things

Beyond that, healing a team that is experiencing jealousy and dissension is no easy feat, especially when you’re in the middle of a project. Books and seminars on team-building abound; study, learn, and apply.

Prevention

Go through the questions in Detection but cast them in the future tense. Then, as with Extraction, proactively work to build the team and establish the authority of the chief architect.

Contributor/References

Webster, from Pitfalls of Object-Oriented Development (1995).

…the computer-operated baggage system has crashed and luggage is now being sorted manually before being loaded on to planes.

Twelve return flights to short-haul destinations have been cancelled. . . .

In just over a week the £4.3 billion investment has seen an unimaginable series of PR catastrophes.

Around 19,000 lost bags had to be transported to Milan to be sorted, hundreds of flights have been cancelled, and thousands of furious passengers have sworn they will never fly BA again.

A lack of staff parking and faulty IT systems started the rapid descent into terminal chaos which has seen BA suffer a 5% fall in business class passengers and left it with a £16 million bill.

This, of course, will remind those of us who track such things on this side of the Atlantic of the 10-year failed effort by United Airlines to automate the baggage system at Denver International Airport. One can only hope that our British cousins get their problems solved and solved a bit quicker. ..bruce..

Ten years after it was first announced and almost $100 million later, Sydney is no closer to a cashless public transport ticketing system after the NSW Government was forced to terminate its contract for the troubled Tcard.

Transport Minister John Watkins announced the contract with Integrated Transit Solutions Ltd (ITSL) to develop the Tcard was cancelled at 1pm (AEDT) because of repeated delays.

He said the Government would now pursue a damages claim through the courts to try to recoup as much of the $95 million of taxpayers money spent on the failed project as possible.

If you read the whole article (and you should), you’ll note that this project was announced in 1997 and was originally intended to be completed in time for the 2000 Olympics in Sydney. The current contractor, ISTL, took over in 2003, but (according to the NSW Goverment) repeatedly failed to meet milestones.

UPDATE: Here are a few more articles on the same canceled project:

• $95m down the drain, and transport card is years off (refers to ERG, the parent company of ISTL)
  
• Public transport card plan dropped
• NSW scraps Tcard project (refers to other troubled IT projects contracted by the gov’t)

PC gamers waited with bated breath, to the point of asphyxiation, for Half-Life 2, but the game took an eternity to complete. Likewise, Half-Life 2: Episode 2 teased us with expectation until its eventual release, alongside Team Fortress 2 (nine years in the making), in Valve’s The Orange Box. Doom 3, BioShock, Crysis: all were delayed.

Meanwhile, other PC games – such as Duke Nukem Forever, Too Human, and StarCraft: Ghost – have turned release dates into Monty Python-esque jokes. When we eventually play these games, we’ll be flying around in hovercars.

And the fun continues: Spore’s release has been put back until at least April, Warhammer Online was moved from early this year to the summer, and who honestly knows when the world will see GTA IV?

The article doesn’t go into as much depth as I’d like, but does note at least three major issues: delays in getting access to licensed rights and technology; overly ambitious designs; and deliberate slips so as to even out cash flow.

Note that commercial computer games form a multi-billion-dollar industry (Americans now spend nearly twice as much on video games than they do on movie tickets).

Summary: A client starts a large IT project that involves one or more outside firms (consultants, vendors, developers, manufacturers). Within the client organization, a champion arises (or is appointed) for this project. The project moves ahead with the typical bumps and setbacks, but the champion keeps it moving forward. Then the champion disappears from that role: s/he leaves the client, gets transferred, gets reassigned, gets fired, or even (as in one case we worked on) dies. With the champion gone, the client starts rethinking the project. Whoever inherits the project (within client management) may have other priorities, goals, and/or projects and has no particular investment in this one. The client decides to scale back or cancel the project rather than push it to completion.

Causes: Extensive literature exists on the champion’s critical role in project success (e.g., this). The simple fact of the departure/demotion of the original project champion can have a significant impact on the project’s success. However, what I have seen time and again is more than the mere absence of the original project champion. The underlying dynamic is that the individual who inherits the project typically has little invested — professionally, politically, or personally — in seeing the project to completion and often has conflicting priorities, often (though not always) budget-related (hence the “buyer’s remorse”).

In such cases, the new “project owner” starts looking around for a way to bring the project to an end. This often takes the form of blaming the developer — legitimately or otherwise — for alleged or real problems with the project. The customer may suddenly withhold all further payments, in spite of contractual agreements; may suddenly expand the scope and/or contract the schedule of the project; or may simply halt the project altogether. From the developer’s point of view, this sudden shift is completely unexpected and may even be in direct contradiction to what the customer was saying just weeks or even days beforehand.

In fairness, the customer may well take these steps for a truly troubled IT project, and with good reason, up to and including removing the existing project champion. But that’s not the situation I’m talking about. Here are a few real-world examples:

CASE #1: One national office of a global firm had agreed on a multi-year, fixed-price plan to phase in modified off-the-shelf software and phase out their existing home-grown systems. The vendor had first taken the customer through two short-term pilot projects to ensure feasibility and to more accurately judge the scope of the main project. The CEO of the national office was the project champion. A project review (several months into the main project) by the firm’s global CIO and CTO resulted in approval for the approach and progress to date. Yet a few weeks later, the same CIO came back, claiming the project was unacceptable, that feature delivery would have to be accelerated, and that no further funds would be paid until the project was completed. The vendor was stunned, and the firm’s national CEO was doing his best to mediate things — until he died, unexpectedly, due to undiagnosed health problems. The dispute ended up in mediation, with the vendor claiming various contractual payments that were still due (the vendor won).

What we found out (via discovery) was that during those few weeks between the global CIO’s glowing review and the sudden change of demands on the customer’s part is that the firm had appointed a new regional director over the part of the world where this project was going on, and that regional directory had held a meeting with the global CIO to review all IT projects in that region. After that meeting, the global CIO suddenly had a completely different viewpoint about the project.

CASE #2: A professional services firm retained a mid-sized consulting organization to come in and replace its existing, home-grown, UNIX-based mission-critical customer management and accounting system with commercial off-the-shelf accounting software and a custom-developed customer management system. The consultants went through a very detailed requirements gathering process and produced a detailed (200+ pages) specification along with a fixed price bid. The customer did not want to pay that much, and so features were removed until the fixed price bid was acceptable to the customer. The consulting firm started work and successfully delivered the first three (out of four) milestone deliveries.

However, by this time the original project champion had left the customer’s firm, and the firm’s CEO had taken over project oversight. When the consultants delivered the fourth (and final) milestone delivery, the customer did not formally accept or reject it within 7 days (as was called for in the contract); instead, the customer came back and started asking for features to be added back in that had previously been removed to get the fixed price bid down. The consultants accommodated up to a point, but when it became clear after several weeks that the customer was trying to go back to the original full specification without paying any additional money, they ceased work on the project (which the contract explicitly allowed them to do in the event of the customer’s failure to formally accept or reject the delivery). The dispute ended up in mediation (the consultants won).

CASE #3: A major utility firm hired a major utility-oriented development group to come in and replace their existing, mostly home-grown systems with a mixture of custom and off-the-shelf software and hardware. This was a multi-year, multi-million-dollar project. The developing firm went through a rigorous methodology to not only specify the deliverable in the project but to specify and have both parties agree upon the conditions under which the deliverables would be accepted. The project moved ahead for many months, with all deliverables being formally accepted by the customer.

However, nearly two years into the project, the utility firm was acquired by a multi-state utility holding company that already owned several other utility firms. The holding company brought in its own private consultants to review the project. Discovery later indicated that these consultants were specifically tasked to find reasons to halt the project, since the systems under development were customized to the utility firm, and the holding company wanted to use standardized systems across all its holdings. The utility firm halted delivery of the software literally days before the first major production software delivery by the development group, then put the entire project into limbo without making the contractual payments required either by a suspension or a cancellation of the project. The dispute ended up in mediation, but eventually settled.

Recommendations: Champions matter, and they especially matter when they disappear or otherwise lose influence. I saw this pattern often enough in IT systems failure lawsuits that I contacted my former boss, Tony Gibson (of OSG Incorporated) and told him that this was a major risk factor that he should monitor for all of OSG’s projects with its customers. Any vendor/developer/consulting firm should carefully track the political dynamics within its customer organizations; you may not be able to prevent an abrupt course change by your customer, but you may be able to prepare for and soften the blow.

Having reviewed these cases and the patterns they exhibit, some practical suggestions come to mind.

Lesson 1: Get expert legal and IT guidance before signing anything.

Most of the legal pitfalls in IT business deals are well documented. Too often though, clients, vendors, and manufacturers sign contracts and agreements without having them reviewed by lawyers who understand IT-specific pitfalls. This may seem obvious, yet case after case in the survey focuses on the terms of signed agreements and the efforts by parties on one or both sides to find promises and protections that the courts find were never put in writing.

It’s also a good idea to run the agreement past technical people in relevant IT departments. For vendors and manufacturers, such people are likely to point out “overly enthusiastic” promises that the vendor might have difficulty keeping. For clients, such people can help spot flaws in technical commitments that relate to what is to be delivered and when.

Lesson 2: Specify critical terms and articulate protections before agreeing to a sale of goods or services.

In many of the cases we surveyed, one side or the other ended up losing because the parties failed to define important technical terms with sufficient specificity. When reaching agreement on what will be delivered and when, parties should endeavor to define clearly and in detail key terms. These include the various types of quality assurance (QA) that will be performed, the IT requirements that will need to be met for the client accepting the system, and the process for having the vendor/manufacturer deal with defects that show up after acceptance.

IT quality assurance aspects and activities that you could define and agree upon might include: expertise; guidelines and standards; metrics (measurements of progress); quality reviews; the various forms of testing; defect management; configuration management; and product release cycles, including technical support, maintenance, and upgrades.

Likewise, a good starting point for defining acceptance criteria is the list of quality attributes given earlier: reliability, performance, functionality, compatibility, lifespan, deployment, support, and cost. If both sides have a clear, written understanding in advance of what the system will and won’t do in each of those areas, the project has a higher chance of success.

Lesson 3: Act quickly when problems arise.

The foundation of both the “Faulty Towers” and “Never-ending Story” patterns is the client’s willingness to let the situation drag out for months or even years in spite of apparent repeated failures on the part of the vendor/manufacture to deliver or perform.

Why does this so often happen? Three factors, individually or combined, underlie this unwillingness to pull the plug on a project. First, the client may have a substantial monetary investment in the project and sees going forward as a better option than pulling out. Second, the executives and managers within the client firm who made the decision to acquire or develop the system in the first place often have a strong professional and personal interest in seeing it go forward and not having their original support for the project held against them. Third, migration off old systems that has already occurred may make going back difficult.

Even so, letting such projects go ahead as they are is almost always the worst decision. Case after case, both in our survey and in software engineering literature, shows that without direct and dramatic intervention, the client will end up spending more time and money in a project that in the end fails anyway. Instead, the client should document clearly the problems and attendant risks and consequences, then review them internally to determine the best course: proceed ahead, redirect the project (such as by scaling it back or changing the requirements), or cancel the project altogether. The client should then work with the vendor/manufacturer to achieve the desired goal.

Lesson 4: Remember that new technology entails risks.

The phrase “new technology” here refers to either relatively new commercial IT system offerings from a commercial vendor or manufacturer, or custom IT systems being built specifically for the client. While many IT system project failures stem from mismanagement or simple human failings, some founder on genuine technical issues: that which is being attempted may just not be feasible. Adele Goldberg, an IT software pioneer, has said, “Only optimists build complex systems.” One might amend that to read “build or buy”, but the corollary is that it is often optimists who end with project failures. Some healthy skepticism and caution, as well as a sober realization that large, complex IT projects have a high rate of late delivery or failure, should be taken into account when planning and negotiating the acquisition of such systems.

Conclusions

Most of the observations and recommendations made in this paper are common sense. That said, we have to ask ourselves why such common sense is so often set aside or ignored. The court documents and other literature on these cases don’t spell it out, but professional experience suggests that it largely boils down to human failings, ranging from naïveté to dishonesty.

The patterns of litigation involving information technology are readily identified. Judicious thought, consultation, and agreement on detailed terms, especially before entering into a legal agreement, will go a long ways to avoiding those patterns, reducing the risk of or need for a lawsuit. And that, in the end, is better for all parties involved.

Summary: The manufacturer makes some change in the functionality or configuration of the system, which is already in use. The change results in unpleasant or unintended consequences for one or more clients.

Causes: Someone at the vendor/manufacturer mandates or proposes a change, and it gets made without careful consideration of its impact on existing systems or proper testing. The resulting consequences for one or more clients lead to legal action. Note that the only class-action lawsuits represented in our survey all fall into this category and all involve commercial products or services.

Recommendations: Think carefully about the consequence of changes. Test modified products thoroughly. Roll them out in limited numbers with trusted, friendly clients to flush out problems. Act quickly to fix problems that show up.

Summary: The client buys a system from the vendor. Some time later, the client discovers that the system either no longer meet its needs or that the vendor/manufacturer will no longer support it.

Causes: Unplanned obsolescence cases are not the result of a natural flow of upgrades and replacements in IT systems. These cases stem either from a sudden abandonment of a product version or line by the manufacturer/vendor or from some built-in flaw and previously unknown (at least to the client) flaw, such as the Year 2000 issue. The former usually stems from financial issues, with the vendor/manufacturer abandoning an unprofitable line; the latter from software engineering flaws.

Recommendations: Most vendors and manufacturers give sufficient advance notice of such phase-outs, with a migration path for current users and sometimes a support plan (usually expensive) for those clients who wish for whatever reason to continue to use the old version. However, market and other considerations can sometimes constrain such notification.[1] Still, any vendor or manufacturer planning an abrupt retirement of a product or product line had best provide a migration path for customers or be prepared to face exactly these type of lawsuits.

Clients should have contingency plans for migrating off IT technologies they use but do not control. The level of detail should correspond to the size and stability of the company in question, the proprietary nature of the system technology, and its market share. Clients will worry less about products from internationally prominent manufacturers of mainframes, servers, workstations, PCs, and corresponding software. However, the smaller the firm and the more custom or proprietary the software, the greater the risk.

Summary: The client contracts with the manufacturer to develop and install a system. The project starts. The completion date slips. It keeps slipping. Each time the adjusted delivery date approaches, the project slips yet again. At some point, one of three things happens: the manufacturer/vendor abandons the project; the client cancels the project; or the manufacturer delivers a system that the client terms wholly inadequate and unacceptable. In some cases, the effort has gone on for years, with millions of dollars spent and little to show for it.

Causes: This is actually one of the most well-known patterns in software development, particularly for large, in-house projects. Entire books are devoted to this subject, so it’s hard to summarize the causes here. Two factors show up time and again, though. One is a lack of clear, stable, and constrained requirements on the part of the client. The other is a lack of qualified technical managers and developers on the part of the manufacturer. Chances are that both factors occur in a given project failure, giving each side plenty of reason to point fingers at the other.

Recommendations: To use one software engineering maxim, “Start out stupid and work up from there.” Most failures of this type come from attempts to implement large, complex systems from scratch. Experience has shown that success in building such systems comes more often from implementing small, comprehensible systems that work, then evolving them into the desired large systems.

Beyond that, you should do a thorough risk assessment of the entire project at the start and take steps to reduce any risks and protect your interests accordingly. Again, this may sound obvious, but many system project failures, large and small, have come about because no one with sufficient authority was willing to raise risk issues at the start. Risk issues that should be addressed include the scope and inherent feasibility of the project, the stability of the client requirements, the development and quality practices of the manufacturer, and how realistic the estimates are for time, money, and other resources allocated.

Summary: This pattern actually lumps together two sub-patterns. In the first, the client purchases an IT system from the vendor by way of a leasing firm. The client is dissatisfied with the system and stops payment, whereupon the leasing firm sues the client. In the second sub-pattern, the client hires a consultant to recommend, select, or add value to system(s), vendor(s) and/or manufacturer(s). Problems occur in the development, installation, and/or use of the selected and possibly modified systems and the client blames the consultant, who may in turn blame the vendor/manufacturer. In both cases, someone other than the client and the vendor is being impacted by alleged problems with the system.

Causes: In a vendor/leasing firm/client triangle, the client usually signs a lease with a “hell or high water” clause, obligating it for the full lease regardless of the quality and usefulness of the system. At that point, the client is stuck with that obligation, and in the cases reviewed, the court consistently upheld that clause.

The various vendor/consultant/client triangles typically boil down to mutual finger pointing, with each party seeking someone else to blame. Consultants were most at risk in cases where they were making recommendations to clients; some courts found that the consultants were acting in a professional capacity and thus were held to a higher standard.

Recommendations: In at least one leasing case, while the court upheld the “hell or high water” clause on behalf of the leasing company, it ultimately ruled against the vendor because the vendor agreed to assume full liability under the clause should the system prove unacceptable to the client. Clients should keep this in mind, recognizing that otherwise the vendor has only limited exposure, if any, should the system prove unsatisfactory.

As for consultants, value-added retailers (“VARs”), and other third parties, the need for clear communication and agreement among all parties is critical. Make sure that the client knows exactly what you are (and are not) providing; likewise, be sure you have confidence in whatever systems you are using, acquiring, or recommending.