The second column, “Do Not Defer The Difficult in IT Projects”, describes the all-too-human tendency in IT development to put off dealing with the toughest problems until last — at which point, you may not be able to solve them all. It also explains why so many IT projects get 80-90% “done” and then suddenly slip for weeks or months without making much progress.

Feedback is always welcome.

Here’s my abstract of the talk:

INSIDE-OUT: Organizations too often treat software reliability as an ‘after the fact’ consideration, performing testing as a last step and then constraining it due to schedule and financial pressures. Webster will present a simple “inside-out” software lifecycle model where all software development activing (not just coding) takes place within a framework covering a broad spectrum of quality-related activities.

I’ll post the presentation slides (PPT, 340KB) here after the talk. ..bruce w..

Categories: managerial

This is a classic pitfall in software engineering. Typically, insufficient time is allocated for the problem specification, research, design, architecture, and review that should occur before coding and during each development cycle. Likewise, software quality assurance (SQA) is often given little time, money, or people. The whole focus is on coding, and often that is underestimated by assuming that development will be linear and that nothing will go wrong.

Adopting a new technology or methodology (the “TOM”) tends to compound this effect in several ways, some of which are discussed elsewhere. First, unrealistic expectations can creep in. Second, rapid prototyping and feature development can cause a false sense of progress. Third, coding time may well be genuinely reduced by the TOM.

As a result, an expectation can arise that all aspects of the development cycle, not just the coding portion, will be compressed. The first few times you use the TOM, it may require more time up front for design and architecture, and it’s likely to require as much or more testing time. As your development group gains more experience and expertise in the TOM, the entire cycle may start to compress, but that comes with time.

Symptoms

Non-coding tasks taking longer than the time allotted to them. Slowdowns during development due to a need to rethink design. Alpha and beta testing taking much longer than planned.

Consequences

Slipped schedules and missed deadlines. Rude surprises as design must be repeated or testing takes much longer than expected.

Detection

Apply Brooks’ rule of thumb: the time required for a project should break down into one-third for design and prototyping, one-sixth for implementing, and half for testing. If your proportions are radically different, then you may have misjudged the relative costs or at least, mislabeled them; a lot of design and testing gets buried inside implementation.

If you’re far along in your project, keep a close eye on time required for SQA and particularly for testing, which is usually underestimated. Agile methodologies do a better job of addressing testing up front, but SQA remains the poor cousin for most software projects.

Extraction

First, throw out your current schedule and do a hard reset of expectations, particularly among upper management. This is not easy to do, but honesty is always the best policy. Good luck.

Second, set up a development cycle—specify, design, prototype, review, implement, test—with the time allocated to each step in the cycle based roughly on the proportions given above. Recognize that the actual time for each step in each cycle will vary: An early cycle will tend to have more specification and design and less testing; a later cycle will reverse those proportions.

Third, manage through one complete cycle and see how well your estimated costs match reality. Adjust and repeat.

Prevention

First, get some project management software. It doesn’t have to be fancy; it just has to automate the task of adding the estimated times for each task, noting critical paths, and calculating a final date. This is important; attempting to schedule anything except the smallest and simplest project in your head will lead to unpleasant surprises.

Second, use steps two and three under Extraction to set up your schedule and to estimate relative costs. If possible, try this first with a relatively small project and then work up to larger projects. The goal is to be able to estimate both relative and absolute costs within a certain margin of error (say, 10 percent) on a regular basis.

References

Brooks, Frederick P., Jr. The Mythical Man-Month. Reading, MA: Addison-Wesley, 1995.

Webster, from Pitfalls of Object-Oriented Development (1995).

…the computer-operated baggage system has crashed and luggage is now being sorted manually before being loaded on to planes.

Twelve return flights to short-haul destinations have been cancelled. . . .

In just over a week the £4.3 billion investment has seen an unimaginable series of PR catastrophes.

Around 19,000 lost bags had to be transported to Milan to be sorted, hundreds of flights have been cancelled, and thousands of furious passengers have sworn they will never fly BA again.

A lack of staff parking and faulty IT systems started the rapid descent into terminal chaos which has seen BA suffer a 5% fall in business class passengers and left it with a £16 million bill.

This, of course, will remind those of us who track such things on this side of the Atlantic of the 10-year failed effort by United Airlines to automate the baggage system at Denver International Airport. One can only hope that our British cousins get their problems solved and solved a bit quicker. ..bruce..

So now the Minnesota Department of Public Safety has reluctantly sued CMI to force CMI to make its source code available, which the MDPS apparently has a right to under the terms of its contract with CMI. Even as it pursues this suit, the MDPS claims in its pleadings that “[s]ource code review is not a generally accepted means of determining whether a scientific measuring instrument is fit for a particular purpose, nor does the BCA [Minnesota Bureau of Criminal Apprehension] believe that source code review is an appropriate means to determine whether this particular instrument works as represented by the manufacturer.”

However, it’s unclear how MDPS — which apparently has never seen the source code itself — can make that statement. On the contrary: it’s not at all unusual in IT-related lawsuits for source code to be reviewed to see whether or not certain functionality exists, how it is implemented, and what defects might occur during operation. In cases such as this, the manufacturer typically releases its source code under an “attorneys eyes only” designation, and the other side retains a technical expert who must sign a protective order (the legal equivalent of a non-disclosure agreement) before s/he can review and analyze the source code.

I suspect that CMI software engineers (and their managers) would really like to do some cleaning up (and rewriting) of the source code first, since issues of code documentation, architecture, design, implementation, performance, reliability, and user interface are all fair game. At the same time, the expert might well ask to see defect tracking logs or databases to see how many open defects existed for the version(s) in question. And there might even be issues of how well the software integrates and interacts with the hardware of the device itself.

However, CMI would be obligated to release the exact version(s) of the source code used in the device(s) in question, warts and all. This raises another issue for the DWI defense: has CMI practiced source code control with sufficient rigor to be able to produce the exact source code version for the serial number of each breath-test machine in question? If not, then some judges might well continue to dismiss the corresponding DWI cases, and the state of Minnesota could be compelled to upgrade all of its breath-test machines to a software version that can be reliably produced.

As with the electronic voting machine issue, this case should be interesting going forward and could ripple into other states as well. ..bruce..

Atos Origin is the information technology partner for the International Olympic Committee (IOC) with the job of designing, building and operating the invisible IT infrastructure that supplies results, events and athlete information to the media, spectators and the world.

It also designs the platforms for accreditation, transportation, hotel accommodation and other services without which the Olympics would fall apart.

On the 11th floor of Digital Beijing, a banner trumpets the slogan “At the Olympic Games, there is no second chance.”

Known and unknown risks are analysed and every system is backed up by a plan B, plan C, or even a plan D.

“We have a very high degree of redundancy (backup),” said Patrick Adiba, the firm’s vice president responsible for the Olympics and major events.

That backup includes an entire replica data centre set up at a secret location elsewhere in Beijing that will kick in if the main centre fails.

“We just have to be ready with a response to known and unknown problems,” said Adiba.

Atos Origin has been running the IT platform for every Olympics since 2002 Salt Lake City Winter Games and has signed on to do the same for the 2010 Vancouver Winter Games and the 2012 London Summer Olympics.

Note at least three key success factors that appear in the brief excerpt above:

• A high degree of redundancy in systems and processes
• A strong focus on testing, again for both systems and processes
• A vendor who has performed this exact same task (running an Olympics) multiple times

All things to keep in mind for your next large-scale IT systems development effort. ..bruce..