My latest Baseline column  is up, and it talks about why you should read these five books now, if you haven’t already…and if you have read them, you should probably re-read them.  ..bruce..

The second column, “Do Not Defer The Difficult in IT Projects”, describes the all-too-human tendency in IT development to put off dealing with the toughest problems until last — at which point, you may not be able to solve them all. It also explains why so many IT projects get 80-90% “done” and then suddenly slip for weeks or months without making much progress.

Feedback is always welcome.

Categories: managerial

The impulse to constantly add new and incremental features to a software program certainly isn’t unique to modern software develoment, or to a particular technology or methodology. It derives largely from three sources. Upper management and marketing want, and sometimes need, those additional features to win a customer, to better position the product against a competitor, or to better justify the expense of an in-house project. Engineers often find it far more fun, more interesting, and more rewarding to add or extend features than to make existing features work completely and perfectly. And customers supply a constant flow of demand for new and improved functionality.

The problem is that many modern technologies and methodologies intensify all these tendencies. With a decent software design and implementation, based on current technology, it can take literally only a few minutes to add or extend features. Frankly, that’s the payoff that lures many engineers to modern technologies in the first place: They get to do really neat stuff, do it quickly, and impress the boss while they’re at it. The boss, having seen how quickly engineers can extend or create functionality, doubles or trebles the list of must-have features. The technical manager, caught in the middle, wonders how to deal with all this.

Symptoms

Engineers focusing on adding new features instead of getting old ones working completely. Upper management passing down long (or even short) lists of additional features when the engineering schedule will be hard-pressed to accommodate the current ones. Customers failing to distinguish between what they’d like and what they’re willing to live with.

Consequences

Features that don’t work as completely or well as they were intended. Missed milestones and schedule slippage.

Detection

Review all existing features with the engineering team and get an honest assessment of where each feature is and what it will take to complete it. If necessary, do a design and code review for each feature. Compare the documented “essential features” list—and the schedule required to complete them—with the current feature set as well as those proposed by engineers and upper management, and find where the differences are coming in.

Extraction

Nothing fancy here: Get the absolute “drop dead” release date for the software and work backward from there, allocating time to design, implement, debug, and test each feature that is not yet completed. Show everyone (especially customers) the gap between the time and resources allocated those required for just the currently planned and requested features. Use whatever process works in your company for performing feature triage: keeping those that are absolutely essential, dropping those that can wait for the next release, and negotiating on the others.

Prevention

Use the process given in Extraction before a line of code is ever written. As features are proposed, collect them in a list, but do not allow any work on them. At intervals, repeat the process: You might have time (or need) to schedule some of the proposed features, but you may well have to drop previously required ones.

As with a prior pitfall on features, this pitfall is so an obvious and well-known that I’m almost embarrassed to include it here. Again, it is still so common that I must.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: managerial

Why would the use of a new technology or methodology (the “TOM”) cause managers and developers to neglect or even abandon solid software engineering practices? Because those practices are under pressure from the start. Many engineers don’t know them and aren’t willing to spend (or aren’t given) the time to learn them. Those who do know them often aren’t willing to spend (or aren’t given) the time to practice them. Technical managers, who often do understand such techniques, find themselves caught in a vise, trapped between ever-increasing (and possibly unrealistic) demands of upper management and the productivity level of the engineers they supervise.

The introduction of a new TOM can cause a new set of problems. Upper management, having read articles about the benefits of this particular TOM, may see it as an opportunity to increase demands. Developers, already under pressure to produce, milk the benefits of the TOM for all they’re worth and may find even less incentive to focus on solid software engineering. And the technical managers in the middle have to cope with the specifics of the TOM while still fighting the software engineering battles.

Symptoms

Upper management expects the TOM to immediately shorten the software development lifecycle and is unwilling to allocate the necessary time and resources pilot projects. Engineers neglect common sense and best practices, feeling instead that the TOM somehow magically supersedes both. Unrealistic expectations from upper management puts pressure on the development team, resulting in even more shortcuts.

Consequences

Failure to achieve expected benefits of the TOM, possibly leading to (unwarranted) rejection of the TOM for future projects. Delayed, buggy, and/or failed projects.

Detection

It’s usually not hard to detect the pressure from upper management; indeed, the problem is resisting the pressure. As for the engineering-level problems, the best detection comes from regular team meetings and project reviews to go over standard (TOM-independent) software engineering practices.

Extraction

This pitfall is a hard one to get out of, because upper management needs to allocate more time and resources, and the engineers need the self-discipline and education to change how they’re doing things. The key is to educate everyone about the benefits of good software engineering — established practices, predictable schedules, and higher quality — and then work the TOM into that setting.

Prevention

It is sad that there is often so much resistance to doing things right in the first place. The approach is direct: realistic scheduling, definition of engineering standards, and enforcement of engineering practices. For starters, make everyone — especially upper managemen t– reads The Mythical Man-Month by Frederick P. Brooks. Then refer to it again and again as these problems arise.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

CATEGORIES: political

You know the drill. By hook or crook, through long weeks and late hours and ruthless compromising, you finally deliver the project. It’s finished, it’s out the door, and you have taken a few weeks to remind yourself what real life is like. Now you have the opportunity to go back and make right all the hacks and shortcuts and ugly patches you used to get version 1.0 of the project to the customers. Besides, having done the project once, you now have a far better idea of how it should have been done in the first place. You rub your hands together and…

…you are handed a list of new features required by customers or potential customers for version 2.0 of the project, which has to ship far sooner than you would have thought. After some study, you realize that you’re not going to be able to deliver all those features within the required time frame — and even as you realize that, more features get handed down.

Your plans for cleaning up the code and the architecture are rapidly getting pushed off to version 3.0, and you really wonder whether things will be any different then.

Chances are, they won’t.

Why does this happen so often? There are several reasons, really. One may be economic necessity. Your company (division, group) may have to fund itself, which means that sales have to be sufficient to meet all expenses, including your salary and benefits. If that’s your situation, then don’t worry as much about this pitfall; architectural purity is nice, but a paycheck is better.

Another, less acceptable reason may be a short-term focus on the part of upper management. Determined to milk a current market opportunity and concerned primarily about next quarter’s results, they may not be receptive to an engineering investment that would yield more in the long run at the expense of smaller profits right now.

A third reason — not necessarily exclusive of the other two, and sometimes justified — is a suspicion on the part of upper management that engineers are more interested in doing something “cool” or “elegant” than in doing something profitable. What upper management may not understand is that elegance — architecture and code that are concise, yet clear and comprehensive, tending toward orthogonality — always pays off. Always. The only question is how soon and how much, and that is the fulcrum for the balancing act of the technical manager. Ironically, when the engineering staff is able to rapidly deliver the features requested by upper management, it is often because of a previous investment in elegance; likewise, when features take a long time to implement or bugs take a long time to fix, it’s often because of architectural gaps and past short-cuts.

Symptoms

Constantly having to push engineering work into the next planned release and not the current one.

Consequences

The cost of adding new features or extending current ones goes up, not down, over time. The program gets larger, less stable, and less cohesive. Bugs multiply, possibly causing a customer backlash.

Detection

Sit down with upper management and detail the engineering work that needs to be done. Indicate what impact this will have on the current list of desired features. See what kind of reaction you get.

Extraction

It ain’t easy. The reaction you get under Detection should give you a pretty good idea of what things will be like. There are two approaches, which should probably be used together. First, actively work with upper management and customers to determine which features are absolutely necessary and which are merely desirable — ones they could live without until the next feature release. It’s critical to talk directly with the customers, because they may not have done their own prioritizing; the list you get from management may reflect everything from essential requirements to blue-sky wishes.

Second, build engineering cleanup time into each feature so that an engineer allocated four weeks for a given feature spends, say, three weeks on the feature and one week on architecture.

Prevention

Besides using the steps described above in Extraction, you need to emphasize the need to make the proper ongoing engineering investment in order to gain the desired benefits, particularly if a new technology or methodology is involved. This means investing the time to educate management and customers, to set proper expectations, and (if possible) to run a trial project to demonstrate the process.

Contributor/References

Webster, from Pitfalls of Object-Oriented Development (1995).

CATEGORIES: political, architectural, managerial

While discussing the challenges of software development with Taligent trainer Tom Affinito back in the mid-1990s, I mentioned — citing Fred Brooks — the need for a chief architect. Tom immediately responded, “Yes, and ultimately architecture is a political act.”

That observation has stuck with me ever since. Unless you are both sole architect and sole implementor — and maybe even then — architecture is a political act, and it is especially true in development groups of any size. Developers like to think themselves above politics. Not so; they are as human as anyone else. Developers can be quite political and will use a variety of techniques, good and bad, to achieve their ends: persuasion, hyperbole, fear-mongering, caucusing, lobbying, consensus building, rumors, backbiting, leadership, intimidation, sacrifice, tantrums, threats, and subversion, to name a few. The fact that developers are generally very bright just increases the effectiveness of whatever techniques they use.

Architecture is the lodestar of developer politics because it is ultimately the most prestigious role in software development. Brooks goes to great lengths (and rightfully so) to assert the intellectual and creative challenges of implementation, noting that architecture is not the be-all and end-all. Still, it is Frank Lloyd Wright’s name that remains attached to the structures he designed, not the name of the contractors who built them. Or, to update an old programming joke, the verb “to develop” is conjugated this way: “I architect, you implement, he tests.”

Yet we may fail to recognize or acknowledge this situation and its implications. Why? First, we want to believe that all developers have (with regard to the project) the same intentions and goals. Second, we want to believe that the best course is the one that is obvious to us and that everyone else will agree with that. Third, we don’t want to have to deal with politics ourselves, particularly politics that can lead to confrontations.

Symptoms

Assuming — without verification — that all other members of the development group have the same goals, ideas, and talents. Not wanting to assert architectural issues. Finger-pointing as design problems arise.

Consequences

Hidden or overt team dissension, leading to poor morale and lack of cooperation as well as a weakened architecture.

Detection

Ask yourself these questions:

• Who is the chief architect of this project?
• How well do the other developers support the chief architect?
• How effective is the chief architect in her role?
• What are the obstacles she faces?
• What are all the political issues involved?

Answering these questions should go a long way toward determining whether you have underestimated the politics of architecture.

Extraction

The first issue to address is that of authority commensurate with responsibility: if there is a chief architect on the team, does she really have the authority to compel adherence to the architecture? This kind of authority is different than being able to say, “Do this or you’re off the project,” which often leads to subtle or overt rebellion by team members. This kind of authority accrues from several factors:

• a proven track record
• respect from teammates
• support from technical and upper management
• a mutual pact with the developers that the chief architect will give serious consideration to all their ideas and suggestions but that they will abide by her decisions
• a willingness to give in on the small things so that she can hold her ground on the big things

Beyond that, healing a team that is experiencing jealousy and dissension is no easy feat, especially when you’re in the middle of a project. Books and seminars on team-building abound; study, learn, and apply.

Prevention

Go through the questions in Detection but cast them in the future tense. Then, as with Extraction, proactively work to build the team and establish the authority of the chief architect.

Contributor/References

Webster, from Pitfalls of Object-Oriented Development (1995).

…the computer-operated baggage system has crashed and luggage is now being sorted manually before being loaded on to planes.

Twelve return flights to short-haul destinations have been cancelled. . . .

In just over a week the £4.3 billion investment has seen an unimaginable series of PR catastrophes.

Around 19,000 lost bags had to be transported to Milan to be sorted, hundreds of flights have been cancelled, and thousands of furious passengers have sworn they will never fly BA again.

A lack of staff parking and faulty IT systems started the rapid descent into terminal chaos which has seen BA suffer a 5% fall in business class passengers and left it with a £16 million bill.

This, of course, will remind those of us who track such things on this side of the Atlantic of the 10-year failed effort by United Airlines to automate the baggage system at Denver International Airport. One can only hope that our British cousins get their problems solved and solved a bit quicker. ..bruce..

Atos Origin is the information technology partner for the International Olympic Committee (IOC) with the job of designing, building and operating the invisible IT infrastructure that supplies results, events and athlete information to the media, spectators and the world.

It also designs the platforms for accreditation, transportation, hotel accommodation and other services without which the Olympics would fall apart.

On the 11th floor of Digital Beijing, a banner trumpets the slogan “At the Olympic Games, there is no second chance.”

Known and unknown risks are analysed and every system is backed up by a plan B, plan C, or even a plan D.

“We have a very high degree of redundancy (backup),” said Patrick Adiba, the firm’s vice president responsible for the Olympics and major events.

That backup includes an entire replica data centre set up at a secret location elsewhere in Beijing that will kick in if the main centre fails.

“We just have to be ready with a response to known and unknown problems,” said Adiba.

Atos Origin has been running the IT platform for every Olympics since 2002 Salt Lake City Winter Games and has signed on to do the same for the 2010 Vancouver Winter Games and the 2012 London Summer Olympics.

Note at least three key success factors that appear in the brief excerpt above:

• A high degree of redundancy in systems and processes
• A strong focus on testing, again for both systems and processes
• A vendor who has performed this exact same task (running an Olympics) multiple times

All things to keep in mind for your next large-scale IT systems development effort. ..bruce..

CATEGORIES: organizational, conceptual

The software development process–creating software to solve a particular problem–is long and complex and has many activities and stages. The exact list will vary depending on what book or article you read but can generally be said to include the following:

• becoming aware of and identifying the problem
• deciding to solve the problem
• deciding what resources (time, money, people, mind share, opportunity, authority) to expend solving the problem
• securing and managing those resources for the duration of the project
• defining the scope of the resulting project and getting it started
• analyzing the problem
• defining the solution
• designing the solution
• implementing the solution
• testing the solution
• verifying that the solution solves the problem in an acceptable manner
• delivering the solution
• modifying and refining the solution to solve new problems or fix existing ones
• performing additional necessary quality assurance activities (development standards, deliverable reviews, configuration management, etc.)

Far more detail is possible here, including issues of communications, skill level, politics, and user feedback, but you get the idea. And each item has its own set of problems, challenges, and, yes, pitfalls that are quite independent of the development methodology used.

So, here’s the problem: How many of these activities will the new technology or management (“the TOM”) actually affect favorably, especially if you haven’t used the TOM before? By contrast, how much training and experience will be required to effectively apply the appropriate TOM skills in each of these areas?

Symptoms

People who say, in effect, when problems are raised, “Don’t worry, we’re using TOM; that’ll solve this problem.” I’m not sure which is more frightening: when nontechnical people say this or when technical people do. As Fred Brooks seems destined to have to prove continually, there is no “silver bullet” to slay the software development monster.

Consequences

The project slips or even fails when you discover not only that many problems are not solved by using the TOM but also that they’re made worse by doing so.

Detection

Go through the list above and test your own understanding of each of these areas and the challenges they present. Then do the same with all the other key people in the projects at all levels.

Extraction

Education of those who misunderstand. This can be difficult if they are upper-management people who have approved projects based on what turn out to be false reassurances. It can be even more difficult if these people have passed on those reassurances to others.

Prevention

Establish a process or methodology that takes you through the list above (though not necessarily in a linear fashion). List the tasks, resources, and challenges associated with each step. Judge the impact–or lack thereof–that the TOM will have in each area. Be especially aware of the investment in training and skill development that will be required at each level as a result of adopting the TOM. And, most of all, identify all the problems that the TOM will do nothing to solve and may actually make worse.

Contributor/References

Webster, adapted from Pitfalls of Object-Oriented Development (1995).