Categories: managerial

Self delusion and group delusion are all too common in software development projects. Several factors combine to bring this about. One is the natural optimism prevalent among software engineers, particularly when they are not allowed, encouraged, or required to spend sufficient time specifying and designing what they will develop. Another cause is the presumption that no major roadblocks or difficulties will be encountered along the way, either technical or organizational. A third factor is the persistent (if illogical) attitude on the part of upper management that because a project must be completed by a certain date, therefore it can and will be completed by that date, and information to the contrary is not welcome. Mix these and related factors and you have a major disaster in the making.

New technologies and/or methodologies (“TOMs”) have a way of magnifying these tendencies. Engineers become even more optimistic, technical managers see fewer roadblocks, and upper management, having read all the glowing articles about these particular TOMs in the business and trade publications, expects unrealistic results.

Symptoms

Irritation, anger, and disbelief when someone gives unpopular but honest appraisals of how things are going. Long, earnest explanations of why the standard rules of software development don’t apply in this case. And, of course, serious discrepancies between the planned schedule and actual results.

Consequences

Constant slips in the schedule, as expectations are continually reset to adjust to the new version of reality. Projects that ship very late, if at all. Loss of credibility for managers and developers. In the end, you end up looking dishonest, incompetent, or both.

Detection

Take aside the developers, managers, and others involved in the project, one at a time, and ask each person, “Do you think there’s anything we’re fooling ourselves about?” Make notes of the points each person brings up. Make a second pass through, but this time show them the list you’ve compiled and ask them what they agree with, what they disagree with, and what they’d like to add. The number and significance of items on the list should give you a good idea whether you and others are fooling yourselves about how the project is going.

Extraction

Given a significant list of items, you need to reschedule and plan again based on the information collected. This may not be easy or popular; in some cases, it may not even be possible, depending on how upper management reacts. Your choice may then be to either push ahead as best you can or find another job.

Prevention

Use the process described under Detection from the start of project planning and design. You may sacrifice popularity, but you’ll gain credibility—provided, of course, that they don’t reassign (or fire) you and put someone more amenable in charge.

References

[Professional experience]

Categories: managerial

This is a classic pitfall in software engineering. Typically, insufficient time is allocated for the problem specification, research, design, architecture, and review that should occur before coding and during each development cycle. Likewise, software quality assurance (SQA) is often given little time, money, or people. The whole focus is on coding, and often that is underestimated by assuming that development will be linear and that nothing will go wrong.

Adopting a new technology or methodology (the “TOM”) tends to compound this effect in several ways, some of which are discussed elsewhere. First, unrealistic expectations can creep in. Second, rapid prototyping and feature development can cause a false sense of progress. Third, coding time may well be genuinely reduced by the TOM.

As a result, an expectation can arise that all aspects of the development cycle, not just the coding portion, will be compressed. The first few times you use the TOM, it may require more time up front for design and architecture, and it’s likely to require as much or more testing time. As your development group gains more experience and expertise in the TOM, the entire cycle may start to compress, but that comes with time.

Symptoms

Non-coding tasks taking longer than the time allotted to them. Slowdowns during development due to a need to rethink design. Alpha and beta testing taking much longer than planned.

Consequences

Slipped schedules and missed deadlines. Rude surprises as design must be repeated or testing takes much longer than expected.

Detection

Apply Brooks’ rule of thumb: the time required for a project should break down into one-third for design and prototyping, one-sixth for implementing, and half for testing. If your proportions are radically different, then you may have misjudged the relative costs or at least, mislabeled them; a lot of design and testing gets buried inside implementation.

If you’re far along in your project, keep a close eye on time required for SQA and particularly for testing, which is usually underestimated. Agile methodologies do a better job of addressing testing up front, but SQA remains the poor cousin for most software projects.

Extraction

First, throw out your current schedule and do a hard reset of expectations, particularly among upper management. This is not easy to do, but honesty is always the best policy. Good luck.

Second, set up a development cycle—specify, design, prototype, review, implement, test—with the time allocated to each step in the cycle based roughly on the proportions given above. Recognize that the actual time for each step in each cycle will vary: An early cycle will tend to have more specification and design and less testing; a later cycle will reverse those proportions.

Third, manage through one complete cycle and see how well your estimated costs match reality. Adjust and repeat.

Prevention

First, get some project management software. It doesn’t have to be fancy; it just has to automate the task of adding the estimated times for each task, noting critical paths, and calculating a final date. This is important; attempting to schedule anything except the smallest and simplest project in your head will lead to unpleasant surprises.

Second, use steps two and three under Extraction to set up your schedule and to estimate relative costs. If possible, try this first with a relatively small project and then work up to larger projects. The goal is to be able to estimate both relative and absolute costs within a certain margin of error (say, 10 percent) on a regular basis.

References

Brooks, Frederick P., Jr. The Mythical Man-Month. Reading, MA: Addison-Wesley, 1995.

Webster, from Pitfalls of Object-Oriented Development (1995).

Well, now I’ve been hired by Ziff Davis Enterprises to write a weekly column on IT Management for the online version of Baseline. That column is titled “Surviving Complexity” and, again, draws upon work I’m doing for the book. My first column is up: “Lies, Damned Lies, and Metrics (Part I)”; here’s the opening paragraph:

When Capers Jones published Assessment and Control of Software Risks (Yourdon Press, 1994), he identified the most serious software risk in IT projects as “Inaccurate Metrics,” and the second most serious software risk as “Inadequate Measurement”. I remember being startled when I first read that back in 1995—they certainly weren’t what I would have chosen—and other authorities in the field criticized his choices. Yet, in the intervening years, I have moved closer and closer to Jones’ point of view.

Please feel free to check out the new column. Thanks. ..bruce..

Categories: managerial

The impulse to constantly add new and incremental features to a software program certainly isn’t unique to modern software develoment, or to a particular technology or methodology. It derives largely from three sources. Upper management and marketing want, and sometimes need, those additional features to win a customer, to better position the product against a competitor, or to better justify the expense of an in-house project. Engineers often find it far more fun, more interesting, and more rewarding to add or extend features than to make existing features work completely and perfectly. And customers supply a constant flow of demand for new and improved functionality.

The problem is that many modern technologies and methodologies intensify all these tendencies. With a decent software design and implementation, based on current technology, it can take literally only a few minutes to add or extend features. Frankly, that’s the payoff that lures many engineers to modern technologies in the first place: They get to do really neat stuff, do it quickly, and impress the boss while they’re at it. The boss, having seen how quickly engineers can extend or create functionality, doubles or trebles the list of must-have features. The technical manager, caught in the middle, wonders how to deal with all this.

Symptoms

Engineers focusing on adding new features instead of getting old ones working completely. Upper management passing down long (or even short) lists of additional features when the engineering schedule will be hard-pressed to accommodate the current ones. Customers failing to distinguish between what they’d like and what they’re willing to live with.

Consequences

Features that don’t work as completely or well as they were intended. Missed milestones and schedule slippage.

Detection

Review all existing features with the engineering team and get an honest assessment of where each feature is and what it will take to complete it. If necessary, do a design and code review for each feature. Compare the documented “essential features” list—and the schedule required to complete them—with the current feature set as well as those proposed by engineers and upper management, and find where the differences are coming in.

Extraction

Nothing fancy here: Get the absolute “drop dead” release date for the software and work backward from there, allocating time to design, implement, debug, and test each feature that is not yet completed. Show everyone (especially customers) the gap between the time and resources allocated those required for just the currently planned and requested features. Use whatever process works in your company for performing feature triage: keeping those that are absolutely essential, dropping those that can wait for the next release, and negotiating on the others.

Prevention

Use the process given in Extraction before a line of code is ever written. As features are proposed, collect them in a list, but do not allow any work on them. At intervals, repeat the process: You might have time (or need) to schedule some of the proposed features, but you may well have to drop previously required ones.

As with a prior pitfall on features, this pitfall is so an obvious and well-known that I’m almost embarrassed to include it here. Again, it is still so common that I must.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: managerial

Let’s start by freely acknowledging that, with rare exceptions, software of any complexity changes between original specification and actual delivery. This is to be expected, and to a certain extent encouraged, when the changes represent a refinement of our understanding of the problem domain and the required solutions. Hence Brooks’ assertion that we must be prepared to “throw one away” — we just don’t know for sure what we’ll need to build and how to build it until we’ve built one. One of the advantages of modern technologies and methodologies is that such changes and adaptations can often be made readily and at lower cost in time and risk than for earlier development approaches.

Let’s also acknowledge that, given the time required to deliver a significant piece of software, customer requirements, technical expectations, and market conditions may change dramatically between the time a project is specified and when it is delivered. As John Donovan noted many years ago, the distance between the leading edge of technology and the trailing edge appears to be constantly shrinking. You may need to redirect or revamp projects as they go along in order to respond to such shifts.

But that’s not what this pitfall is about. This pitfall is about specifications that aren’t nailed down, or that aren’t firmly specified at all. A given feature or aspect of the project is discussed (or maybe it isn’t). As time goes on, the person(s) implementing it have one idea as to what it means or may even have new and improved ideas for it. Those not implementing it may be assuming something quite different from what is being written. Those not involved—including, and especially, upper management—may have quite a different (and erroneous) idea of what the feature will be. And customer expectations may be something else again. When the time comes for feature completion, code integration, project review, or customer acceptance, all hell breaks loose.

Symptoms

Lack of specification documentation. Engineers showing off “new features.” Mismatched subsystems. Constant delays in completing given features.

Consequences

Schedule slips due to ever-changing specification and integration problems. Lack of management, marketing, or customer acceptance of the delivered project.

Detection

First, see whether you can put your hands on a detailed list of features. If no such list exists or if it is incomplete or vague, then you have problems.

Second, see whether you can find a detailed description for each feature, including clear information about what it will not do. Missing or incomplete information is another danger sign.

Third, compare the specification against what is actually being implemented. Variations are yet another indication of trouble.

Fourth, propose a change in specification and see what is the process for making it. The more informal and undocumented the process, the greater the danger of this pitfall.

Extraction

The Detection exercise should give you a good idea about your current state and how to get out of it: Produce the documents described and then use them as a hard spec for the project.

A second (not necessarily exclusive) approach is to write a complete user’s manual for the project and let that be the final specification. This is what we did many years ago at Pages Software: the user documentation, complete with screen shots, was printed a full nine months before the product was released. The president and CEO of Pages gave a copy to every engineer and said, “Here’s the final specification.” It saved all of us from the temptation of making further changes.

Prevention

Establish a process to create, review, modify, and enforce the specification. This is, in effect, the documentation of the design phase of your methodology. This process won’t be easy or popular, but it will pay off significantly in the long run.

This is such an obvious and well-known pitfall that I’m almost embarrassed to include it here. But it is still so common that I must.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: managerial

If your organization is adopting some new technology or methodology (the “TOM”), it is likely because of wonderful claims about how it will improve your software engineering efforts: faster development time, higher quality, lower complexity, and so on. Leaving aside the likelihood of achieving such claims, these hypothetical benefits still inspire or tempt managers and developers alike to leap feet first into adopting the TOM, using it to tackle large projects with short deadlines.

The result is like crossing a swamp. The ground starts out a bit damp and the undergrowth a bit thick, but you can make good progress with your new TOM machete. As you push ahead, however, you face mud, vines, snakes, water, and alligators in ever- increasing proportions. You may choose to push ahead or to go back, but in either case the effort and cost are greater than if you had used caution before plunging in. And those above you want to know what happened to all the wonderful benefits of this TOM.

Symptoms

A project that doesn’t seem to be making much progress or one that seems ill-defined.

Consequences

Schedule slips and project failure. If initial adoption of the TOM in question (or some key variant thereof), loss of acceptance of that technology and loss of professional credibility.

Detection

Enumerate all that you hope to accomplish with this project. See how soon in the process the coding and prototyping started. Measure current status against the announced schedule and make a realistic assessment of where you are.

Extraction

This can be really tough. Once you’re in the middle of a project, you have a lot of people counting on you to deliver certain solutions by a particular date. You might be hesitant to stop everything and plan the entire development effort again. On the other hand, there is a good chance — in fact, almost a certainty — that if you have stumbled into this pitfall, your efforts to get out of it may not be fully understood or appreciated by those in authority.

Nevertheless, the best solution is to halt development and take the time to scale down the project and train developers; then you can set realistic deadlines.

Prevention

Look again at the pitfall: too much, too soon, too fast. First, start out small when using the new TOM for the first (or second or third!) time. Use well-defined, small-scale pilot projects. Or whittle down your large project into a small one on which you can build in later development cycles.

Second, take time for education, analysis, and design. Make sure that everyone involved understands the concepts, techniques, and — yes — the pitfalls of this particular TOM. Resist the temptation to start prototyping and coding immediately; go through an explicit, detailed analysis phase and take time to go through a design phase.

Third, plan for your initial TOM-based projects to take at least as long as they would if developed using traditional software engineering approaches. You may be pleasantly surprised; you almost certainly won’t have a rude awakening. After you get several projects under your belt, you’ll become more accurate at projecting development cycles.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: managerial

One of the defining moments in American politics during the 2nd half of the 20th century came early in the 1980 presidential campaign. Senator Ted Kennedy, heir apparent to the Kennedy legacy, was challenging his party’s incumbent, Jimmy Carter, for the nomination. Carter was seen as vulnerable to a Republican challenger, so Kennedy was ready to step up to fill the shoes of his two martyred brothers. Whatever his past troubles, he was still considered a shoo-in for the Democratic nomination and a credible response to the Republicans.

Then during an interview for a CBS news profile on Ted Kennedy, journalist Roger Mudd asked Kennedy why he wanted to be president. For only a minute or two — but for what must have seemed like an eternity — Kennedy struggled inarticulately, unable to utter a coherent sentence, much less a compelling, thoughtful response. It seemed that either the question had never occurred to him or he was unwilling to express his true reasons and unable to come up with an acceptable substitute on a moment’s notice. Whatever the case, his support peaked at that moment and slid downward from there. He lost the nomination to Carter, who in turn lost the election to Ronald Reagan.

What’s the point? Just this: If you’re going to adopt a new technology or methodology (the “TOM”), be sure you know exactly why you’re doing it and what you hope to get out of it. It is not uncommon for development groups to decide to adopt a new TOM because it is a Good Thing; yet they fail to establish what exactly they expect to get out of it. “Faster development,” “greater flexibility,” and “code reuse” are nice phrases, but so are “the flag,” “motherhood,” and “apple pie.” Many groups explore a particular TOM because it’s interesting and new. A pilot project may be well defined — for example, the creation of a simple custom application — but often no clear follow-up exists.

Symptoms

Lack of measurable progress; lack of defined deliverables; lack of understanding as to what comes next.

Consequences

Loss of focus and direction; pilot projects seem to drag on forever; real projects are constantly redefined.

Detection

Describe in writing the ideal state of this TOM in your firm in, say, two to five years (depending upon the size of your company and team). If this is hard to do, or if what you write seems vague, then you have a good indication that you’ve fallen into this pitfall.

Once you are satisfied with this statement, then describe in detail — working backward from your long-term goal — each of the steps to get you from where you are now to where you want to be. If you run into gaps or leaps of faith, then you know you’re facing this pitfall.

Extraction

Follow the process in Detection: work backward from your long-term goal in a series of steps, filling in any gaps that you encounter. Be very about identifying and defining the objectives for each step.

Prevention

Follow the process described in Detection and Extraction. You may end up deciding that your long-term goal is unrealistic; if so, adjust it. Remember that conditions will likely change over three to five years, so make sure that you aren’t locked into that single goal. Instead, make sure that at each step or phase, you have a chance to re-evaluate or reset the eventual goal and shift accordingly.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: political

Face it: technology adoption is a gamble, and one often made at gunpoint. External market forces lead internal political forces to demand advances — often unrealistic ones — in information technology. Those who adopt too soon come to understand the cliché “bleeding edge”. Those who adopt too late lose opportunities and possibly market share. And so you find yourself looking a dozen new technologies and trying to decide which to adopt, knowing that only two or three are likely to succeed: that is, deliver what you need and grow with your ever-increasing demands. A winner is always easier to pick in retrospect, but you seldom have that luxury. Instead, you need to decide between adopting a “safe” technology — which may not, in the end, really meet your needs — or leading out with a newer technology and thus run the risk of finding yourself in a dead end with a non-standard or even unsupported solution or approach.

But that’s why they pay you the big bucks, right?

Just as a personal note, I spent five years (1990-95) and $7 million in venture capital as part of a software start-up that developed and shipped a design-oriented desktop publishing system for . . . the NeXTstep operating system. As I tell people, it seemed like a good idea at the time. It was a great operating system (still is — go look at Mac OS X) and certainly better than anything else out in 1990. And we were in good company at the time, with firms such as WordPerfect, Ashton-Tate, and Adobe all developing for NeXTstep as well.

But it was the wrong horse.

Symptoms

Peers and managers above you start questioning your choice of technologies. The number of other organizations adopting the technology in question doesn’t seem to be growing much, or is even shrinking. Developers experienced with the technology are hard to find.

Consequences

Your system and/or development effort struggles due to the lack of developers and third-party tools for the technologies in question. If you’re developing a commercial product, you may find a very limited market, assuming you find one at all. If you’re developing an in-house system, you may end up deploying it, but the system may be difficult to maintain and keep in production for its original hoped-for lifetime. And if you were the champion for adopting that particular technology, you may find yourself losing influence, missing out on promotions, and possibly even out of a job.

Detection

Track down and talk frequently with other firms adopting the same technology. Fact-check, as best you can, claims by the technology’s vendor regarding adoption and market share. Pay particular attention to the products or systems based on this technology that are actually deployed into real-world use.

Extraction

If you think you’ve picked the wrong horse, then start to find the right one, quickly. Figure out what it’s going to take to move over to the better (or best) technology. Work up a careful transition plan — as accurate and conservative as you can make it — and then figure out what it’s going to take to sell this to upper management. There may be no good or easy paths at this point.

Note that there are cases where you may well be best off staying on the technology you originally picked, particularly if this is an in-house system. That is, it may be more cost effective to push into production, then plan for an earlier-than-expected end-of-life for that system, rather than try to switch horses in mid-race.

Prevention

This is hard precisely for the reasons stated above: in many cases, you just don’t know what will succeed and what won’t. If you’re talking about a major decision — such was what operating system to develop for and deploy on — your choices may be limited by other factors. In general, though, you want to avoid the leading/bleeding edges — never count on version 1.0 (or even 1.x) of anything. You want to choose technologies that have been out in the marketplace long enough to have the flaws exposed and — one hopes — the biggest problems addressed.

References

[Professional observation]

Categories: political

A particular technology or methodology (the “TOM”) is wonderful. At least, you think it is, based on anything from a breathless magazine article to years of experience with solid, successful software development using this TOM. Or you may not think it’s wonderful, but you do think it may offer advantages and benefits to your development efforts. And, of course, what’s obvious to you should be obvious—or, at least, understandable—to everyone else.

So you blithely push ahead…until people start pushing back. And before you know it, you’re engaged in a political battle, fighting resistance to the TOM that may range from guerrilla sniping to a full frontal assault.

Why might people resist accepting object technology? The reasons are varied and may range from the well-founded to the completely irrational to the deliberately obstructionist. Here are examples:

• They don’t understand the TOM.
• They don’t want to understand the TOM.
• They’re afraid they won’t be able to understand the TOM (and thus will have less value to the company).
• They know they won’t be able to understand the TOM, because they’ve been just skating by for the last ten years–taking advantage of the IT job shortage–and know that this will expose them for what they are.
• They have very strong feelings about the language and/or programming methodology, or tools or all three that they currently use.
• They really detest the TOM and/or the associated language and/or tools that you’re proposing they use.
• They’re worried that the reasons for adopting the TOM aren’t well thought through.
• They’re worried that the plan for adopting the TOM has significant problems or flaws.
• They want to adopt the TOM, but they want to do it their way.
• They want you to fail so that they (or someone they like better) can get your job.
• They read this book.

Compounding the situation is the fact that you may face several of the reasons simultaneously, possibly from different sources.

Symptoms

Delays in getting approval or support. Rumors circulating behind your back. Objections constantly brought up in meetings. Former supporters distancing themselves from you. Reduction in project priority and resources.

Consequences

Significant project delays or project failure. Loss of influence. Loss of job.

Detection

If you suspect that resistance is deeper or more entrenched than you thought, you need to find out the sources of resistance and the reasons behind it. This can be hard, because the people resisting you may not be honest about what they’re doing, or why they’re doing it.

Furthermore, if the resistance is coming from above, your boss(es) may feel no obligation to explain their reasons.

Extraction

You have four choices, not necessarily mutually exclusive, based on the nature, depth, and source of the resistance:

• Push ahead in spite of it.
• Mollify or enlist those who resist.
• Redirect the project to quell the objections.
• Abandon the project gracefully and (maybe) try again later.

Prevention

Make a list of everyone who might have any input or influence and judge where they stand. If possible, meet with each one privately to sound out her or him—but recognize that you may not get an honest answer. Offer each person a chance to make criticisms and recommendations; let everyone feel a part of the project and the process. Build enthusiasm, pointing out specific benefits, particularly those of interest to each individual. Use each source to cross-check others. This process is known as “getting your ducks in a row,” and you’d better do that before you start. This process may be more critical for those below you than those above you; don’t underestimate the power of developers to make your life wonderful or miserable.

Having done that, assess the costs and risks in pushing this project forward compared to the possible benefits and rewards. Factor that with the probability of success and make your call. If it looks too dangerous, scale back or redirect. If it looks impossible, find somewhere else to work.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: political

There is an oft-cited dictum in technology development groups: “It is easier to ask forgiveness than permission.” It is often true and sometimes crucial to circumvent bureaucratic foot-dragging and politics. But it is not always the best course, and the danger of following it is commensurate with the technical, financial, and political risks involved. And all three risks abound when an organization is moving to a new technology or methodology (the “TOM”).

Nevertheless, it is not uncommon for a development group to make the switch to the TOM with only minimal involvement and education of upper management. Indeed, management may not care very much or may even be supportive—having read an article about the wonders of this particular TOM in a weekly business magazine.

And that is where the pitfall lies. For unless the project comes in within acceptable time and budget limits, upper management will suddenly be asking hard questions reflecting reasonable or unreasonable expectations, given what they know or were told.

Symptoms

Apparent lack of knowledge or overly positive expectations or both on the part of management about the TOM and its role in the relevant projects. Sudden distancing or self- protecting activities if problems crop up.

Consequences

Lack of support and possibly active hostility from upper management if problems arise or if their expectations aren’t met. Finding yourself twisting in the wind. Career damage, lack of promotion, demotion, loss of job.

Detection

Sit down with upper management and find out how much they understand about this particular TOM and how it’s being applied in the relevant project(s). Have them detail their expectations. Find out their level of enthusiasm or concern for the use of this TOM.

Extraction

There’s little to do except start the education and enrollment process much later than it should have been. It may be really tough, depending on how current expectations compare to reality, and the truth is, there’s no good reason for not having done this before things got started.

Prevention

From the pitfall itself, it’s obvious that there are two separate (if related) tasks: educating and enlisting. The first comprises letting management know exactly what adopting this TOM entails, which means that you had better know yourself, and you’d better know it well enough to explain it to nontechnical people. Work to contain your own enthusiasm for this TOM; remember that it is safer to underpromise and overdeliver.

Second, and this is probably tougher, you need to enlist key people: those who can affect your budget and resources, and those who can affect the scope and direction of your project. If you can, prove yourself with small projects to establish credibility and to give your sponsors a track record to point to (and take credit for).

Reference

Webster, from Pitfalls of Object-Oriented Development (1995).