RISE: The Psychology of Computer Programming (Gerald M. Weinberg, 1971/1998)

[The first of a planned series of posts on “Readings in Software Engineering“]

[Version 1.1 of this post, revised/extended on 05/22/2012]

The Psychology of Computer Programming, Gerald M. Weinberg, Van Nostrand Reinhold Company, New York, 1971. Hardbound, 288 pages. Personal acquisition date: 17 Oct 1978. Original edition out of print.

The Psychology of Computer Programming (Silver Anniversary Edition), Gerald M. Weinberg, Dorset House Publishing, New York, 1998. Softbound, 292 pages. Personal acquisition date: unknown. Currently in print (and in ebook form).

Seminal work; highest recommendation.

===========================

(All quotes and citations are from the 1998 Silver Anniversary edition, which contains the unedited text and original page numbering of Weinberg’s original book interspersed with his observations 25 years later using a different pagination scheme.)

If The Mythical Man Month (Frederick P. Brooks, Jr., 1975/1995) is the book that most IT managers own but few ever read (and even fewer follow), The Psychology of Computer Programming (Gerald M. Weinberg, 1971/1998) is the book that almost nobody nowadays owns…at least, nobody under a certain age. And that’s a tragedy, because Weinberg wrote one of the first and still one of the best works on the human factors in IT project management and software engineering.

I bought my own first copy of Weinberg’s book in the fall of 1978, nearly 35 years ago, just a few months after joining General Dynamics with my freshly-minted BS in computer science from BYU. Weinberg’s book had a tremendous impact on me and forever changed my view of software development from a technical activity to a human, social activity. In fact, Weinberg’s first sentence in his first chapter is just that: “Computer programming is a human activity.” That single insight — with all it implies — is something that the IT industry seems to re-discover on a regular basis and then forget again just as quickly as it seeks the ever-elusive magic amalgam of technology, methodology, and language (with minimum pesky humans) that will make software development fast, cheap, and good. Weinberg touched upon that very quest early on in the book as well :

[1971] Over the years, executives have backed their desire to eliminate programmers with staggering funds. Dozens of simplistic schemes have been heaped with money and praise on the promise — as yet not kept — of going directly from a sales proposal to a working data-processing system. (p.4)

[1998] The only thing that’s changed here in twenty-five years is the fact that the funds dedicated by executives to eliminating programmers from their payrolls have become far more staggering than I imagined back then. (P1.i)

Furthermore, Weinberg got to many of Brooks’ key insights four years before Brooks published The Mythical Man-Month, including touching (in a slightly different way) on the most famous one: adding more people does not get a job done more quickly, and using exactly the same analogy that gave Brooks’ book its title:

For example, certain programming work cannot be done by a team of trainees, no matter how large, so that doubling the number of “warm bodies” — as they are so often called in the trade — still will not get the work done. Schedule is similarly limiting — we need only cite the apocryphal experiment which tried to make a baby in one month by putting nine women to work on the job as a team. (p. 68)

As per the table of contents given at the end of this post, Weinberg covered four major themes in his book: Programming as Human Performance; Programming as a Social Activity; Programming as an Individual Activity; and Programming Tools. As Weinberg himself points out (in his 1998 commentary on his 1971 work), he was really writing more about the anthropology of computer programming, rather than the psychology per se. The book is highly readable, largely because Weinberg has a conversational style of writing and fills the book with actual stories and anecdotes of observed behavior on programming projects.

Here is just a small sampling of the insights from Weinberg’s book (my comments are in brackets and italics):

Fisher’s Fundamental Theorem states — in terms appropriate to the present context — that the better adapted a system is to a particular environment, the less adaptable it is to new environments. (p. 21) [Thus anticipating the core problem in the entire OO/reuse movement of the 1990s]

The organization chart is a nice toy for a manager, but little programming work would ever get done if interactions among programmers had to follow its narrow, straight lines. . . . But human interactions are never narrow, never straight, and hardly ever in the directions shown on an organization chart. Many serious mistakes have been made in imagining that formal structure was the only structure in an organization. (p. 48)

A programmer who truly sees his program as an extension of his own ego is not going to be trying to find all the errors in that program. . . . And let there be no mistake about it: the human eye has an almost infinite capacity for not seeing what it does not want to see. (p. 55) [Weinberg then goes on to discuss his somewhat controversial (at the time) concept of ‘egoless programming’, which really boils to to getting other programmers to review your code, now considered to be a best practice: ‘many eyes’, pair-programming, end-of-day review swaps, etc.]

[By having programmers review each other’s code:] Not only is the variation in debugging time reduced, but since there is more than one person familiar with the program, it is easier to get realistic estimates on the amount of real progress that has been made. It is not necessary to rely on a single judgment — and of the person least likely to be unbiased. (p. 59) [Weinberg goes on to note that you generally end up with more readable code, and you lower the impact of turnover within the programming group, since more than one person understands the code in question.]

Managers tend to select themselves from the “aggressive” component of society and have difficulty appreciating the fact that other people do not completely share their goals of money and prestige. They are especially at a loss to understand the smooth functioning of a programming group based on mutual respect for individual talent and cooperating in the common cause. Instead, they tend to view people as working for money or under threat — as they themselves do. (p. 61) [Some things are timeless.]

So many failures to meet programming deadlines can be traced back to an initial schedule and plan of attack which assumed the most optimistic conditions — no days lost through illness, no machine trouble, no compiler problems, no “impossible” bugs. Because each of us has had the experience of, say, a six-month period in which everyone on the team was in top health or there was no machine trouble or there were no compiler problems or there were no “impossible” bugs, we can slip into imagining — if the price is right — that we can have a six-month period in which all of these fortunate circumstances happen at once. (pp. 68-69) [Again, plus ça change…]

[After a story of upper management trying — unsuccessfully — to force a team leader into accepting an impossible schedule:] Did Harold lose his job? He didn’t, in this case; but that wasn’t really important, for he knew he could get another at least as good. If [keeping his job] had been important, he wouldn’t have been able to do what he did. One of the paradoxes of leadership is simply this: only the leader who is ready to step down has a real chance of success. (p. 85)

It is possible to be too smart for programming — if the person is not smart enough to use his intelligence to modify his social behavior and methods of communication. (p. 88) [Yes, there have always been prima donnas.]

In a sense, a programming project or team is like a river which remains the same river even thought its water is undergoing constant change. Many project managers are unable to grasp this view of a project. Their view of the project’s structure is, instead, much like that of a house — a structure that might collapse should one of the beams be removed. Their actions with respect to the people in the project — especially so-called “key” people — reflect this static view, often with disastrous consequences. (p. 96)

[Telling a story about the ‘filtering’ process of reporting project progress up a chain of command:] The net result of six or seven stages of such filtering was a report that monthly presented a consistent forward progress, a few areas slightly behind or slightly ahead, a few problems solved from last month, a few new problems, and a few problems still open. There was, in short, no measurable relationship between what had been reported at the bottom and what came out the top. (p. 101) [Thus describing what I years later termed “the thermocline of truth“. Note that Weinberg himself commented on my post and argued for the gradual layer-by-layer changes that he describes in his book; I countered with two real-world examples where a single individual was responsible for most of the filtering.]

Pressure from higher up is classically recognized by management manuals as both the way to get work done and the way to destroy a reporting system. (p. 103)

Any testing group is also put into a difficult relationship with other groups, because it is their job to criticize. (p. 107) [Thus capturing the classic tension between QA and development, though he goes on to describe ways to overcome it.]

When a supervisor is responsible for work he does not understand, he begins to reward workers not for work, but for the appearance of work. Programmers who arrive early in the morning are through to be better programmers than ones who are seen to arrive after official starting time. Programmers who work late, however, may not be rewarded because the manager is not likely to see that they are working late. Programmers who are observed talking to others are not considered to be working, because the manager has an image that programming work involves scratching out secret messages to the computer. (p. 110) [Again, some things just don’t change.]

Programs, like any other man-made objects, are designed — or should be designed — with a definite lifespan and scope of application in mind. . . . a program should have neither overdesigned or underdesigned parts. Yet it is an occupational disease of programmers to spend more time on those program parts that present, for some reason, the most intellectual challenge rather than on those that require the most work. (p. 126)

Another fallacy which we shall have to lay to rest is that “programming” is some sort of uniform effort requiring a set of uniform talents. For the professional, at least, the job from getting to specifications to delivered program demands various kinds of work, wihch, in turn, demand various talents. (p. 132) [Something which I subsequently discussed in the ‘talent’ section of my TEPES post a few years back and some 37 years after Weinberg’s observation.]

Programming is often described as a process moving from problem definition through analysis to flow diagramming, then coding, followed by testing, and finishing with documentation. Although this rough view contains some truth, it distorts the truth in several ways. First of all, the actual sequence is not so fixed, because, for example, documentation may precede testing, coding, flow diagramming and even analysis. Second, not all steps need to be present, as when we are recoding a program for a new machine or language. Thirdly, it need not be a sequence at all — and, in actual practice, rarely is. Who has not experienced a problem definition that changes as discoveries are made in analysis, flow diagramming, coding, testing, and documentation? Or who has ever seen a flow diagram that remains unmodified throughout the coding — or code that remains unmodified throughout testing? (p. 133) [Yes, Weinberg in 1971 is dismissing the ‘waterfall’ model and arguing not just the need but the inevitability of incremental/iterative development.]

Although the average programming manager would say that intelligence is more important that personality in programming success, very few could cite cases of people who turned out to not to be intelligent enough to program, but everyone knows of cases of people who were not temperamentally suited to the programmer’s job. It is in this sense that we can assert that personality is more important than intelligence in programming. (p. 148)

[Regarding ideal personality traits for programmers]…we can probably say with assurance that someone without the ability to tolerate stressful situations for a period of a week or more is not good programmer material…people who are not in some measure adaptable to rapid change will probably have trouble as professional programmers….one of the most easily identifiable personality needs in programming is a modicum of neatness. We are not speaking here of personal grooming…What we mean is a slight compulsion to keep one’s papers in order, without which the computer’s paper-generating capacity inexorably leads to grief….Another essential personality factor is at least a small dose of humility. Without humility, a programmer is foredoomed to the classic pattern of Greek drama: success leading to overconfidence (hubris) leading to blind self-destruction….The other side of the coin of humility is assertiveness, or force of character. A programmer’s job is to get things done, and getting things done sometimes requires moving around obstacles, jumping over them, or simply knocking them down….Last among the essential personality traits for programming, we might list sense of humor. The computer “doth make fools of us all”, so that any fool without the ability to share a laugh on himself will be unable to tolerate programming for long. (pp. 149-150) [Still probably one of the best summaries of what makes a great programmer, and particularly a great co-worker on a programming team.]

One of the best known and accepted results of motivation research is that increasing “driving force” will first increase performance to a maximum, beyond which addition of further driving forces will quickly drive performance to zero. (p. 182) [Can you say “death march“, boys and girls? I knew you could.]

The fear of new things, the expectation of failure, and the reluctance to admit weakness all have a direct retarding effect on learning, whether in a formal classroom setting or on the job. (p. 191)

A programmer would not really be a programmer who did not at some time consider his program as an esthetic object. This part is not quite symmetrical; that part is clumsy and doesn’t flow in an appropriate manner; the whole thing does not look proper on the page. To be sure, it is fashionable among programmers to be rough and tough and pragmatic, but deep down each programmer knows that it is not enough for a program just to work — it has to be “right” in other ways. Later, when we discuss language design and program testing, we shall see that the correlation between the esthetic and the pragmatic value of a program is not accidental — the more pleasing to the eye and mind, the more likely to be correct. (p. 209) [Thus anticipating the whole emergence of the “elegance” value in software design and implementation.]

What these quotes — which represent a tiny fraction of the book — don’t contain are the aforementioned anecdotes. Weinberg illustrates point after point with real-world stories that — aside from the archaic technology — sound very contemporary and very true to life, and are usually entertaining to boot.

Weinberg also has questions for IT managers and for programmers themselves at the end of each chapter. As he himself notes, most of these hold up very well (though there are what he terms some “hilarious exceptions”). Here are some examples

[For managers:] On what basis do you reward programmers? Are certain of your criteria mutually contradictory, as in asking for efficient but general programs? How explicit are you with your programmers in indicating what you are looking for in their programs? Or do you just tell them that you want the programs to be fast, small, neat, easily modifiable, errorless, and done in a week? (p. 25)

[For programmers:] When you have just found a bug, do you ever sit back and try to trace out the paths you took in your mind? Try doing this on the next bug you find, and write a brief report or outline on what you find. (p. 40)

[For programmers:] Do you refer to your work as “my” program? Try passing one week without using the personal possessive in reference to programs, and take notes on the effects you observe. (p. 65)

[For managers:] Do you ever do things to try to inflate the appearance of your technical competence in front of the people who work for you? Describe some of these incidents, and also some incidents in which it was discovered that your technical competence was in at least one respect inferior to one of the people who work for you. What were the consequences of that discovery, and do they justify attempting to cover up? (p. 92)

[For programmers:] Has your manager ever done anything to make you doubt his honesty? If so, describe the incident, what ultimately happened to your doubt, and how your work was affected. (p. 93)

[For managers:] Which do you reward most, accurate information or pleasing information? Do your programmers know what the information you require is used for? Do they see the final reports which are the destination of their information? If not, why not? (p. 114)

[For programmers:] Have you ever yielded to group or managerial pressure against your best technical judgment? Describe the situation and the consequences. (p. 115)

[For programmers:] Make a list of things that your software does for you that fifteen years ago you would have have to do for yourself. Or don’t you know anything about what software was like fifteen years ago? Do you think a professional should know something about the history of his profession? (p. 138)

[For managers:] Do you use any aptitude tests now in choosing programmers, or does your personality department use them? If so, what do you know about their validity? Do you make any effort to validate them by evaluating programmers after a period of time on the job? What methods do you use for this evaluation? How convinced are you of their effectiveness? (p. 177) [Still very relevant questions, particularly for the organizations that like to use the ‘brain teaser‘ approach to interviewing programmers.]

[For programmers:] Recall from your experience a “tiny” error that had a big ultimate cost. What debugging tools or techniques could have prevented that error? Would their cost have been greater than the cost of that one error? (p. 271)

Weinberg — who was only 38 years old when this book came out — would go on to write “more than 40 books and over 400 articles“, and he is still going strong.

In short, Weinberg really was there first with an overarching view, set down in nearly 300 pages over 40 years ago, of the human and social factors that affect software development and project management. As the excerpts above show, his observations remain utterly relevant for today. Professionally, I have been dealing with troubled or failed IT projects, either as a consultant or as an expert witness, since 1995. What I can tell you is that Weinberg and Brooks (whose book I’ll review next) pretty much nailed all the core issues some 40 years ago. During those four decades, tens of billions of dollars — if not a great deal more — have been wasted on failed or sub-performing IT projects. A lot of that money lost — both directly and indirectly — could have been saved if folks just read — and believed, and followed — these “old” books. ..bruce..

TABLE OF CONTENTS (adapted from the 1998 Silver Anniversary Edition)

PART 1. PROGRAMMING AS HUMAN PERFORMANCE

1. Reading Programs

2. What Makes a Good Program?

3. How Can We Study Programming?

PART 2. PROGRAMMING AS A SOCIAL ACTIVITY

4. The Programming Group

5. The Programming Team

6. The Programming Project

PART 3. PROGRAMMING AS AN INDIVIDUAL ACTIVITY

7. Variations in the Programming Task

8. Personality Factors

9. Intelligence, or Problem-Solving Ability

10. Motivation, Training, and Experience

PART 4. PROGRAMMING TOOLS

11. Programming Languages

12. Some Principles for Programming Language Design

13. Other Programming Tools

PART 5. EPILOGUE

About the Author: bfwebster

Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com.