The post, ”Forensic Fraud: Now Available on Daytime TV” by Marielle Dirkx, highlights two cases. The first is the notorious Jodi Arias murder trial taking place in Arizona right now. At that trial, a forensic expert presented what he claimed was evidence from a photograph taken by Arias of her boyfriend, Travis Alexander, just before she killed him (something she admits, but claims was in self-defense). The expert blew up the photograph and purported to show a reflection of Arias in Alexander’s eye. Furthermore, he claimed that his enhancement of the reflection showed that Arias was not holding a weapon at the time she took the photograph.

See for yourself:

Obvious, right?

Clear, right? And from this processed enlargement, “According to the expert, Arias was standing a few feet away from Alexander, holding the camera with both hands at the time the picture was taken; and he further asserted that he could state with scientific certainty that she was not holding either a gun or a knife.”

Facepalm.

According to Dirkx, the expert’s testimony and evidence were not admitted.

The second cases is too detailed to summarize here, but it involves a “go-to” expert for prosecutors in Mississippi, a dentist who “routinely testified as a wound pattern expert, a trace metals expert, a gun shot residue expert, a gunshot reconstruction expert (he once performed a ballistics analysis by shooting dogs from the pound), a crime scene investigator, a blood spatter expert, a ‘tool mark’ expert, a fingernail scratch expert, and an expert in ‘liquid splash patterns.’” In the case in question, this dentist(!) used Photoshop and his PC to “enhance” surveillance video that  he claimed showed details about the alleged crime. Problem was, the FBI had processed the same video and said no such details appeared. Apparently, the prosecution did not see fit to share that FBI analysis with the defense or the court, and the defendants were sent to prison for ten years before their conviction was overturned due to that omission.

Go read the whole thing; it’s worth it.  ..bruce..

[Note: I am currently in transit from Colorado to Florida and am composing this post as I have time and 'net access.]

“All the most important mistakes are made on the first day.”

- The Art of Systems Architecting (Maier & Rechtin)

Project Orca was the Romney campaign’s technological effort to track in near-real-time actual voting in precincts all over the United States, with the intent of using that information in combination with individual-specific demographic databases to increase or pull back phone-call and door-knock get-out-the-vote (GOTV) efforts based on what was happening there. Since it appears more and more that Romney lost largely because of a lack of Republican turnout, it’s safe to say that the Romney/RNC GOTV effort was a failure. What is less clear is how much of an impact Orca had on that failure.

Three important disclosures up front. First, I was a Project Orca volunteer in Colorado. Second, I have no direct knowledge of how Orca was envisioned, developed, and operated beyond my own experience as an end user. Third, I’ve worked in software engineering and information technology since 1974, and my major professional focus since 1994 or so has been on how and why IT projects fail or succeed — a subject on which I’ve taught seminars, published books and articles, consulted in large corporations, and testified in courts and before Congress.

That said, the flap over Orca has made it all the way to the Drudge Report, largely due to two key discussions. The first is by JohnE over at Ace of Spades, who wrote the initial scathing report on Orca and followed it up with a reply to some of the push-back from the Romney campaign. The second is by Joel Pollack over at Breitbart, who had a Romney worker in Colorado who spoke of some of the problems.

My own experience matches much of what JohnE (who was also in Colorado) stated. I participated in four training calls. The first three all gave the same information about how the system would work, though — as JohnE noted — the Romney people kept refering to the Orca software as an “app”, even though it was simply a website that you logged into. (From JohnE’s comments, it sounds as though the use of the term “app” confused at least some Orca volunteers.)

Unlike JohnE, I was very clear from both the calls and the e-mails I receive that I needed to pick up my poll watcher certification and bring it with me to my assigned polling place on Election Day, so I am curious about his confusion. On the other hand, for my particular county (Douglas County), it appears that the certifications weren’t ready until the day before Election Day, which strikes me as cutting it a bit close. But I did drive down to Castle Rock on Monday afternoon and picked it up.

There was one more call the night before Election Day, but that was just a very brief, rah-rah call. After that Monday night call, I tried logging into the Orca web site to test out my password, but the log-in failed. I found that puzzling, but just assumed that they had the system locked down ether for security or administrative purposes.

Since Colorado is one of the few states that does not allow any electronic devices (phones, tablets, laptops) in polling places, I was also very clear from the training that I would have to print out the list of registered voters, bring it with me, mark off voters as they identified themselves at the polling place, and then periodically run out to my car and enter voting information via my iPad or iPhone. I printed out the list — and immediately discovered what I considered the first, admittedly minor, glitch: the list, a PDF file over 60 pages long, was formatted in such a way that you could not three-hole-punch the sheets (to put in a binder) without punching through the check boxes (for tracking voting and reporting) for some of the voters on each page. As I said, minor, but my technical writing background made me frown a bit. Since I own Adobe Acrobat, I solved the problem via the kludge of adding headers and footers to each page and then telling the pages to shrink themselves to fit.

So, with two binders in hand (I printed two copies of the list), with iPhone, my iPad, and my laptop in the car — along with two Black & Decker power boxes, and a couple of folding camp chairs — I showed up at my polling place at 6:45 am. I also brought with me three dozen fresh donuts (a suggestion from one of the training calls) that were greatly welcomed and did much to smooth relations with the actual poll workers.

Everything went well until the first time I went out to the car to report voters. I tried to log into the Orca website, and once again I got a “login failed” error message. (Second minor but telling item: that error message had a typo in it. The fact that such a typo was in a high-probability error message in a live system with over 30,000 anticipated users speaks to haste and sloppiness.) I called the help number given in the error message, and got a support person at the Boston command center who said they would reset my password and gave me the new code, but that it would take a little while for the new code to go live.

When I came back out again an hour or so later and tried to log in, the new password didn’t work either. The Orca system also had a call-in option, so I tried that and was told that my PIN was invalid.

At this point, it was about 8:30 in the morning. The next eight or so hours were all more of the same — I would call, the password would be reset, I would wait a while, I would try to log in, it would fail. Rinse and repeat. I was told by some of the support people I spoke with that almost no one in Colorado was able to log in — a geographical uniqueness that I found puzzling, since the login screen just asked for username (an e-mail address) and password. Another support person mentioned that there were similar problems being reported for North Carolina. Finally, sometime around 4:30 to 5:00 pm, I was given a new pin for the dial-in line — and that finally worked. I reported all the voters who had voted so far that day and did updated reports twice more before the poll finally closed at 7:00 pm.

JohnE pushes back (with justifiable skepticism) against Zac Moffat’s claims that “the campaign had voting data from 91 percent of counties” — not that they is necessarily inaccurate, but that it could be largely irrelevant. As JohnE points out, there can be hundreds of precincts in a single county; beyond that, having voting data from a given county does not necessarily mean having actionable, timely, and sufficiently complete data throughout the day in order to make GOTV decisions for that county (or for a given precinct in that county).

There is still no evidence that there were significant Orca problems outside of Colorado; we’ll have to wait to see if someone on the inside of Orca development decides to leak. But here are some observations from an large-scale IT system development perspective.

“A complex system that works is invariably found to have evolved from a simple system that works. . . . A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.”

– Systemantics (John Gall)

First, the Orca folks basically released version 1.0 (or, perhaps more accurately, version 0.9 or lower) into a massive, live environment on the single most critical day, Election Day. This is commonly known in IT circles as the “Big Bang” approach, and it is a well-known harbinger of disaster. Even if the Orca developers were all geniuses, this still would be a grave error. We’re not talking about an iOS or Android app being used by a single person on his/her device for entertainment or productivity; we’re talking about a massive web-based system that has to accept data from (not just deliver content to) possibly tens of thousands of users simultaneously and in real time. There should have been days, if not weeks, of live tests and trials to see how the system performed and to get the kinks worked out. Instead, as far as I can tell, the first time most Orca volunteers actually used (or tried to use) the system was on Election Day. Even if the system worked perfectly, I doubt that the majority of users — who had no real training on the system — would have used it in an effective manner.

“There are products that you shouldn’t develop, companies you shouldn’t challenge, customers you shouldn’t win, markets you shouldn’t enter, recommendations from the board of directors you shouldn’t follow.”

– The Art of ‘Ware (Webster)

Beyond that — and as per the quote above (paraphrasing Sun Tzu), as well as the one at the start of this post — the real question is: should this have been attempted in the first place? There are anecdotal reports that the focus on Orca took away resources — especially feet-on-the-ground resources — for local GOTV efforts. And GOTV is exactly where the Republicans failed. If work on Orca had been started 3 or 4 years ago, and if it had been used on Election Day 2010 — both as a real-world test and to verify the expected benefits of matching actual voters with demographic information to predict results in a given precinct — then it might have been ready for prime time on Election Day 2012.

But to spend signficant time, money, and resources (including tying up some 30,000+ local volunteers) for an untested system and an untried concept was a recipe for disaster — a well-known recipe, I might add, for those of us who have studied large-scale IT projects (which, frankly, are prone to failure anyway).

“But,” you ask, “you signed up to volunteer.” Yes, I did. But I had no visibility into the project and no inherent reason in advance to assume that things weren’t being done well, beyond the few niggling questions I had prior to Election Day. And, to be perfectly fair, I have no evidence of the extent of problems beyond Colorado, aside from the acknowledged 90-minute downtime on Election Day morning. But, even if those are the only two failures, can you seriously argue that it was a success? A time-critical, mission-critical system whose entire purpose for existence is to be used in a single 12-hour window is down for over 10% of that time in all states and is down all day in one critical swing state?

And, finally, the bottom line: if the purpose of Orca was to provide real-time intelligence on actual voting trends and make GOTV more effective, then where are the results? By all accounts, the Romney campaigned was stunned by their loss; if Orca was providing accurate, real-time indication of voting, as the project directors claimed it would in the training calls (actual quote [from memory]: “We’ll know ahead of anyone else how the election is turning out.”), then Romney should have had his concession speech written well in advance. And if its goal was to trigger targeted GOTV efforts in swing states, then why did Romney lose almost every swing state? Measured by its stated goals, Orca was a failure, pure and simple. Worse yet, it may have diverted critical resources from more effective GOTV efforts.

“Start out stupid and work up from there.”
– Bruce Henderson

I’d love to be able to do a project post mortem on Orca, as I have done on so many troubled, failing, or failed projects. But even without access to people and documents, I can hazard a guess at some of the likely factors:

• A late start and an impossibly short schedule.
• Because of these above, a waterfall (single pass) approach to design, development and deployment, instead of an iterative approach.
• Lack of a single chief software architect, with resulting problems at subsystem interfaces.
• A large development team with no prior track record developing and releasing software as a single team.
• Along the same lines, key components divided up among separate teams with relatively late integration and testing.
• And possibly some turnover (departure) of key personnel through the project, along with new personnel added late.
• Insufficient resources (money, personnel, equipment, time) devoted to quality assurance — and by “quality assurance”, I mean not just a wide range of testing (including, as JohnE points out, stress testing), but also individuals and teams with proper expertise, agreed-upon standards and guidelines, reviews, metrics, defect management, and so on.
• A firm belief — without any real-world evidence — that the idea would actually work and that the results would actually be worth the diversion of resources (including volunteers).

I’m happy to talk with anyone from the project who wants to set me straight or explain what really happened. But, in the end, what really happened was that Orca failed to achieve its goals, and Romney lost what should have been a very winnable election. ..bruce w..

The Mythical Man-Month: Essays on Software Engineering, Frederick P. Brooks, Jr. Addison-Wesley Publishing Company, Reading, MA, 1975. Softbound, 195 pages. Personal acquisition date: unknown. Original edition out of print.

The Mythical Man-Month: Essays on Software Engineering (Anniversary Edition), Frederick P. Brooks, Jr. Addison-Wesley, Boston, MA, 1995. Softbound, 322 pages. Personal acquisition date: unknown (I’ve bought and given away several copies over the years). Available in Kindle edition as well.

Seminal work; highest recommendation.

===========================

(All quotes and citations are from the 1995 Anniversary edition, which contains the slightly-edited text and original page numbering of Brooks’s original book, followed by four new chapters: two additional essays, and two retrospectives on the original edition.)

The Mythical Man-Month is probably the single best-known work in software engineering, as well it should be: it is a classic, in every sense of the word.

First of all, it is (for a thin volume) a remarkably comprehensive look at the practice of professional software engineering. Brooks’s essays cover a wide ranges of topics, with a particular focus on where things are likely to go wrong and what must be done to make things go right. For years, whenever I thought I had conceived some new issue or principle in software development, I would go back and look through Brooks and discover he had already noted it.

Second, it is highly accessible. Brooks’s style is clear, concise, and very readable. It is a book that anyone in a given organization should be able to read and understand. Furthermore, these are true essays: wonderfully written and downright lyric in places. One of my favorite passages is at the end of his discussion as to the ‘Joys of the Craft [of Programming]‘:

Finally, there is the delight of working in such a tractable medium. The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. (As we shall see later, this very tractability has its own problems.)

Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.” (pp. 7-8)

Finally, it remains highly relevant for our day, perhaps sadly so. Much of my professional work deals with troubled or failed IT projects, and my analysis of their struggles — whether in vivo or post mortem — usually reveal the same core problems that Brooks warned us of nearly 40 years ago. It seems that in the software industry, we not only insist on re-inventing the wheel, we insist on re-inventing the flat tire. As I said in my look at The Psychology of Computer Programming, billions of dollars wasted in failed IT projects could have been saved if those involved had read and heeded Brooks and Weinberg.

While Brooks is, in fact, mostly focusing on the development of what he calls programming systems products — in effect, writing commercial operating systems and related utilities — most of what he has to say applies to any large scale software development effort. Here, then, are some selected quotes from Brooks (all emphasis in the original):

The Woes of the [Programming] Craft. . . .
First, one must perform perfectly. The computer resembles the magic of legend in this respect, too. If one character, on pause, of the incantation is not strictly in proper form, the magic doesn’t work. . . .

Second, other people set one’s objectives, provide one’s resources, and furnish one’s information. One rarely controls the circumstances of his work, or even its goal. . . .

The next woe is that designing grand concepts is fun; finding nitty little bugs is just work. . . .

Next, one finds that debugging has a linear convergence, or worse, where one somehow expects a quadratic sort of approach to the end. So testing drags on and on, the last difficult bugs taking more time to find than the first. . . .

The last woe, and sometimes the last straw, is that the product over which one has labored so long appears to be obsolete upon (or before) completion. (pp. 8-9, “The Tar Pit”)

The second fallacious thought mode is expressed in the very unit of effort used in estimating and scheduling: the man-month. Cost does indeed vary as the product of the number of men and the number of months. Progress does not. Hence the man-month as a unit for measuring the size of a job is a dangerous and deceptive myth. It implies that that men and months are interchangeable.

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communications among them. This is true of reaping wheat or picking cotton; it is not even approximately true of system programming. (p. 16, “The Mythical Man-Month”)

Oversimplifying outrageously, we state Brooks’s Law: Adding manpower to a late project makes it later.  (p. 25, “The Mythical Man-Month”)

This, then, is the problem with the small, sharp team concept: it is too slow for really big systems. (p. 31, The Surgical Team)

I will contend that conceptual integrity is the most important consideration in system design. It is better to have a system omit certain anomalous features and improvements, but to reflect one set of design ideas, than to have one that contains many good but independent and uncoordinated ideas. . . .

Conceptual integrity in turn dictates that the design must proceed from one mind, or from a very small number of agreeing resonate minds. (pp, 42 & 44, “Aristocracy, Democracy, and System Design”)

An architect’s first work is apt to be spare and clean. He knows he doesn’t know what he’s doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used “next time.” Sooner or later, the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. . . . The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one. (p. 55, “The Second-System Effect”)

The manual, or written specification, is a necessary tool, though not a sufficient one. The manual is the external specification of the product. It describes and prescribes every detail of what the user sees. As such, it is the chief product of the architect.

Round and round goes its preparation cycle, as feedback from users and implementers show where the design is awkward to use or build. For the sake of implementers it is important that the changes be quantized — that there be dated versions appearing on the schedule.

The manual must not only describe everything the user does see, including all interfaces; it must also refrain from describing what the user does not see. That is the implementer’s business, and there his design freedom must be unconstrained. The architect must always be prepared to show an implementation for any feature he describes, but he must not attempt to dictate the implementation. (p. 62, “Passing the Word”)

If there are n workers on a project, there are (n²-n)/2 interfaces across which there may be communication, and there are potentially almost 2ⁿ teams within which communication much occur. The purpose of organization is to reduce the amount of communication and coordination necessary; hence organization is a radical attack on the communication problems treated above. (pp. 78-79, “Why Did The Tower of Babel Fail?”)

[Charles Portman] found his programming teams missing schedules by about one-half — each job was taking approximately twice as long as estimated. The estimates were very careful, done by experienced teams estimating man-hours for several hundred subtasks on a PERT chart. When the slippage pattern occurred, he asked them to keep careful daily logs of time usage. These showed that the estimating error could be entirely accounted for by the fact that his teams were only realizing 50 percent of the working week as actual programming and debugging time. Machine downtime, higher-priority short unrelated jobs, meetings, paperwork, company business, sickness, personal time, etc., accounted for the rest. In short, the estimates made an unrealistic assumption about the number of technical hours per man-year.” (pp 89-90, “Calling the Shot”)

Beyond craftsmanship lies invention, and it is here that lean, spare, fast programs are born. Almost always these are the result of strategic breakthrough rather than tactical cleverness. Sometimes the strategic breakthrough will be a new algorithm . . . Much more often, strategic breakthrough will come from redoing the representation of your data or table. This is where the heart of a program lies. Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious. (p. 102, “Ten Pounds in a Five-Pound Sack”)

Why Have Formal Documents?

First, the writing the decisions down is essential. Only when one writes do the gaps appear and the inconsistencies protrude. The act of writing turns out to require hundreds of mini-decisions, and it is the existence of these that distinguishes clear, exact policies from fuzzy ones.

Second, the documents will communicate the decisions to others. The manager will be continually amazed that policies he took for common knowledge are totally unknown by some member of his team. Since his fundamental job is to keep everybody going in the same direction, his chief daily task will be communication, not decision-making, and his documents will immensely lighten his load.

Finally, a manager’s documents give him a data base and a checklist. By reviewing them periodically, he sees where he is, and he sees what changes of emphasis or shifts in direction are needed. (p. 111, “The Documentary Hypothesis”)

The management question, therefore, is not whether to build a pilot system and throw it away. You will do that. The only question is whether to plan in advance to build a throwaway, or to promise to deliver the throwaway to customers. Seen this way, the answer is much clearer. Delivering that throwaway to customers buys time, but it does so only at the cost of agony for the user, distraction for the builders while they do the redesign, and a bad reputation for the product that the best redesign will find hard to live down.

Hence, plan to throw one away; you will, anyhow. (p. 116, “Plan to Throw One Away”)

The manager of a project, then, needs to establish a philosophy and set aside resources for the building of common [development] tools. At the same time, he must recognize the need for specialized tools, and not begrudge his working teams their own tool-building. This temptation is insidious. One feels that if all those scattered tool builders were gathered in to augment the common tool team, greater efficiency would result. But it is not so. (p. 128, “Sharp Tools”)

Testing the Specification. Long before any code exists, the specification must be handed to an outside testing group to be scrutinized for completeness and clarity. As [V. A.] Vyssotsky [of Bell Labs] says, the developers themselves cannot do this: “They won’t tell you they don’t understand it; they will happily invent their way through the gaps and obscurities.” (p. 142, “The Whole and the Parts”)

When one hears of disastrous schedule slippage in a project, he imagines that a series of major calamities must have befallen it. Usually, however, the disaster is due to termites, not tornadoes, and the schedule has slipped imperceptibly but inexorably. Indeed, major calamities are easier to handle; one responds with major force, radical reorganization, the invention of new approaches. The whole team rises to the occasion.

But the day-by-day slippage is harder to recognize, harder to prevent, harder to make up. Yesterday, a key man was sick, and a meeting couldn’t be held. Today the machines are all down, because lightning struck the building’s power transformer. Tomorrow the disk routines won’t start testing, because the first disk is a week late from the factory. Snow, jury duty, family problems, emergency meetings with customers, executive audits — the list goes on and on. Each one only postpones some activity by a half-day or a day. And the schedule slips, one day at a time. (p. 154, “Hatching a Catastrophe”)

The flow chart is a most thoroughly oversold piece of program documentation. Many programs don’t need flow charts at all; few programs need more than a one-page flow chart. (p. 167, “The Other Face”)

Not only are there no silver bullets now in view, the very nature of software makes it unlikely that there will be any — no inventions that will do for software productivity, reliability, and simplicity what electronics, transistors, and large-scale integration did for computer hardware. . . .

I believe that the hard part of building software to be the specification, design, and testing of the conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared to the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet. (pp. 181-182, “No Silver Bullet”)

The biggest mistake in the “Build one to throw away” concept is that it implicitly assumes the classical sequential or waterfall model of software construction.” (p. 265, “The Mythical Man-Month after 20 Years”)

“I dismissed Parnas’s concept [of information hiding and encapsulation] as a “recipe for disaster” in Chapter 7. Parnas was right, and I was wrong. I am now convinced that information hiding, today often embodied in object-oriented programming, is the only way of raising the level of software design.” (p. 272, “The Mythical Man-Month after 20 Years”)

The “outrageously simplified” statement of Brooks’s Law is made more useful by these careful treatments [by other researchers] of the proper qualifications. On balance, I stand by the bald statement as the best zeroth-order approximation of truth, a rule of thumb to warn managers against blindly making the instinctive fix to a late project. (p. 275, “The Mythical Man-Month after 20 Years”)

The tar pit of software engineering will continue to be sticky for a long time to come. One can expect the human race to continue attempting systems just within or just beyond our reach; and software systems are perhaps the most intricate of man’s handiworks. This complex craft will demand our continual development of the discipline, our learning to compose in larger units, our best use of new tools, our best adaptation of proven engineering management methods, liberal application of common sense, and a God-given humility to recognize our fallibility and limitations. (p. 289, “The Mythical Man-Month after 20 Years”)

As with Weinberg, some of Brooks’s discussions are dated technically…or, at least, appear to be. One entire essay, “Ten Pounds in a Five-Pound Sack”, is devoted to dealing with memory management and tradeoffs between memory usage, functionality, and speed. That would seem irrelevant in today’s world of multi-gigabyte RAM, much less having virtual memory management standard in just about every operating system. And yet we have watched the emergence of ‘bloatware’ over the past few decades, where software keeps loading slower, running slower, and using more system resources, even as memory gets cheaper and processors get faster (or divide into multiple cores). What used to be craft is now institutionalized sloppiness.

If you are involved in software engineering and IT project management, there are many, many books you should read and be familiar with. But if you need to start somewhere, start here, and read this book. Read it a few times, in fact. The project — and the job — you save may be your own.  ..bruce..

TABLE OF CONTENTS (adapted from the 1995 Anniversary Edition)

Chapter 1: The Tar Pit
Chapter 2: The Mythical Man-Month
Chapter 3: The Surgical Team
Chapter 4: Aristocracy, Democracy, and System Design
Chapter 5: The Second-System Effect
Chapter 6: Passing the Word
Chapter 7: Why Did The Tower of Babel Fail?
Chapter 8: Calling the Shot
Chapter 9: Ten Pounds in a Five-Pound Sack
Chapter 10: The Documentary Hypothesis
Chapter 11: Plan to Throw One Away
Chapter 12: Sharp Tools
Chapter 13: The Whole and the Parts
Chapter 14: Hatching a Catastrophe
Chapter 15: The Other Face

(Added to the Anniversary Edition)

Chapter 16: No Silver Bullet — Essence and Accident
Chapter 17: “No Silver Bullet” Refired
Chapter 18: Propositions of The Mythical Man-Month: True or False?
Chapter 19: The Mythical Man-Month after 20 Years

[Version 1.1 of this post, revised/extended on 05/22/2012]

The Psychology of Computer Programming, Gerald M. Weinberg, Van Nostrand Reinhold Company, New York, 1971. Hardbound, 288 pages. Personal acquisition date: 17 Oct 1978. Original edition out of print.

The Psychology of Computer Programming (Silver Anniversary Edition), Gerald M. Weinberg, Dorset House Publishing, New York, 1998. Softbound, 292 pages. Personal acquisition date: unknown. Currently in print (and in ebook form).

Seminal work; highest recommendation.

===========================

(All quotes and citations are from the 1998 Silver Anniversary edition, which contains the unedited text and original page numbering of Weinberg’s original book interspersed with his observations 25 years later using a different pagination scheme.)

If The Mythical Man Month (Frederick P. Brooks, Jr., 1975/1995) is the book that most IT managers own but few ever read (and even fewer follow), The Psychology of Computer Programming (Gerald M. Weinberg, 1971/1998) is the book that almost nobody nowadays owns…at least, nobody under a certain age. And that’s a tragedy, because Weinberg wrote one of the first and still one of the best works on the human factors in IT project management and software engineering.

I bought my own first copy of Weinberg’s book in the fall of 1978, nearly 35 years ago, just a few months after joining General Dynamics with my freshly-minted BS in computer science from BYU. Weinberg’s book had a tremendous impact on me and forever changed my view of software development from a technical activity to a human, social activity. In fact, Weinberg’s first sentence in his first chapter is just that: “Computer programming is a human activity.” That single insight — with all it implies — is something that the IT industry seems to re-discover on a regular basis and then forget again just as quickly as it seeks the ever-elusive magic amalgam of technology, methodology, and language (with minimum pesky humans) that will make software development fast, cheap, and good. Weinberg touched upon that very quest early on in the book as well :

[1971] Over the years, executives have backed their desire to eliminate programmers with staggering funds. Dozens of simplistic schemes have been heaped with money and praise on the promise — as yet not kept — of going directly from a sales proposal to a working data-processing system. (p.4)

[1998] The only thing that’s changed here in twenty-five years is the fact that the funds dedicated by executives to eliminating programmers from their payrolls have become far more staggering than I imagined back then. (P1.i)

Furthermore, Weinberg got to many of Brooks’ key insights four years before Brooks published The Mythical Man-Month, including touching (in a slightly different way) on the most famous one: adding more people does not get a job done more quickly, and using exactly the same analogy that gave Brooks’ book its title:

For example, certain programming work cannot be done by a team of trainees, no matter how large, so that doubling the number of “warm bodies” — as they are so often called in the trade — still will not get the work done. Schedule is similarly limiting — we need only cite the apocryphal experiment which tried to make a baby in one month by putting nine women to work on the job as a team. (p. 68)

As per the table of contents given at the end of this post, Weinberg covered four major themes in his book: Programming as Human Performance; Programming as a Social Activity; Programming as an Individual Activity; and Programming Tools. As Weinberg himself points out (in his 1998 commentary on his 1971 work), he was really writing more about the anthropology of computer programming, rather than the psychology per se. The book is highly readable, largely because Weinberg has a conversational style of writing and fills the book with actual stories and anecdotes of observed behavior on programming projects.

Here is just a small sampling of the insights from Weinberg’s book (my comments are in brackets and italics):

Fisher’s Fundamental Theorem states — in terms appropriate to the present context — that the better adapted a system is to a particular environment, the less adaptable it is to new environments. (p. 21) [Thus anticipating the core problem in the entire OO/reuse movement of the 1990s]

The organization chart is a nice toy for a manager, but little programming work would ever get done if interactions among programmers had to follow its narrow, straight lines. . . . But human interactions are never narrow, never straight, and hardly ever in the directions shown on an organization chart. Many serious mistakes have been made in imagining that formal structure was the only structure in an organization. (p. 48)

A programmer who truly sees his program as an extension of his own ego is not going to be trying to find all the errors in that program. . . . And let there be no mistake about it: the human eye has an almost infinite capacity for not seeing what it does not want to see. (p. 55) [Weinberg then goes on to discuss his somewhat controversial (at the time) concept of 'egoless programming', which really boils to to getting other programmers to review your code, now considered to be a best practice: 'many eyes', pair-programming, end-of-day review swaps, etc.]

[By having programmers review each other's code:] Not only is the variation in debugging time reduced, but since there is more than one person familiar with the program, it is easier to get realistic estimates on the amount of real progress that has been made. It is not necessary to rely on a single judgment — and of the person least likely to be unbiased. (p. 59) [Weinberg goes on to note that you generally end up with more readable code, and you lower the impact of turnover within the programming group, since more than one person understands the code in question.]

Managers tend to select themselves from the “aggressive” component of society and have difficulty appreciating the fact that other people do not completely share their goals of money and prestige. They are especially at a loss to understand the smooth functioning of a programming group based on mutual respect for individual talent and cooperating in the common cause. Instead, they tend to view people as working for money or under threat — as they themselves do. (p. 61) [Some things are timeless.]

So many failures to meet programming deadlines can be traced back to an initial schedule and plan of attack which assumed the most optimistic conditions — no days lost through illness, no machine trouble, no compiler problems, no “impossible” bugs. Because each of us has had the experience of, say, a six-month period in which everyone on the team was in top health or there was no machine trouble or there were no compiler problems or there were no “impossible” bugs, we can slip into imagining — if the price is right — that we can have a six-month period in which all of these fortunate circumstances happen at once. (pp. 68-69) [Again, plus ça change...]

[After a story of upper management trying -- unsuccessfully -- to force a team leader into accepting an impossible schedule:] Did Harold lose his job? He didn’t, in this case; but that wasn’t really important, for he knew he could get another at least as good. If [keeping his job] had been important, he wouldn’t have been able to do what he did. One of the paradoxes of leadership is simply this: only the leader who is ready to step down has a real chance of success. (p. 85)

It is possible to be too smart for programming — if the person is not smart enough to use his intelligence to modify his social behavior and methods of communication. (p. 88) [Yes, there have always been prima donnas.]

In a sense, a programming project or team is like a river which remains the same river even thought its water is undergoing constant change. Many project managers are unable to grasp this view of a project. Their view of the project’s structure is, instead, much like that of a house — a structure that might collapse should one of the beams be removed. Their actions with respect to the people in the project — especially so-called “key” people — reflect this static view, often with disastrous consequences. (p. 96)

[Telling a story about the 'filtering' process of reporting project progress up a chain of command:] The net result of six or seven stages of such filtering was a report that monthly presented a consistent forward progress, a few areas slightly behind or slightly ahead, a few problems solved from last month, a few new problems, and a few problems still open. There was, in short, no measurable relationship between what had been reported at the bottom and what came out the top. (p. 101) [Thus describing what I years later termed "the thermocline of truth". Note that Weinberg himself commented on my post and argued for the gradual layer-by-layer changes that he describes in his book; I countered with two real-world examples where a single individual was responsible for most of the filtering.]

Pressure from higher up is classically recognized by management manuals as both the way to get work done and the way to destroy a reporting system. (p. 103)

Any testing group is also put into a difficult relationship with other groups, because it is their job to criticize. (p. 107) [Thus capturing the classic tension between QA and development, though he goes on to describe ways to overcome it.]

When a supervisor is responsible for work he does not understand, he begins to reward workers not for work, but for the appearance of work. Programmers who arrive early in the morning are through to be better programmers than ones who are seen to arrive after official starting time. Programmers who work late, however, may not be rewarded because the manager is not likely to see that they are working late. Programmers who are observed talking to others are not considered to be working, because the manager has an image that programming work involves scratching out secret messages to the computer. (p. 110) [Again, some things just don't change.]

Programs, like any other man-made objects, are designed — or should be designed — with a definite lifespan and scope of application in mind. . . . a program should have neither overdesigned or underdesigned parts. Yet it is an occupational disease of programmers to spend more time on those program parts that present, for some reason, the most intellectual challenge rather than on those that require the most work. (p. 126)

Another fallacy which we shall have to lay to rest is that “programming” is some sort of uniform effort requiring a set of uniform talents. For the professional, at least, the job from getting to specifications to delivered program demands various kinds of work, wihch, in turn, demand various talents. (p. 132) [Something which I subsequently discussed in the 'talent' section of my TEPES post a few years back and some 37 years after Weinberg's observation.]

Programming is often described as a process moving from problem definition through analysis to flow diagramming, then coding, followed by testing, and finishing with documentation. Although this rough view contains some truth, it distorts the truth in several ways. First of all, the actual sequence is not so fixed, because, for example, documentation may precede testing, coding, flow diagramming and even analysis. Second, not all steps need to be present, as when we are recoding a program for a new machine or language. Thirdly, it need not be a sequence at all — and, in actual practice, rarely is. Who has not experienced a problem definition that changes as discoveries are made in analysis, flow diagramming, coding, testing, and documentation? Or who has ever seen a flow diagram that remains unmodified throughout the coding — or code that remains unmodified throughout testing? (p. 133) [Yes, Weinberg in 1971 is dismissing the 'waterfall' model and arguing not just the need but the inevitability of incremental/iterative development.]

Although the average programming manager would say that intelligence is more important that personality in programming success, very few could cite cases of people who turned out to not to be intelligent enough to program, but everyone knows of cases of people who were not temperamentally suited to the programmer’s job. It is in this sense that we can assert that personality is more important than intelligence in programming. (p. 148)

[Regarding ideal personality traits for programmers]…we can probably say with assurance that someone without the ability to tolerate stressful situations for a period of a week or more is not good programmer material…people who are not in some measure adaptable to rapid change will probably have trouble as professional programmers….one of the most easily identifiable personality needs in programming is a modicum of neatness. We are not speaking here of personal grooming…What we mean is a slight compulsion to keep one’s papers in order, without which the computer’s paper-generating capacity inexorably leads to grief….Another essential personality factor is at least a small dose of humility. Without humility, a programmer is foredoomed to the classic pattern of Greek drama: success leading to overconfidence (hubris) leading to blind self-destruction….The other side of the coin of humility is assertiveness, or force of character. A programmer’s job is to get things done, and getting things done sometimes requires moving around obstacles, jumping over them, or simply knocking them down….Last among the essential personality traits for programming, we might list sense of humor. The computer “doth make fools of us all”, so that any fool without the ability to share a laugh on himself will be unable to tolerate programming for long. (pp. 149-150) [Still probably one of the best summaries of what makes a great programmer, and particularly a great co-worker on a programming team.]

One of the best known and accepted results of motivation research is that increasing “driving force” will first increase performance to a maximum, beyond which addition of further driving forces will quickly drive performance to zero. (p. 182) [Can you say "death march", boys and girls? I knew you could.]

The fear of new things, the expectation of failure, and the reluctance to admit weakness all have a direct retarding effect on learning, whether in a formal classroom setting or on the job. (p. 191)

A programmer would not really be a programmer who did not at some time consider his program as an esthetic object. This part is not quite symmetrical; that part is clumsy and doesn’t flow in an appropriate manner; the whole thing does not look proper on the page. To be sure, it is fashionable among programmers to be rough and tough and pragmatic, but deep down each programmer knows that it is not enough for a program just to work — it has to be “right” in other ways. Later, when we discuss language design and program testing, we shall see that the correlation between the esthetic and the pragmatic value of a program is not accidental — the more pleasing to the eye and mind, the more likely to be correct. (p. 209) [Thus anticipating the whole emergence of the "elegance" value in software design and implementation.]

What these quotes — which represent a tiny fraction of the book — don’t contain are the aforementioned anecdotes. Weinberg illustrates point after point with real-world stories that — aside from the archaic technology — sound very contemporary and very true to life, and are usually entertaining to boot.

Weinberg also has questions for IT managers and for programmers themselves at the end of each chapter. As he himself notes, most of these hold up very well (though there are what he terms some “hilarious exceptions”). Here are some examples

[For managers:] On what basis do you reward programmers? Are certain of your criteria mutually contradictory, as in asking for efficient but general programs? How explicit are you with your programmers in indicating what you are looking for in their programs? Or do you just tell them that you want the programs to be fast, small, neat, easily modifiable, errorless, and done in a week? (p. 25)

[For programmers:] When you have just found a bug, do you ever sit back and try to trace out the paths you took in your mind? Try doing this on the next bug you find, and write a brief report or outline on what you find. (p. 40)

[For programmers:] Do you refer to your work as “my” program? Try passing one week without using the personal possessive in reference to programs, and take notes on the effects you observe. (p. 65)

[For managers:] Do you ever do things to try to inflate the appearance of your technical competence in front of the people who work for you? Describe some of these incidents, and also some incidents in which it was discovered that your technical competence was in at least one respect inferior to one of the people who work for you. What were the consequences of that discovery, and do they justify attempting to cover up? (p. 92)

[For programmers:] Has your manager ever done anything to make you doubt his honesty? If so, describe the incident, what ultimately happened to your doubt, and how your work was affected. (p. 93)

[For managers:] Which do you reward most, accurate information or pleasing information? Do your programmers know what the information you require is used for? Do they see the final reports which are the destination of their information? If not, why not? (p. 114)

[For programmers:] Have you ever yielded to group or managerial pressure against your best technical judgment? Describe the situation and the consequences. (p. 115)

[For programmers:] Make a list of things that your software does for you that fifteen years ago you would have have to do for yourself. Or don’t you know anything about what software was like fifteen years ago? Do you think a professional should know something about the history of his profession? (p. 138)

[For managers:] Do you use any aptitude tests now in choosing programmers, or does your personality department use them? If so, what do you know about their validity? Do you make any effort to validate them by evaluating programmers after a period of time on the job? What methods do you use for this evaluation? How convinced are you of their effectiveness? (p. 177) [Still very relevant questions, particularly for the organizations that like to use the 'brain teaser' approach to interviewing programmers.]

[For programmers:] Recall from your experience a “tiny” error that had a big ultimate cost. What debugging tools or techniques could have prevented that error? Would their cost have been greater than the cost of that one error? (p. 271)

Weinberg — who was only 38 years old when this book came out — would go on to write “more than 40 books and over 400 articles“, and he is still going strong.

In short, Weinberg really was there first with an overarching view, set down in nearly 300 pages over 40 years ago, of the human and social factors that affect software development and project management. As the excerpts above show, his observations remain utterly relevant for today. Professionally, I have been dealing with troubled or failed IT projects, either as a consultant or as an expert witness, since 1995. What I can tell you is that Weinberg and Brooks (whose book I’ll review next) pretty much nailed all the core issues some 40 years ago. During those four decades, tens of billions of dollars – if not a great deal more — have been wasted on failed or sub-performing IT projects. A lot of that money lost — both directly and indirectly — could have been saved if folks just read — and believed, and followed — these “old” books. ..bruce..

TABLE OF CONTENTS (adapted from the 1998 Silver Anniversary Edition)

PART 1. PROGRAMMING AS HUMAN PERFORMANCE

1. Reading Programs

2. What Makes a Good Program?

3. How Can We Study Programming?

PART 2. PROGRAMMING AS A SOCIAL ACTIVITY

4. The Programming Group

5. The Programming Team

6. The Programming Project

PART 3. PROGRAMMING AS AN INDIVIDUAL ACTIVITY

7. Variations in the Programming Task

8. Personality Factors

9. Intelligence, or Problem-Solving Ability

10. Motivation, Training, and Experience

PART 4. PROGRAMMING TOOLS

11. Programming Languages

12. Some Principles for Programming Language Design

13. Other Programming Tools

PART 5. EPILOGUE

Progress, far from consisting in change, depends upon retentiveness….Those who will not remember the past are condemned to repeat it. (The Life of Reason, 1905)

In my opinion, most of core issues in software engineering and IT project project management were identified decades ago, but our industry suffers from vast institutional ignorance of these works — ‘ignorance’ both in the sense of ‘unaware of’ and ‘paying no attention to’. As I once said in a rather interesting setting¹:

Humanity has been developing information technology for half a century. That experience has taught us this unpleasant truth: virtually every information technology project above a certain size or complexity is significantly late and over budget or fails altogether; those that don’t fail are often riddled with defects and difficult to enhance. Fred Brooks explored many of the root causes over twenty years ago in The Mythical Man-Month, a classic book that could be regarded as the Bible of information technology because it is universally known, often quoted, occasionally read, and rarely heeded.

To that end, and because I have not been posting much here, I am introducing a new series of posts: Readings in Software Engineering, or RISE. Each post will cover one book (including subsequent editions). I will give what context I have for the book, then quote interesting or relevant excerpts from the book, followed at the end by the book’s table of contents. My intent is not to present “one page summaries” of these books, as are often found for various business books; instead, it is to pull out relevant quotes, in the authors’ own words, in hopes that you’ll actually track down the books and read them yourselves.

A final note: I reserve the right to go back and substantially edit my RISE posts after the fact, particularly as I get several of these under my belt and start to settle in on a standard approach and format. Iterative writing, so to speak. ..bruce..

¹In testimony before Congress in 1998. Long story, had to do with Y2K.

A few years later — in 1982 — my close friend Wayne Holder hired me into his nascent software company, Oasis Systems, in part to help with his existing and planned word processing utilities (The Word Plus, Punctuation + Style), but mostly to develop computer games. And we did, developing Sundog: Frozen Legacy on the Apple II, a game for which I still get e-mails (and which Wayne is even now working on resurrecting for modern platforms). In January 1984, a few months before Sundog shipped, we were invited by Guy Kawasaki to come up to Apple to see a preview of the Mac and to talk about what software we could port to the Mac. Through my connections with computer stores in San Diego, I was able to get a personal loan of a Mac for a few days at home prior to the official announcement in Cupertino later that month, which Wayne and I attended as well. That was my first time seeing Steve Jobs in person, and it remains a memorable highlight of my professional life.

When the Mac shipped a few days later, I went down to the one computer store in San Diego that I knew would be getting machines from Apple. I took $3000 in cash with me and managed to convince the store owner — a friend — to let me have one of the three Macs he had to sell. Through a connection with Phil Lemmons — editor-in-chief at BYTE — I ended up writing the official BYTE review of the 128K Macintosh (August 1984 issue). By the end of 1984, I was writing full-time for BYTE, including on-going coverage of the Macintosh, particularly once my BYTE column started in mid-1985. After a few years of writing for BYTE, I switched to writing for Macworld magazine. Steve was now long-gone from Apple, and Apple was having some of its own problems going forward.

But in late 1987, I was contacted by Addison-Wesley. They were interested in having me write a book about Steve Jobs’ new project at NeXT. Folks at NeXT had apparently suggested me to Addison-Wesley, probably due to my writing at BYTE and Macworld. I leapt at the opportunity, particularly since in coincided with our family moving from Utah to just outside Santa Cruz (where I would be doing technical writing for Borland on a consulting basis). Once there, I found myself invited to visit NeXT HQ on Deer Creek Road, sit in on meetings, and attend the 0.3 NeXTstep Dev Camp. And, yes, that meant getting actual face time with Steve Jobs as well — not a lot, but this was a man whose creations had been impacting my personal and professional life for over a decade at this point.

The writing of the book dragged out as I waited to get my hands on an actual NeXT cube, which finally happened (if I recall correctly) at the end of 1988 or early 1989. I wrote the first several drafts of the book on that NeXT cube itself. The book came out in the fall of 1989; it remains the single most successful book I’ve ever written, due to the intense interest in NeXT itself, more than any particular writing skills or technical insight on my part.

The following year, I found myself working with a world-class typographer (Mike Parker) and graphic designer (Vic Spindler) to create a design-oriented desktop publishing system. I was doing all the software prototyping on my NeXT cube, and we made the decision to make the NeXT our first target platform. For five years — 1990 to 1995 — I served as chief architect and CTO at Pages Software Inc, where we developed Pages by Pages and then WebPages, while spending nearly two years just trying to raise venture funding. We closed on funding at the start of 1992 and shipped our first version of Pages in early 1994. We quickly sold all that we were going to in the all-too-small NeXTstep market. My frustrations at seeing larger firm try to leverage off of NeXT’s incredible innovations led to an op-ed piece in the November 1994 issue of BYTE, “Whither NextStep?” The day that issue came out was the last time that Steve Jobs and I spoke — he called me from the back of a car somewhere to ask me what the hell I was doing writing that. I said, telling the truth. Pages would close its door the next year, unable to secure additional funding to move its technology to Windows.

When Steve engineered his brilliant reverse takeover of Apple — getting Apple to buy NeXT for $400 million, then slowly moving himself into the CEO seat — I was not optimistic. I still had unconditional praise for the NextStep technology, but I was dubious about Steve’s ability to sell technology to markets and to compete with Microsoft.

Boy, was I wrong. I was not only wrong about his abilities at Apple, I was wrong in my BYTE article about NextStep being on a downward slope. NextStep, of course, was the foundation of Mac OS X, and Steve transformed Apple into the most-admired, most-imitated, and most-valuable company in the world. And I was tickled that, when Apple brought out its own word processor, it was named “Pages”. Steve had always liked that name when we were developing (and shipping) our own product years before; glad he was able to use it.

To quote John Perry Barlow over on FB, “The world is suddenly a less interesting place.” ..bruce w..

[1] The first was an HP-67 card-reading programmable calculator.

[Cross-posted from And Still I Persist]

Now, thanks to advances in artificial intelligence, “e-discovery” software can analyze documents in a fraction of the time for a fraction of the cost. In January, for example, Blackstone Discovery of Palo Alto, Calif., helped analyze 1.5 million documents for less than $100,000.

Some programs go beyond just finding documents with relevant terms at computer speeds. They can extract relevant concepts — like documents relevant to social protest in the Middle East — even in the absence of specific terms, and deduce patterns of behavior that would have eluded lawyers examining millions of documents.

“From a legal staffing viewpoint, it means that a lot of people who used to be allocated to conduct document review are no longer able to be billed out,” said Bill Herr, who as a lawyer at a major chemical company used to muster auditoriums of lawyers to read documents for weeks on end. “People get bored, people get headaches. Computers don’t.”

Since I work in litigation support myself and have to search and read documents — I’ve had individual cases with over a million pages of documents — I welcome such tools as these. Still, the overall theme in the article seems to be that humans will lose jobs due to these tools:

Quantifying the employment impact of these new technologies is difficult. Mike Lynch, the founder of Autonomy, is convinced that “legal is a sector that will likely employ fewer, not more, people in the U.S. in the future.” He estimated that the shift from manual document discovery to e-discovery would lead to a manpower reduction in which one lawyer would suffice for work that once required 500 and that the newest generation of software, which can detect duplicates and find clusters of important documents on a particular topic, could cut the head count by another 50 percent.

What the article does not address is the enormous financial burden that e-discovery has imposed on both parties in modern civil litigation (see also here and here).

It used to be that each side would gather physical documents from files and storage boxes, review them for responsiveness, then produce them to the other side. Outside of large corporations, most parties would not have great volumes of such documents.

However, with the advent and pervasiveness of digital technology — computers, e-mail, servers, instant messaging, and so on — even a relatively small firm or organization can find itself with gigabytes, if not terabytes, of potentially relevant production. And, to avoid court sanctions, all of that has to be combed through and reviewed for responsiveness. Most organizations below a certain size just can’t afford to do that the old-fashioned way with “a platoon of lawyers and paralegals who [work] for months at high hourly rates”.

The article also does not address the existing document organization and search tools that are a bit less artificially intelligent — and a lot cheaper — than the tools he covers, but that still allow complex search terms to be set up. I make heavy use of dtSearch in most of the cases I work on (and can recommend it highly). I’ve also used such standard legal document management tools as Summation and Concordance.

After conducting a series of experiments comparing a sample Corsair 64GB SSD with a conventional Hitachi 80GB magnetic hard drive (HDD), the team found a layer cake of data recovery problems caused by the ‘garbage collection’ or purging algorithms used in SSDs to keep them at peak performance.

After examining an SSD for traces of data after it had been quick formatted, the team expected the purging routines to kick in around 30-60 minutes later, a process that must happen on SSDs before new data can be written to those blocks. To their surprise, this happened in only three minutes, after which only 1,064 out of 316,666 evidence files were recoverable from the drive.

Going a stage further, they removed the drive from the PC and connected a ‘write blocker’, a piece of hardware designed to isolate the drive and stop any purging of its contents. Incredibly, after leaving this attached for only 20 minutes, almost 19 percent of its files had been wiped for good, a process the researchers put down the ability of SSDs to initiate certain routines independent of a computer.

For comparison, on the equivalent hard drive all data was recoverable, regardless of the time elapsed, as a forensic examiner would expect

“Even in the absence of computer instructions, a modern solid-state storage device can permanently destroy evidence to a quite remarkable degree, during a short space of time, in a manner that a magnetic hard drive would not,” the team concludes.

The results are concerning on a number of levels, forensic, legal and technical.

Yeah, I’ll say, though I suspect most of the concerns are legal and forensic. Hat tip to Slashdot.  ..bruce..

According to Gleick, the impact of information on human affairs came in three installments: first the history, the thousands of years during which people created and exchanged information without the concept of measuring it; second the theory, first formulated by Shannon; third the flood, in which we now live. The flood began quietly. The event that made the flood plainly visible occurred in 1965, when Gordon Moore stated Moore’s Law. Moore was an electrical engineer, founder of the Intel Corporation, a company that manufactured components for computers and other electronic gadgets. His law said that the price of electronic components would decrease and their numbers would increase by a factor of two every eighteen months. This implied that the price would decrease and the numbers would increase by a factor of a hundred every decade. Moore’s prediction of continued growth has turned out to be astonishingly accurate during the forty-five years since he announced it. In these four and a half decades, the price has decreased and the numbers have increased by a factor of a billion, nine powers of ten. Nine powers of ten are enough to turn a trickle into a flood.

I’ve lived through most of the information age (if we peg it at 1940 or so; I was born in 1953), and I’ve worked in information technology for almost 40 years (since 1974). I remember seeing a second megabyte of core memory that was being added to the IBM 360/65 on campus in the mid-1970s: it was the size of a giant shoebox, and it cost $250,000. Now the cost per megabyte for 1GB RAM SIMMs sold over Amazon is as low as $0.02, and there’s 1,024 megabytes on that little stick. I have something approaching 15 terabytes of hard disk storage on the various computers in our house, and — as per Dyson’s review and Gleick’s book — my biggest challenge is keeping it organized and accessible. There’s a high level of redundancy, with numerous minor (or major) variations; at the same time, I fear that a disk failure will forever deprive me of files that I might someday want or need again, whether that’s tomorrow or ten years from now.

I’ll read the book when it comes out (I have a copy preordered from Amazon); in the meantime, Dyson’s review is well worth reading.

At the same time, classic rotating storage devices — such as platter-based hard disk drives — are slowly being supplemented and in some cases supplanted by solid-state media, that is, storage devices that have no moving parts at all. In particular, solid-state disks (SSDs) look and behave just like classic hard disk drives, but with usually faster response times and much lower chance of hardware failure. In particular, USB drives have become ubiquitous as a mechanism for moving files between unconnected computers, while laptops are starting to offer SSDs as the principal internal storage for both speed and weight improvements.

So far, so good. Except that out of the University of California at San Diego comes research that shows that in many cases, standard file- and disk-erasing techniques leave behind recoverable data when used on SSDs:

At the Non-volatile Systems Laboratory we have designed a procedure to bypass the flash translation layer (FTL) on SSDs and directly access the raw NAND flash chips to audit the success of any given sanitization technique. Our results show that naïvely applying techniques designed for sanitizing hard drives on SSDs, such as overwriting and using built-in secure erase commands is unreliable and sometimes results in all the data remaining intact. Furthermore, our results also show that sanitizing single files on an SSD is much more difficult than on a traditional hard drive. We are working on designing new FTLs that correct these issues and also exploit properties of flash memory to maintain performance while sanitizing the flash drive.

I imaging that word of this will quickly spread to companies that perform computer forensics (including recovery of data for storage devices). In the meantime, many organizations may continue to make us of USB drives and internal SSDs in laptops (and, eventually, desktops), and by so doing may leave themselves open to discovery of data that they thought purged in the course of normal, documented operations.