Webster & Associates LLC

Cybercrime Review is one of the blogs I read daily, due to my ongoing interest in the intersection between law and technology. The posts generally have to do with court rulings and decisions in cases involving computer technology, such as whether someone can be compelled to provide a password for an encrypted data source. But a post today caught my attention, because it dealt with expert testimony, something I’m personally familiar with. The cases involved were criminal cases, whereas I’ve only worked in civil cases, but I still cringed reading the post. In cases where I have served as an expert, I am always methodical and cautious — I will testify to what I can support and defend, but no farther. I am frank with my clients as to my findings and conclusions, and what I am and am not willing or able to say; to their credit, my clients have always handled that well, even if they have at times been disappointed.

The post, ”Forensic Fraud: Now Available on Daytime TV” by Marielle Dirkx, highlights two cases. The first is the notorious Jodi Arias murder trial taking place in Arizona right now. At that trial, a forensic expert presented what he claimed was evidence from a photograph taken by Arias of her boyfriend, Travis Alexander, just before she killed him (something she admits, but claims was in self-defense). The expert blew up the photograph and purported to show a reflection of Arias in Alexander’s eye. Furthermore, he claimed that his enhancement of the reflection showed that Arias was not holding a weapon at the time she took the photograph.

See for yourself:

Obvious, right?

Clear, right? And from this processed enlargement, “According to the expert, Arias was standing a few feet away from Alexander, holding the camera with both hands at the time the picture was taken; and he further asserted that he could state with scientific certainty that she was not holding either a gun or a knife.”

Facepalm.

According to Dirkx, the expert’s testimony and evidence were not admitted.

The second cases is too detailed to summarize here, but it involves a “go-to” expert for prosecutors in Mississippi, a dentist who “routinely testified as a wound pattern expert, a trace metals expert, a gun shot residue expert, a gunshot reconstruction expert (he once performed a ballistics analysis by shooting dogs from the pound), a crime scene investigator, a blood spatter expert, a ‘tool mark’ expert, a fingernail scratch expert, and an expert in ‘liquid splash patterns.’” In the case in question, this dentist(!) used Photoshop and his PC to “enhance” surveillance video that  he claimed showed details about the alleged crime. Problem was, the FBI had processed the same video and said no such details appeared. Apparently, the prosecution did not see fit to share that FBI analysis with the defense or the court, and the defendants were sent to prison for ten years before their conviction was overturned due to that omission.

Go read the whole thing; it’s worth it.  ..bruce..

Cross posted from And Still I Persist]

[Note: I am currently in transit from Colorado to Florida and am composing this post as I have time and 'net access.]

“All the most important mistakes are made on the first day.”

- The Art of Systems Architecting (Maier & Rechtin)

Project Orca was the Romney campaign’s technological effort to track in near-real-time actual voting in precincts all over the United States, with the intent of using that information in combination with individual-specific demographic databases to increase or pull back phone-call and door-knock get-out-the-vote (GOTV) efforts based on what was happening there. Since it appears more and more that Romney lost largely because of a lack of Republican turnout, it’s safe to say that the Romney/RNC GOTV effort was a failure. What is less clear is how much of an impact Orca had on that failure.

Three important disclosures up front. First, I was a Project Orca volunteer in Colorado. Second, I have no direct knowledge of how Orca was envisioned, developed, and operated beyond my own experience as an end user. Third, I’ve worked in software engineering and information technology since 1974, and my major professional focus since 1994 or so has been on how and why IT projects fail or succeed — a subject on which I’ve taught seminars, published books and articles, consulted in large corporations, and testified in courts and before Congress.

That said, the flap over Orca has made it all the way to the Drudge Report, largely due to two key discussions. The first is by JohnE over at Ace of Spades, who wrote the initial scathing report on Orca and followed it up with a reply to some of the push-back from the Romney campaign. The second is by Joel Pollack over at Breitbart, who had a Romney worker in Colorado who spoke of some of the problems.

My own experience matches much of what JohnE (who was also in Colorado) stated. I participated in four training calls. The first three all gave the same information about how the system would work, though — as JohnE noted — the Romney people kept refering to the Orca software as an “app”, even though it was simply a website that you logged into. (From JohnE’s comments, it sounds as though the use of the term “app” confused at least some Orca volunteers.)

Unlike JohnE, I was very clear from both the calls and the e-mails I receive that I needed to pick up my poll watcher certification and bring it with me to my assigned polling place on Election Day, so I am curious about his confusion. On the other hand, for my particular county (Douglas County), it appears that the certifications weren’t ready until the day before Election Day, which strikes me as cutting it a bit close. But I did drive down to Castle Rock on Monday afternoon and picked it up.

There was one more call the night before Election Day, but that was just a very brief, rah-rah call. After that Monday night call, I tried logging into the Orca web site to test out my password, but the log-in failed. I found that puzzling, but just assumed that they had the system locked down ether for security or administrative purposes.

Since Colorado is one of the few states that does not allow any electronic devices (phones, tablets, laptops) in polling places, I was also very clear from the training that I would have to print out the list of registered voters, bring it with me, mark off voters as they identified themselves at the polling place, and then periodically run out to my car and enter voting information via my iPad or iPhone. I printed out the list — and immediately discovered what I considered the first, admittedly minor, glitch: the list, a PDF file over 60 pages long, was formatted in such a way that you could not three-hole-punch the sheets (to put in a binder) without punching through the check boxes (for tracking voting and reporting) for some of the voters on each page. As I said, minor, but my technical writing background made me frown a bit. Since I own Adobe Acrobat, I solved the problem via the kludge of adding headers and footers to each page and then telling the pages to shrink themselves to fit.

So, with two binders in hand (I printed two copies of the list), with iPhone, my iPad, and my laptop in the car — along with two Black & Decker power boxes, and a couple of folding camp chairs — I showed up at my polling place at 6:45 am. I also brought with me three dozen fresh donuts (a suggestion from one of the training calls) that were greatly welcomed and did much to smooth relations with the actual poll workers.

Everything went well until the first time I went out to the car to report voters. I tried to log into the Orca website, and once again I got a “login failed” error message. (Second minor but telling item: that error message had a typo in it. The fact that such a typo was in a high-probability error message in a live system with over 30,000 anticipated users speaks to haste and sloppiness.) I called the help number given in the error message, and got a support person at the Boston command center who said they would reset my password and gave me the new code, but that it would take a little while for the new code to go live.

When I came back out again an hour or so later and tried to log in, the new password didn’t work either. The Orca system also had a call-in option, so I tried that and was told that my PIN was invalid.

At this point, it was about 8:30 in the morning. The next eight or so hours were all more of the same — I would call, the password would be reset, I would wait a while, I would try to log in, it would fail. Rinse and repeat. I was told by some of the support people I spoke with that almost no one in Colorado was able to log in — a geographical uniqueness that I found puzzling, since the login screen just asked for username (an e-mail address) and password. Another support person mentioned that there were similar problems being reported for North Carolina. Finally, sometime around 4:30 to 5:00 pm, I was given a new pin for the dial-in line — and that finally worked. I reported all the voters who had voted so far that day and did updated reports twice more before the poll finally closed at 7:00 pm.

JohnE pushes back (with justifiable skepticism) against Zac Moffat’s claims that “the campaign had voting data from 91 percent of counties” — not that they is necessarily inaccurate, but that it could be largely irrelevant. As JohnE points out, there can be hundreds of precincts in a single county; beyond that, having voting data from a given county does not necessarily mean having actionable, timely, and sufficiently complete data throughout the day in order to make GOTV decisions for that county (or for a given precinct in that county).

There is still no evidence that there were significant Orca problems outside of Colorado; we’ll have to wait to see if someone on the inside of Orca development decides to leak. But here are some observations from an large-scale IT system development perspective.

“A complex system that works is invariably found to have evolved from a simple system that works. . . . A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.”

– Systemantics (John Gall)

First, the Orca folks basically released version 1.0 (or, perhaps more accurately, version 0.9 or lower) into a massive, live environment on the single most critical day, Election Day. This is commonly known in IT circles as the “Big Bang” approach, and it is a well-known harbinger of disaster. Even if the Orca developers were all geniuses, this still would be a grave error. We’re not talking about an iOS or Android app being used by a single person on his/her device for entertainment or productivity; we’re talking about a massive web-based system that has to accept data from (not just deliver content to) possibly tens of thousands of users simultaneously and in real time. There should have been days, if not weeks, of live tests and trials to see how the system performed and to get the kinks worked out. Instead, as far as I can tell, the first time most Orca volunteers actually used (or tried to use) the system was on Election Day. Even if the system worked perfectly, I doubt that the majority of users — who had no real training on the system — would have used it in an effective manner.

“There are products that you shouldn’t develop, companies you shouldn’t challenge, customers you shouldn’t win, markets you shouldn’t enter, recommendations from the board of directors you shouldn’t follow.”

– The Art of ‘Ware (Webster)

Beyond that — and as per the quote above (paraphrasing Sun Tzu), as well as the one at the start of this post — the real question is: should this have been attempted in the first place? There are anecdotal reports that the focus on Orca took away resources — especially feet-on-the-ground resources — for local GOTV efforts. And GOTV is exactly where the Republicans failed. If work on Orca had been started 3 or 4 years ago, and if it had been used on Election Day 2010 — both as a real-world test and to verify the expected benefits of matching actual voters with demographic information to predict results in a given precinct — then it might have been ready for prime time on Election Day 2012.

But to spend signficant time, money, and resources (including tying up some 30,000+ local volunteers) for an untested system and an untried concept was a recipe for disaster — a well-known recipe, I might add, for those of us who have studied large-scale IT projects (which, frankly, are prone to failure anyway).

“But,” you ask, “you signed up to volunteer.” Yes, I did. But I had no visibility into the project and no inherent reason in advance to assume that things weren’t being done well, beyond the few niggling questions I had prior to Election Day. And, to be perfectly fair, I have no evidence of the extent of problems beyond Colorado, aside from the acknowledged 90-minute downtime on Election Day morning. But, even if those are the only two failures, can you seriously argue that it was a success? A time-critical, mission-critical system whose entire purpose for existence is to be used in a single 12-hour window is down for over 10% of that time in all states and is down all day in one critical swing state?

And, finally, the bottom line: if the purpose of Orca was to provide real-time intelligence on actual voting trends and make GOTV more effective, then where are the results? By all accounts, the Romney campaigned was stunned by their loss; if Orca was providing accurate, real-time indication of voting, as the project directors claimed it would in the training calls (actual quote [from memory]: “We’ll know ahead of anyone else how the election is turning out.”), then Romney should have had his concession speech written well in advance. And if its goal was to trigger targeted GOTV efforts in swing states, then why did Romney lose almost every swing state? Measured by its stated goals, Orca was a failure, pure and simple. Worse yet, it may have diverted critical resources from more effective GOTV efforts.

“Start out stupid and work up from there.”
– Bruce Henderson

I’d love to be able to do a project post mortem on Orca, as I have done on so many troubled, failing, or failed projects. But even without access to people and documents, I can hazard a guess at some of the likely factors:

• A late start and an impossibly short schedule.
• Because of these above, a waterfall (single pass) approach to design, development and deployment, instead of an iterative approach.
• Lack of a single chief software architect, with resulting problems at subsystem interfaces.
• A large development team with no prior track record developing and releasing software as a single team.
• Along the same lines, key components divided up among separate teams with relatively late integration and testing.
• And possibly some turnover (departure) of key personnel through the project, along with new personnel added late.
• Insufficient resources (money, personnel, equipment, time) devoted to quality assurance — and by “quality assurance”, I mean not just a wide range of testing (including, as JohnE points out, stress testing), but also individuals and teams with proper expertise, agreed-upon standards and guidelines, reviews, metrics, defect management, and so on.
• A firm belief — without any real-world evidence — that the idea would actually work and that the results would actually be worth the diversion of resources (including volunteers).

I’m happy to talk with anyone from the project who wants to set me straight or explain what really happened. But, in the end, what really happened was that Orca failed to achieve its goals, and Romney lost what should have been a very winnable election. ..bruce w..

[The second in a series of posts on Readings in Software Engineering. Previous post: The Psychology of Computer Programming, Gerald M. Weinberg (1971/1998)]

The Mythical Man-Month: Essays on Software Engineering, Frederick P. Brooks, Jr. Addison-Wesley Publishing Company, Reading, MA, 1975. Softbound, 195 pages. Personal acquisition date: unknown. Original edition out of print.

The Mythical Man-Month: Essays on Software Engineering (Anniversary Edition), Frederick P. Brooks, Jr. Addison-Wesley, Boston, MA, 1995. Softbound, 322 pages. Personal acquisition date: unknown (I’ve bought and given away several copies over the years). Available in Kindle edition as well.

Seminal work; highest recommendation.

===========================

(All quotes and citations are from the 1995 Anniversary edition, which contains the slightly-edited text and original page numbering of Brooks’s original book, followed by four new chapters: two additional essays, and two retrospectives on the original edition.)

The Mythical Man-Month is probably the single best-known work in software engineering, as well it should be: it is a classic, in every sense of the word.

First of all, it is (for a thin volume) a remarkably comprehensive look at the practice of professional software engineering. Brooks’s essays cover a wide ranges of topics, with a particular focus on where things are likely to go wrong and what must be done to make things go right. For years, whenever I thought I had conceived some new issue or principle in software development, I would go back and look through Brooks and discover he had already noted it.

Second, it is highly accessible. Brooks’s style is clear, concise, and very readable. It is a book that anyone in a given organization should be able to read and understand. Furthermore, these are true essays: wonderfully written and downright lyric in places. One of my favorite passages is at the end of his discussion as to the ‘Joys of the Craft [of Programming]‘:

Finally, there is the delight of working in such a tractable medium. The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. (As we shall see later, this very tractability has its own problems.)

Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.” (pp. 7-8)

Finally, it remains highly relevant for our day, perhaps sadly so. Much of my professional work deals with troubled or failed IT projects, and my analysis of their struggles — whether in vivo or post mortem — usually reveal the same core problems that Brooks warned us of nearly 40 years ago. It seems that in the software industry, we not only insist on re-inventing the wheel, we insist on re-inventing the flat tire. As I said in my look at The Psychology of Computer Programming, billions of dollars wasted in failed IT projects could have been saved if those involved had read and heeded Brooks and Weinberg.

Read the rest

[The first of a planned series of posts on "Readings in Software Engineering"]

[Version 1.1 of this post, revised/extended on 05/22/2012]

The Psychology of Computer Programming, Gerald M. Weinberg, Van Nostrand Reinhold Company, New York, 1971. Hardbound, 288 pages. Personal acquisition date: 17 Oct 1978. Original edition out of print.

The Psychology of Computer Programming (Silver Anniversary Edition), Gerald M. Weinberg, Dorset House Publishing, New York, 1998. Softbound, 292 pages. Personal acquisition date: unknown. Currently in print (and in ebook form).

Seminal work; highest recommendation.

===========================

(All quotes and citations are from the 1998 Silver Anniversary edition, which contains the unedited text and original page numbering of Weinberg’s original book interspersed with his observations 25 years later using a different pagination scheme.)

If The Mythical Man Month (Frederick P. Brooks, Jr., 1975/1995) is the book that most IT managers own but few ever read (and even fewer follow), The Psychology of Computer Programming (Gerald M. Weinberg, 1971/1998) is the book that almost nobody nowadays owns…at least, nobody under a certain age. And that’s a tragedy, because Weinberg wrote one of the first and still one of the best works on the human factors in IT project management and software engineering.

I bought my own first copy of Weinberg’s book in the fall of 1978, nearly 35 years ago, just a few months after joining General Dynamics with my freshly-minted BS in computer science from BYU. Weinberg’s book had a tremendous impact on me and forever changed my view of software development from a technical activity to a human, social activity. In fact, Weinberg’s first sentence in his first chapter is just that: “Computer programming is a human activity.” That single insight — with all it implies — is something that the IT industry seems to re-discover on a regular basis and then forget again just as quickly as it seeks the ever-elusive magic amalgam of technology, methodology, and language (with minimum pesky humans) that will make software development fast, cheap, and good. Weinberg touched upon that very quest early on in the book as well :

[1971] Over the years, executives have backed their desire to eliminate programmers with staggering funds. Dozens of simplistic schemes have been heaped with money and praise on the promise — as yet not kept — of going directly from a sales proposal to a working data-processing system. (p.4)

[1998] The only thing that’s changed here in twenty-five years is the fact that the funds dedicated by executives to eliminating programmers from their payrolls have become far more staggering than I imagined back then. (P1.i)

Furthermore, Weinberg got to many of Brooks’ key insights four years before Brooks published The Mythical Man-Month, including touching (in a slightly different way) on the most famous one: adding more people does not get a job done more quickly, and using exactly the same analogy that gave Brooks’ book its title:

For example, certain programming work cannot be done by a team of trainees, no matter how large, so that doubling the number of “warm bodies” — as they are so often called in the trade — still will not get the work done. Schedule is similarly limiting — we need only cite the apocryphal experiment which tried to make a baby in one month by putting nine women to work on the job as a team. (p. 68)

As per the table of contents given at the end of this post, Weinberg covered four major themes in his book: Programming as Human Performance; Programming as a Social Activity; Programming as an Individual Activity; and Programming Tools. As Weinberg himself points out (in his 1998 commentary on his 1971 work), he was really writing more about the anthropology of computer programming, rather than the psychology per se. The book is highly readable, largely because Weinberg has a conversational style of writing and fills the book with actual stories and anecdotes of observed behavior on programming projects.

Read the rest

I have been collecting and reading books on software engineering since the 1970s, but I have found over the decades that the vast majority of programmers (and their managers) are unfamiliar with most of them. More’s the pity, for during the 38 years since I first started working in information technology (BYU Translation Sciences Institute, 1974), I have observed not only the truth of the French proverb Plus ça change, plus c’est la même chose (“The more things change, the more they stay the same”) but also the truth of George Santayana’s observation:

Progress, far from consisting in change, depends upon retentiveness….Those who will not remember the past are condemned to repeat it. (The Life of Reason, 1905)

In my opinion, most of core issues in software engineering and IT project project management were identified decades ago, but our industry suffers from vast institutional ignorance of these works — ‘ignorance’ both in the sense of ‘unaware of’ and ‘paying no attention to’. As I once said in a rather interesting setting¹:

Humanity has been developing information technology for half a century. That experience has taught us this unpleasant truth: virtually every information technology project above a certain size or complexity is significantly late and over budget or fails altogether; those that don’t fail are often riddled with defects and difficult to enhance. Fred Brooks explored many of the root causes over twenty years ago in The Mythical Man-Month, a classic book that could be regarded as the Bible of information technology because it is universally known, often quoted, occasionally read, and rarely heeded.

To that end, and because I have not been posting much here, I am introducing a new series of posts: Readings in Software Engineering, or RISE. Each post will cover one book (including subsequent editions). I will give what context I have for the book, then quote interesting or relevant excerpts from the book, followed at the end by the book’s table of contents. My intent is not to present “one page summaries” of these books, as are often found for various business books; instead, it is to pull out relevant quotes, in the authors’ own words, in hopes that you’ll actually track down the books and read them yourselves.

A final note: I reserve the right to go back and substantially edit my RISE posts after the fact, particularly as I get several of these under my belt and start to settle in on a standard approach and format. Iterative writing, so to speak. ..bruce..

¹In testimony before Congress in 1998. Long story, had to do with Y2K.