A few years later — in 1982 — my close friend Wayne Holder hired me into his nascent software company, Oasis Systems, in part to help with his existing and planned word processing utilities (The Word Plus, Punctuation + Style), but mostly to develop computer games. And we did, developing Sundog: Frozen Legacy on the Apple II, a game for which I still get e-mails (and which Wayne is even now working on resurrecting for modern platforms). In January 1984, a few months before Sundog shipped, we were invited by Guy Kawasaki to come up to Apple to see a preview of the Mac and to talk about what software we could port to the Mac. Through my connections with computer stores in San Diego, I was able to get a personal loan of a Mac for a few days at home prior to the official announcement in Cupertino later that month, which Wayne and I attended as well. That was my first time seeing Steve Jobs in person, and it remains a memorable highlight of my professional life.

When the Mac shipped a few days later, I went down to the one computer store in San Diego that I knew would be getting machines from Apple. I took $3000 in cash with me and managed to convince the store owner — a friend — to let me have one of the three Macs he had to sell. Through a connection with Phil Lemmons — editor-in-chief at BYTE — I ended up writing the official BYTE review of the 128K Macintosh (August 1984 issue). By the end of 1984, I was writing full-time for BYTE, including on-going coverage of the Macintosh, particularly once my BYTE column started in mid-1985. After a few years of writing for BYTE, I switched to writing for Macworld magazine. Steve was now long-gone from Apple, and Apple was having some of its own problems going forward.

But in late 1987, I was contacted by Addison-Wesley. They were interested in having me write a book about Steve Jobs’ new project at NeXT. Folks at NeXT had apparently suggested me to Addison-Wesley, probably due to my writing at BYTE and Macworld. I leapt at the opportunity, particularly since in coincided with our family moving from Utah to just outside Santa Cruz (where I would be doing technical writing for Borland on a consulting basis). Once there, I found myself invited to visit NeXT HQ on Deer Creek Road, sit in on meetings, and attend the 0.3 NeXTstep Dev Camp. And, yes, that meant getting actual face time with Steve Jobs as well — not a lot, but this was a man whose creations had been impacting my personal and professional life for over a decade at this point.

The writing of the book dragged out as I waited to get my hands on an actual NeXT cube, which finally happened (if I recall correctly) at the end of 1988 or early 1989. I wrote the first several drafts of the book on that NeXT cube itself. The book came out in the fall of 1989; it remains the single most successful book I’ve ever written, due to the intense interest in NeXT itself, more than any particular writing skills or technical insight on my part.

The following year, I found myself working with a world-class typographer (Mike Parker) and graphic designer (Vic Spindler) to create a design-oriented desktop publishing system. I was doing all the software prototyping on my NeXT cube, and we made the decision to make the NeXT our first target platform. For five years — 1990 to 1995 — I served as chief architect and CTO at Pages Software Inc, where we developed Pages by Pages and then WebPages, while spending nearly two years just trying to raise venture funding. We closed on funding at the start of 1992 and shipped our first version of Pages in early 1994. We quickly sold all that we were going to in the all-too-small NeXTstep market. My frustrations at seeing larger firm try to leverage off of NeXT’s incredible innovations led to an op-ed piece in the November 1994 issue of BYTE, “Whither NextStep?” The day that issue came out was the last time that Steve Jobs and I spoke — he called me from the back of a car somewhere to ask me what the hell I was doing writing that. I said, telling the truth. Pages would close its door the next year, unable to secure additional funding to move its technology to Windows.

When Steve engineered his brilliant reverse takeover of Apple — getting Apple to buy NeXT for $400 million, then slowly moving himself into the CEO seat — I was not optimistic. I still had unconditional praise for the NextStep technology, but I was dubious about Steve’s ability to sell technology to markets and to compete with Microsoft.

Boy, was I wrong. I was not only wrong about his abilities at Apple, I was wrong in my BYTE article about NextStep being on a downward slope. NextStep, of course, was the foundation of Mac OS X, and Steve transformed Apple into the most-admired, most-imitated, and most-valuable company in the world. And I was tickled that, when Apple brought out its own word processor, it was named “Pages”. Steve had always liked that name when we were developing (and shipping) our own product years before; glad he was able to use it.

To quote John Perry Barlow over on FB, “The world is suddenly a less interesting place.” ..bruce w..

[1] The first was an HP-67 card-reading programmable calculator.

[Cross-posted from And Still I Persist]

Now, thanks to advances in artificial intelligence, “e-discovery” software can analyze documents in a fraction of the time for a fraction of the cost. In January, for example, Blackstone Discovery of Palo Alto, Calif., helped analyze 1.5 million documents for less than $100,000.

Some programs go beyond just finding documents with relevant terms at computer speeds. They can extract relevant concepts — like documents relevant to social protest in the Middle East — even in the absence of specific terms, and deduce patterns of behavior that would have eluded lawyers examining millions of documents.

“From a legal staffing viewpoint, it means that a lot of people who used to be allocated to conduct document review are no longer able to be billed out,” said Bill Herr, who as a lawyer at a major chemical company used to muster auditoriums of lawyers to read documents for weeks on end. “People get bored, people get headaches. Computers don’t.”

Since I work in litigation support myself and have to search and read documents — I’ve had individual cases with over a million pages of documents — I welcome such tools as these. Still, the overall theme in the article seems to be that humans will lose jobs due to these tools:

Quantifying the employment impact of these new technologies is difficult. Mike Lynch, the founder of Autonomy, is convinced that “legal is a sector that will likely employ fewer, not more, people in the U.S. in the future.” He estimated that the shift from manual document discovery to e-discovery would lead to a manpower reduction in which one lawyer would suffice for work that once required 500 and that the newest generation of software, which can detect duplicates and find clusters of important documents on a particular topic, could cut the head count by another 50 percent.

What the article does not address is the enormous financial burden that e-discovery has imposed on both parties in modern civil litigation (see also here and here).

It used to be that each side would gather physical documents from files and storage boxes, review them for responsiveness, then produce them to the other side. Outside of large corporations, most parties would not have great volumes of such documents.

However, with the advent and pervasiveness of digital technology — computers, e-mail, servers, instant messaging, and so on — even a relatively small firm or organization can find itself with gigabytes, if not terabytes, of potentially relevant production. And, to avoid court sanctions, all of that has to be combed through and reviewed for responsiveness. Most organizations below a certain size just can’t afford to do that the old-fashioned way with “a platoon of lawyers and paralegals who [work] for months at high hourly rates”.

The article also does not address the existing document organization and search tools that are a bit less artificially intelligent — and a lot cheaper — than the tools he covers, but that still allow complex search terms to be set up. I make heavy use of dtSearch in most of the cases I work on (and can recommend it highly). I’ve also used such standard legal document management tools as Summation and Concordance.

After conducting a series of experiments comparing a sample Corsair 64GB SSD with a conventional Hitachi 80GB magnetic hard drive (HDD), the team found a layer cake of data recovery problems caused by the ‘garbage collection’ or purging algorithms used in SSDs to keep them at peak performance.

After examining an SSD for traces of data after it had been quick formatted, the team expected the purging routines to kick in around 30-60 minutes later, a process that must happen on SSDs before new data can be written to those blocks. To their surprise, this happened in only three minutes, after which only 1,064 out of 316,666 evidence files were recoverable from the drive.

Going a stage further, they removed the drive from the PC and connected a ‘write blocker’, a piece of hardware designed to isolate the drive and stop any purging of its contents. Incredibly, after leaving this attached for only 20 minutes, almost 19 percent of its files had been wiped for good, a process the researchers put down the ability of SSDs to initiate certain routines independent of a computer.

For comparison, on the equivalent hard drive all data was recoverable, regardless of the time elapsed, as a forensic examiner would expect

“Even in the absence of computer instructions, a modern solid-state storage device can permanently destroy evidence to a quite remarkable degree, during a short space of time, in a manner that a magnetic hard drive would not,” the team concludes.

The results are concerning on a number of levels, forensic, legal and technical.

Yeah, I’ll say, though I suspect most of the concerns are legal and forensic. Hat tip to Slashdot.  ..bruce..

According to Gleick, the impact of information on human affairs came in three installments: first the history, the thousands of years during which people created and exchanged information without the concept of measuring it; second the theory, first formulated by Shannon; third the flood, in which we now live. The flood began quietly. The event that made the flood plainly visible occurred in 1965, when Gordon Moore stated Moore’s Law. Moore was an electrical engineer, founder of the Intel Corporation, a company that manufactured components for computers and other electronic gadgets. His law said that the price of electronic components would decrease and their numbers would increase by a factor of two every eighteen months. This implied that the price would decrease and the numbers would increase by a factor of a hundred every decade. Moore’s prediction of continued growth has turned out to be astonishingly accurate during the forty-five years since he announced it. In these four and a half decades, the price has decreased and the numbers have increased by a factor of a billion, nine powers of ten. Nine powers of ten are enough to turn a trickle into a flood.

I’ve lived through most of the information age (if we peg it at 1940 or so; I was born in 1953), and I’ve worked in information technology for almost 40 years (since 1974). I remember seeing a second megabyte of core memory that was being added to the IBM 360/65 on campus in the mid-1970s: it was the size of a giant shoebox, and it cost $250,000. Now the cost per megabyte for 1GB RAM SIMMs sold over Amazon is as low as $0.02, and there’s 1,024 megabytes on that little stick. I have something approaching 15 terabytes of hard disk storage on the various computers in our house, and — as per Dyson’s review and Gleick’s book — my biggest challenge is keeping it organized and accessible. There’s a high level of redundancy, with numerous minor (or major) variations; at the same time, I fear that a disk failure will forever deprive me of files that I might someday want or need again, whether that’s tomorrow or ten years from now.

I’ll read the book when it comes out (I have a copy preordered from Amazon); in the meantime, Dyson’s review is well worth reading.

At the same time, classic rotating storage devices — such as platter-based hard disk drives — are slowly being supplemented and in some cases supplanted by solid-state media, that is, storage devices that have no moving parts at all. In particular, solid-state disks (SSDs) look and behave just like classic hard disk drives, but with usually faster response times and much lower chance of hardware failure. In particular, USB drives have become ubiquitous as a mechanism for moving files between unconnected computers, while laptops are starting to offer SSDs as the principal internal storage for both speed and weight improvements.

So far, so good. Except that out of the University of California at San Diego comes research that shows that in many cases, standard file- and disk-erasing techniques leave behind recoverable data when used on SSDs:

At the Non-volatile Systems Laboratory we have designed a procedure to bypass the flash translation layer (FTL) on SSDs and directly access the raw NAND flash chips to audit the success of any given sanitization technique. Our results show that naïvely applying techniques designed for sanitizing hard drives on SSDs, such as overwriting and using built-in secure erase commands is unreliable and sometimes results in all the data remaining intact. Furthermore, our results also show that sanitizing single files on an SSD is much more difficult than on a traditional hard drive. We are working on designing new FTLs that correct these issues and also exploit properties of flash memory to maintain performance while sanitizing the flash drive.

I imaging that word of this will quickly spread to companies that perform computer forensics (including recovery of data for storage devices). In the meantime, many organizations may continue to make us of USB drives and internal SSDs in laptops (and, eventually, desktops), and by so doing may leave themselves open to discovery of data that they thought purged in the course of normal, documented operations.

The other day, an IT colleague of mine mentioned a conflict at a corporation where he’s working. The corporation has a mission-critical application deployed across a large number of workstations. The set of corporate employees who use this application largely use it and nothing else all day long at dedicated workstations. The application they are using is a customized third-party application; however, the firm has been having chronic problems with this app (let’s call it “QRSApp”), and so is looking at different solutions. The firm could continue to make changes to QRSApp to fix their problems. The firm could switch to a different third-party application; several other vendors market applications of this type within this firm’s industry. Or, as a senior IT manager now wants to do, the firm could develop a completely custom and private application to replace QRSApp, so that the firm has complete control over it.

The question: which solution is best?

Comments welcome here or there.  ..bruce..

Categories: managerial

The impulse to constantly add new and incremental features to a software program certainly isn’t unique to modern software develoment, or to a particular technology or methodology. It derives largely from three sources. Upper management and marketing want, and sometimes need, those additional features to win a customer, to better position the product against a competitor, or to better justify the expense of an in-house project. Engineers often find it far more fun, more interesting, and more rewarding to add or extend features than to make existing features work completely and perfectly. And customers supply a constant flow of demand for new and improved functionality.

The problem is that many modern technologies and methodologies intensify all these tendencies. With a decent software design and implementation, based on current technology, it can take literally only a few minutes to add or extend features. Frankly, that’s the payoff that lures many engineers to modern technologies in the first place: They get to do really neat stuff, do it quickly, and impress the boss while they’re at it. The boss, having seen how quickly engineers can extend or create functionality, doubles or trebles the list of must-have features. The technical manager, caught in the middle, wonders how to deal with all this.

Symptoms

Engineers focusing on adding new features instead of getting old ones working completely. Upper management passing down long (or even short) lists of additional features when the engineering schedule will be hard-pressed to accommodate the current ones. Customers failing to distinguish between what they’d like and what they’re willing to live with.

Consequences

Features that don’t work as completely or well as they were intended. Missed milestones and schedule slippage.

Detection

Review all existing features with the engineering team and get an honest assessment of where each feature is and what it will take to complete it. If necessary, do a design and code review for each feature. Compare the documented “essential features” list—and the schedule required to complete them—with the current feature set as well as those proposed by engineers and upper management, and find where the differences are coming in.

Extraction

Nothing fancy here: Get the absolute “drop dead” release date for the software and work backward from there, allocating time to design, implement, debug, and test each feature that is not yet completed. Show everyone (especially customers) the gap between the time and resources allocated those required for just the currently planned and requested features. Use whatever process works in your company for performing feature triage: keeping those that are absolutely essential, dropping those that can wait for the next release, and negotiating on the others.

Prevention

Use the process given in Extraction before a line of code is ever written. As features are proposed, collect them in a list, but do not allow any work on them. At intervals, repeat the process: You might have time (or need) to schedule some of the proposed features, but you may well have to drop previously required ones.

As with a prior pitfall on features, this pitfall is so an obvious and well-known that I’m almost embarrassed to include it here. Again, it is still so common that I must.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: managerial

If your organization is adopting some new technology or methodology (the “TOM”), it is likely because of wonderful claims about how it will improve your software engineering efforts: faster development time, higher quality, lower complexity, and so on. Leaving aside the likelihood of achieving such claims, these hypothetical benefits still inspire or tempt managers and developers alike to leap feet first into adopting the TOM, using it to tackle large projects with short deadlines.

The result is like crossing a swamp. The ground starts out a bit damp and the undergrowth a bit thick, but you can make good progress with your new TOM machete. As you push ahead, however, you face mud, vines, snakes, water, and alligators in ever- increasing proportions. You may choose to push ahead or to go back, but in either case the effort and cost are greater than if you had used caution before plunging in. And those above you want to know what happened to all the wonderful benefits of this TOM.

Symptoms

A project that doesn’t seem to be making much progress or one that seems ill-defined.

Consequences

Schedule slips and project failure. If initial adoption of the TOM in question (or some key variant thereof), loss of acceptance of that technology and loss of professional credibility.

Detection

Enumerate all that you hope to accomplish with this project. See how soon in the process the coding and prototyping started. Measure current status against the announced schedule and make a realistic assessment of where you are.

Extraction

This can be really tough. Once you’re in the middle of a project, you have a lot of people counting on you to deliver certain solutions by a particular date. You might be hesitant to stop everything and plan the entire development effort again. On the other hand, there is a good chance — in fact, almost a certainty — that if you have stumbled into this pitfall, your efforts to get out of it may not be fully understood or appreciated by those in authority.

Nevertheless, the best solution is to halt development and take the time to scale down the project and train developers; then you can set realistic deadlines.

Prevention

Look again at the pitfall: too much, too soon, too fast. First, start out small when using the new TOM for the first (or second or third!) time. Use well-defined, small-scale pilot projects. Or whittle down your large project into a small one on which you can build in later development cycles.

Second, take time for education, analysis, and design. Make sure that everyone involved understands the concepts, techniques, and — yes — the pitfalls of this particular TOM. Resist the temptation to start prototyping and coding immediately; go through an explicit, detailed analysis phase and take time to go through a design phase.

Third, plan for your initial TOM-based projects to take at least as long as they would if developed using traditional software engineering approaches. You may be pleasantly surprised; you almost certainly won’t have a rude awakening. After you get several projects under your belt, you’ll become more accurate at projecting development cycles.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: managerial

Why would the use of a new technology or methodology (the “TOM”) cause managers and developers to neglect or even abandon solid software engineering practices? Because those practices are under pressure from the start. Many engineers don’t know them and aren’t willing to spend (or aren’t given) the time to learn them. Those who do know them often aren’t willing to spend (or aren’t given) the time to practice them. Technical managers, who often do understand such techniques, find themselves caught in a vise, trapped between ever-increasing (and possibly unrealistic) demands of upper management and the productivity level of the engineers they supervise.

The introduction of a new TOM can cause a new set of problems. Upper management, having read articles about the benefits of this particular TOM, may see it as an opportunity to increase demands. Developers, already under pressure to produce, milk the benefits of the TOM for all they’re worth and may find even less incentive to focus on solid software engineering. And the technical managers in the middle have to cope with the specifics of the TOM while still fighting the software engineering battles.

Symptoms

Upper management expects the TOM to immediately shorten the software development lifecycle and is unwilling to allocate the necessary time and resources pilot projects. Engineers neglect common sense and best practices, feeling instead that the TOM somehow magically supersedes both. Unrealistic expectations from upper management puts pressure on the development team, resulting in even more shortcuts.

Consequences

Failure to achieve expected benefits of the TOM, possibly leading to (unwarranted) rejection of the TOM for future projects. Delayed, buggy, and/or failed projects.

Detection

It’s usually not hard to detect the pressure from upper management; indeed, the problem is resisting the pressure. As for the engineering-level problems, the best detection comes from regular team meetings and project reviews to go over standard (TOM-independent) software engineering practices.

Extraction

This pitfall is a hard one to get out of, because upper management needs to allocate more time and resources, and the engineers need the self-discipline and education to change how they’re doing things. The key is to educate everyone about the benefits of good software engineering — established practices, predictable schedules, and higher quality — and then work the TOM into that setting.

Prevention

It is sad that there is often so much resistance to doing things right in the first place. The approach is direct: realistic scheduling, definition of engineering standards, and enforcement of engineering practices. For starters, make everyone — especially upper managemen t– reads The Mythical Man-Month by Frederick P. Brooks. Then refer to it again and again as these problems arise.

References

Webster, from Pitfalls of Object-Oriented Development (1995).

Categories: political

Face it: technology adoption is a gamble, and one often made at gunpoint. External market forces lead internal political forces to demand advances — often unrealistic ones — in information technology. Those who adopt too soon come to understand the cliché “bleeding edge”. Those who adopt too late lose opportunities and possibly market share. And so you find yourself looking a dozen new technologies and trying to decide which to adopt, knowing that only two or three are likely to succeed: that is, deliver what you need and grow with your ever-increasing demands. A winner is always easier to pick in retrospect, but you seldom have that luxury. Instead, you need to decide between adopting a “safe” technology — which may not, in the end, really meet your needs — or leading out with a newer technology and thus run the risk of finding yourself in a dead end with a non-standard or even unsupported solution or approach.

But that’s why they pay you the big bucks, right?

Just as a personal note, I spent five years (1990-95) and $7 million in venture capital as part of a software start-up that developed and shipped a design-oriented desktop publishing system for . . . the NeXTstep operating system. As I tell people, it seemed like a good idea at the time. It was a great operating system (still is — go look at Mac OS X) and certainly better than anything else out in 1990. And we were in good company at the time, with firms such as WordPerfect, Ashton-Tate, and Adobe all developing for NeXTstep as well.

But it was the wrong horse.

Symptoms

Peers and managers above you start questioning your choice of technologies. The number of other organizations adopting the technology in question doesn’t seem to be growing much, or is even shrinking. Developers experienced with the technology are hard to find.

Consequences

Your system and/or development effort struggles due to the lack of developers and third-party tools for the technologies in question. If you’re developing a commercial product, you may find a very limited market, assuming you find one at all. If you’re developing an in-house system, you may end up deploying it, but the system may be difficult to maintain and keep in production for its original hoped-for lifetime. And if you were the champion for adopting that particular technology, you may find yourself losing influence, missing out on promotions, and possibly even out of a job.

Detection

Track down and talk frequently with other firms adopting the same technology. Fact-check, as best you can, claims by the technology’s vendor regarding adoption and market share. Pay particular attention to the products or systems based on this technology that are actually deployed into real-world use.

Extraction

If you think you’ve picked the wrong horse, then start to find the right one, quickly. Figure out what it’s going to take to move over to the better (or best) technology. Work up a careful transition plan — as accurate and conservative as you can make it — and then figure out what it’s going to take to sell this to upper management. There may be no good or easy paths at this point.

Note that there are cases where you may well be best off staying on the technology you originally picked, particularly if this is an in-house system. That is, it may be more cost effective to push into production, then plan for an earlier-than-expected end-of-life for that system, rather than try to switch horses in mid-race.

Prevention

This is hard precisely for the reasons stated above: in many cases, you just don’t know what will succeed and what won’t. If you’re talking about a major decision — such was what operating system to develop for and deploy on — your choices may be limited by other factors. In general, though, you want to avoid the leading/bleeding edges — never count on version 1.0 (or even 1.x) of anything. You want to choose technologies that have been out in the marketplace long enough to have the flaws exposed and — one hopes — the biggest problems addressed.

References

[Professional observation]