An approach to software release

The following excerpt is taken — with a few minor edits — from a white paper I wrote on quality assurance back in the late 1990s for a large corporate client. Given the rather intense coverage of the failings of the Healthcare.gov website — which clearly was not ready for prime time — I thought it might be useful. Note that while there are no hard-and-fast definitions for “alpha release” and “beta release” in the software industry, the ones I present here reflect a general consensus. Note that “deferred defects” are errors in the program that are found and logged, but that are deliberately left unrepaired for the time being. 

The crowning activity of quality assurance is the final verification that the application (or system) in question does provide the reliability, performance, and functionality required to warrant deployment to the intended users. The commercial software world usually does this with a four-step process: alpha release; beta release; deployment; and technical support and maintenance. The phased release provides an opportunity to test the application under deployment conditions while minimizing the business and productivity impact of any yet-undiscovered defects.

The software release process should be under the control of Configuration Management and under the supervision of the QA leader; it should not be under the control or supervision of the software development leader or the project manager. The goal of this process is not to add last minute capabilities, but to repair defects in reliability and performance while maintaining (or possibly decreasing) functionality. The QA leader must certify that the application is ready for deployment, with all the business and productivity implications that carries.

Note that while the definitions of alpha and beta release vary from source to source, these definitions here reflect the requirements of in-house application and system development and deployment.

Note that you must take great care with source code configuration management (SCCM) through this period. It should be easy to recreate each separate version through alpha, beta, and release; likewise, individual developers should be able to roll back to previous versions of individual classes and components. This is for both technical and management reasons. At this point in development, schedule pressures and work loads will be their greatest, which means that errors are most likely to be made at the point where you can least afford them. The ability to roll back quickly to an earlier build or version will be critical.

Alpha Release

An alpha release of an application (or system) is one that has acceptable functionality for release, though it may still have performance and reliability defects. Furthermore, that functionality should be an upper bound; that is, you should add no new functionality from this point on, and you may remove some in order to meet performance and requirement constraints. This constraint is critical; you should reflect it in a formal baseline. Should mandatory additions or modifications to functionality arise, development must drop back to a pre-alpha phase until you can again place limits on functionality.

The main activities of alpha release are system, performance, stress, destructive, documentation, and usability testing, along with repair or deferral of new and existing defects. You may have limited internal release among target users to supplement efforts by test and development engineers.

Each new alpha release must go through extensive regression testing to detect new defects that developers might have unknowingly introduced. Great care should be taken by development engineers at this point to minimize the impact of defect repairs and to avoid to the temptation to “clean up” sections of code or, worse yet, add unauthorized functionality. While these last two activities are certainly worthy aims, they greatly increase the risk of a prolonged alpha phase due to new defects.

Note that you will need to build and test both “debug” and “non-debug” versions of alpha releases. The debug version will help in tracking down defects, but there are often subtle and important behavioral differences between a version built for debugging and one built for deployment.

Beta Release

The alpha phase ends with the first beta release of the application (or system). A beta release is one that you deem to have acceptable functionality and performance for general deployment, though reliability defects may remain. In beta release, you should make no modifications (either adding or subtracting) to functionality; likewise, make no improvements to performance, except those that might inadvertently come about through defect repair. If you find that performance or functionality is unacceptable, then shift development back to an alpha or pre-alpha phase.

The prime activity of beta release is to repair or defer every remaining defect. You run each beta release through a series of regression tests (system, performance, etc.) to detect undiscovered or developer-introduced defects. You can deploy beta releases more broadly among target users to help uncover defects. Note: the users will have lists of suggested improvements. Save these for future versions of the application and not be implemented in this one.

Build and test beta releases as “non-debug” (deployment) versions. This may make it a bit harder to track down defects, but is critical that you do beta testing in a configuration as close to actual deployment as possible.

Deployment

When you have deferred or repaired all known defects, then you can build the first deployment release candidate. This candidate needs to undergo a complete set of system tests for an appropriate period; at the same time, deploy it to all beta testers. Log any new defects; defer the tolerable ones to be fixed (if ever) in some future version. If, however, you find any unacceptable defects, then you must be carefully repair them, build a new release candidate, and repeat the same testing. Similarly, if the rate of defects discovered seems high, then investigate and repair the cause of those defects.

Once a release candidate goes ends its wringing-out period with no unacceptable defects (i.e., any defects discovered are deferred), then you declare the candidate to be “gold”, and you begin deployment. Archive and store the current project in multiple locations (including off-site) to allow recovery and recreation in the case of catastrophe.

Technical Support and Maintenance

Once you deploy the application (or system), users will begin to discover all the defects that had been deferred. Set up a technical support or help desk to acknowledge these defects and provide any known workarounds. Likewise, users may discover new defects, which you should log and track in the defect management database.

Should business practices and considerations warrant it, propose a revised version of the application and initiate work. Repeat the entire development cycle, though most likely in a more condensed fashion, with the deployment baseline taken as the starting point. You can re-open previously deferred defects for repair, and consider recommendations or requirements for improvements in performance and functionality.

Feedback and Improvement

The final quality assurance activity for the development cycle is to examine the information collected (including metrics) and the actual experience and results of the development effort. If possible,  someone not directly involved with this project should conduct this process. Initial activities include one-on-one interviews with key project personnel, as well as distribution and collection of anonymous surveys among all participants.

From this information, distribute to all project personnel for commentary a draft of the project’s evaluation, along with lessons learned and suggestions for improvement, and hold a formal inspection held. Assign action items as appropriate to implement the recommendations, and the inspected evaluation should become part of the project’s baseline and stored appropriately for future examination.

About the Author: bfwebster

Webster is Principal and Founder at at Bruce F. Webster & Associates, as well as an Adjunct Professor for the BYU Computer Science Department. He works with organizations to help them with troubled or failed information technology (IT) projects. He has also worked in several dozen legal cases as a consultant and as a testifying expert, both in the United States and Japan. He can be reached at 303.502.4141 or at bwebster@bfwa.com.