At Lawrence Livermore, Gary Johnson's Computer Safety and Reliability Group--part of the Fission Energy and Systems Safety Program--has been working with the Nuclear Regulatory Commission for several years to avoid software problems in safety systems at nuclear power plants. Livermore brings to this job decades of systems engineering experience as well as a regulatory perspective from years of working with the NRC and other regulators.Johnson's group and the NRC developed software and computer system design guidance that the NRC uses to evaluate the design of safety-critical systems for U.S. plant retrofits. Overseas, where new nuclear power plants are being built, regulators and designers are using this state-of-the-art guidance to help assure plant safety. For the last few years, representatives from Hungary, the Czech Republic, Ukraine, Korea, Taiwan, and Japan have been calling upon Johnson and his group for assistance in setting criteria for their nuclear power plant control systems.This software design guidance is also applicable to other computer-controlled systems that could endanger human life if they are poorly designed--medical radiation machines, aircraft flight control systems, and railroad signals, for example.
When Software Fails
Perhaps the best-documented example of the harm resulting from poorly designed software involved the Therac-25, an accelerator used in medical radiation therapy. The IEEE Computer Applications in Power reported, "Between June 1985 and January 1987, six known accidents involved massive overdoses by the Therac-25--with resultant deaths and serious injuries" at treatment centers around the U.S. and in Canada.1 Between the patient and the Therac-25's radiation beam was a turntable that could position a window or an x-ray-mode target between the accelerator and patient, depending on which of two modes of operation was being used. If the window was positioned in the beam's path with the machine set to deliver radiation through the x-ray-mode target, disaster could result because software errors allowed the machine to operate in this configuration.
Engineering Reliable Software
Conditions such as these, where software was assigned sole responsibility for safety systems and where a single software error or software-engineering error could have catastrophic results, are precisely what Johnson's group aimed to avoid when it helped to prepare the portion of the NRC's recently published Standard Review Plan2 regarding computer-based safety systems. Their process requires that software for nuclear power plant protection systems be written in accordance with good engineering practices. That is, software should follow a step-by-step approach of planning, defining requirements for worst-case scenarios, designing the software, and following a detailed inspection and testing program--known as verification and validation--during each step of development and installation. This process avoids the pitfalls that can occur when software designers are expected to begin writing code before the design is complete, a process akin to pouring concrete footings for a building before knowing how tall the building will be.
Good engineering practices make for better software. But it is axiomatic in the safety-systems business that software can never be perfect. Figure 1, above, showing IBM's experience from the Space Shuttle program, illustrates the futility of attempting to produce perfect software for every need. The point of diminishing returns is reached well before perfection can be guaranteed.
When the consequences of failure are very high, safety-critical systems should always incorporate "defense-in-depth and diversity," which can be accomplished by incorporating different kinds of hardware, different design approaches, or different software programming languages. The idea is to have different kinds of systems available to accomplish the same goal; for example, a digital system is often backed up with a tried-and-true analog system. Then, if one system fails, a different system is in place to carry on. Simple redundancy, having two versions of the same system, is not enough because both could fail simultaneously as a result of the same flaw (Figure 2).
