U.S. ELECTION ASSISTANCE COMMISSION 1225 New York Ave. NW – Suite 1100 Washington, DC 20005
OFFICE OF THE CHAIRMAN

April 30, 2004

The Honorable Trent Lott
Chairman
Committee on Rules and Administration
United States Senate
305 Senate Russell Office Building
Washington, DC 20510

Dear Senator Lott:

I am pleased to submit the enclosed report to Congress on Improving the Usability and Accessibility of Voting Systems and Products. This document, also known as the Human Factors report, was produced in consultation with the National Institute of Standards and Technology (NIST) to meet the requirements of Section 243 of the Help America Vote Act of 2002 (HAVA).

Despite the fact that the U.S. Election Assistance Commission (EAC) was established in mid-December 2003, and therefore missed the October 2003 statutory deadline for this report, we were able to focus early on this task and the research that NIST had completed. The resulting report describes the findings of that study. It also presents a set of recommended actions that, if implemented, should improve the usability and accessibility of voting products and systems.

We would be pleased to meet with you at your convenience, to discuss our work and the recommendations contained in this report. I can be reached at (202) 566-3100.

:	Sincerely, DeForest B. Soaries, Jr. Chairman

Enclosure

Improving the Usability and Accessibility of Voting Systems and Products

:	Sharon J. Laskowski Marguerite Autry John Cugini William Killam James Yen US DEPARTMENT OF COMMERCE TECHNOLOGY ADMINISTRATION Phillip j. Bond, Under Secretary for Technology NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY Arden L. Bernent, Jr., Director April 2004

Disclaimer

Executive Summary

1. Develop voting system standards for usability that are performance-based, high-level (i.e., relatively independent of the technology), and specific (i.e., precise).
2. Specify the complete set of user-related functional requirements for voting products in the voting system standards.
3. Avoid low-level design specifications and very general specifications for usability. Only those product design requirements that have been validated as necessary to ensure usability should be included as “shall” statements in standards.
4. Build a foundation of applied research for voting systems and products to support the development of usability and accessibility standards.
5. To address the removal of barriers to accessibility, the requirements developed by the Access Board, the current VSS (Voting System Standards), and the draft IEEE (Institute of Electrical and Electronics Engineers) standards should be reviewed, tested, and tailored to voting systems and then considered for adoption as updated VSS standards. The feasibility of addressing both self-contained, closed products and open architecture products should also be considered.
6. Develop ballot design guidelines based on the most recent research and experience of the visual design communities, specifically for use by election officials and in ballot design software.
7. Develop a set of guidelines for facility and equipment layout; develop a set of design and usability testing guidelines for vendor- and state-supplied documentation and training materials.
8. Encourage vendors to incorporate a user-centered design approach into their product design and development cycles including formative (diagnostic) usability testing as part of product development.
9. Develop a uniform set of procedures for testing the conformance of voting products against the applicable accessibility requirements.
10. Develop a valid, reliable, repeatable, and reproducible process for usability conformance testing of voting products against the standards described in recommendation 1) with agreed upon usability pass/fail requirements.

Table of Contents

1. Introduction

After the introduction, we discuss our assumptions and the information we used to generate the recommendations. We describe the current status of and testing process for voting systems, present an overview of the concepts of usability and accessibility, and discuss some related standards. We then present a detailed discussion of approaches that can be applied to improve usability and accessibility based on our review of relevant standards, guidelines, and testing and evaluation methodologies. We conclude with the set of recommendations and a discussion of short and long term activities that can help achieve the recommendations.[1]

1.1 Scope

Our analysis addresses issues pertaining to both voting products and voting systems. A voting product, as defined here, refers to a product procured from a vendor such as a Direct Recording Electronic (DRE) terminal. [2] By voting system we mean the combination of physical environment, voting product, ballot, voter, and other persons involved in the voting process (e.g., poll workers and other election officials). The bulk of the discussion focuses on the usability and accessibility of voting products for the voter. However, we also include usability issues pertaining to ballot design, the influence of the environment on accessibility as well as usability, and the setup and operation of voting systems by poll workers and election administrators. Further, we have constructed our recommendations for improvements so that they will fit into the existing and future qualification and certification testing frameworks for voting systems. Note that we expect that these recommendations will be taken into consideration by the Technical Guidelines Development Committee (TGDC) when it becomes operational under the Election Assistance Commission (EAC) as described in the HAVA.

1.2 Background

It also appears that the problems of voting product usability and accessibility are not felt equally across the voter population. The U.S. Civil Rights Commission reported in (Voting Irregularities, 2001) that “Poorer counties, particularly those with large minority populations, were more likely to use voting systems with higher spoilage rates than more affluent counties with significant white populations.” Further, “Even in counties where the same voting technology was used, blacks were far more likely to have their votes rejected than whites.”

As a result of these and other reported voting irregularities, the U.S. Congress enacted the Help America Vote Act (HAVA) of 2002, Public Law 107-252. In the areas related to human factors, usability, and accessibility, the Election Assistance Commission is mandated to submit a report to Congress. Specifically, “…the Commission, in consultation with the Director of the National Institute of Standards and Technology, shall submit a report to Congress which assesses the areas of human factor research, including usability engineering and human-computer and human-machine interaction, which feasibly could be applied to voting products and systems design to ensure the usability and accuracy of voting products and systems, including methods to improve access for individuals with disabilities (including blindness) and individuals with limited proficiency in the English language and to reduce voter error and the number of spoiled ballots in elections.” This report was written to address this mandate.

1.3 Brief History of Standards and Testing for Voting Systems [3]

No Federal agency at that point had been assigned responsibility for testing voting equipment against the VSS. The National Association of State Election Directors (NASED) subsequently established a “certification” program through which equipment could be submitted by the vendors to an Independent Testing Authority (ITA) for system qualification. The ITAs are accredited by NASED to determine whether voting products are in compliance with the VSS. The results of the qualification tests can be used by States and local jurisdictions to help them assess system integrity, accuracy, and reliability, as part of their own certification testing. The VSS themselves were substantially updated and issued again in 2002, following a three-year development and public review process. This most recent update was accorded favorable review by the General Accounting Office in its preliminary audit (GAO, 2001). This release included functional requirements to improve accessibility by individuals with disabilities. An advisory section was included as guidance to improve user interface and ballot design. There were no specific qualification test criteria developed for this section; hence no formal conformance tests are associated with the guidance.[4]

2. Basic Terminology and Concepts

2.1 Definition of a “System”

In the usability field, the definition of system encompasses the users and all the elements required to accomplish some goal. A specific system is viewed as one (or more) users, attempting to accomplish some activities towards a goal or set of goals, within a specific environment. The activities include all interaction between the user and other parts of the system (the products, the environment, etc.) as well as activities they might do internally, such as decision-making. Various elements of the environment include: (1) the physical environment (lighting, temperature, and ambient noise), (2) the psychological environment (time or social pressure present in the environment), (3) all of the equipment used, and (4) any other users or support personnel involved. For voting, this means that, from a usability perspective, the voting system is defined by:

• The voters themselves
• The physical environment in which they vote (polling station or home for Internet-based voting)
• The psychological environment associated with voting (e.g., stress induced by long lines at polling stations, social pressures, time pressure associated with personal or other deadlines, etc.)
• The equipment, both hardware and software, used for voting (paper-based voting products, computer-based voting products, etc.)
• The ballot itself
• Quality of support provided (if required by the voter) by poll workers
• Any documentation and training provided (either to the voter or the poll workers and other election administrators) A change in any of these elements will redefine the system from a usability perspective. Most significantly from the usability perspective, the characteristics of the user population are significant in an evaluation of system usability. These characteristics include:
• Age and gender
• Background (educational, social, and cultural)
• Physical or mental capabilities
• Psychological factors such as current levels of, and susceptibility to changes in, stress, fatigue, mood, and motivation
• Prior experience with the subject matter
• Prior experience with the equipment to be used These are the “human factors” of the system. Usability is determined by the demands (both physical and psychological) that the other components of the system put on the users and the users’ ability to perform under these demands.

2.2 Definitions of Accessibility and Usability

2.2.2 Accessibility
Accessibility is defined as a measurable characteristic: the degree to which a system is available to and usable by individuals with disabilities. The most common disabilities include those associated with vision, hearing, and mobility, but the definition also includes cognitive disabilities. The HAVA also includes accessibility requirements for Native American and Alaska Native citizens and alternative language access for voters with limited English proficiency (LEP).

Accessibility standards are typically intended to specify designs that will maximize the access of the majority of persons with these types of disabilities, but does not necessarily guarantee access for a specific individual’s disability or combination of disabilities. An example of this approach to accessibility standards is the set of Section 508 Standards (Section 508 Standards, 2000) developed by the U.S. Access Board for Section 508 of the Rehabilitation Act of 1973, as amended in 1998. The U.S. Access Board is an independent Federal agency devoted to accessible design for people with disabilities. Section 508 is a set of accessibility requirements for Federal electronic and information technology. It applies to all Federal agencies when they procure, develop, use or maintain such technology. Accessibility as defined by the Access Board “is a term that describes products or services that meet the Access Board guidelines (in the case of the ADA) or the standards (in the case of 508). Something that is accessible – i.e., meets the guidelines or standards – is not always usable.” The Access Board also recognizes that products are accessible to individuals or groups of individuals. They are never “accessible” as an absolute unless every single person with any type, degree or combination of disabilities would be able to use the products. Products can meet accessibility standards or guidelines. These products are sometimes referred to as “accessible” in this more limited sense. However, these products may still be inaccessible to some people.

The ISO standard TS 16071 defines accessibility as the usability [italics added] of a product, service, environment or facility by people with the widest range of capabilities (ISO/TS 16071, 2003). For the purposes of this report, however, we intentionally make a distinction between accessibility and usability since, in general, meeting accessibility standards does not necessarily imply that a system is usable by a particular individual or even a group of individuals, but only that barriers to access have been removed. This distinction is particularly pertinent to this report and the recommendations for addressing these issues (including the means of testing and certification) are discussed in more detail in Section 6.

2.2.3 Usability and Usability Testing
Usability, for the purposes of this report, is a measure of the effectiveness, efficiency, and satisfaction achieved by a specified set of users performing specified tasks with a given product (ISO 9241-11, 1998). Effectiveness is the accuracy and completeness with which specified users can achieve specified goals in particular environments. Efficiency is defined as the resources expended by the user in relation to the accuracy and completeness of goals achieved. Satisfaction is defined as the subjective comfort and acceptability of the results to its users and other people affected by the results. These definitions have been formulated to provide the means for explicit measurements for usability. Usability testing is a method by which users of a product are asked to perform certain tasks in an effort to measure the product's usability using the metrics of effectiveness, efficiency, and satisfaction. In practice, usability testing is part of a larger set of approaches for evaluating usability, some of which involve users directly and other which do not. Testing is usually separated into formative (or diagnostic) testing and summative (or empirical) testing. Typically, formative (or diagnostic) testing is conducted as part of a product development process while summative (or empirical) testing is conducted after a product is completed.2.2.4 Self-Contained, Closed, Accessible Products
For some types of products, it is the responsibility of the user to provide accessibility-related software or an assistive technology device to make the product, sometimes called an “open architecture product”, accessible. In these cases, the designers are responsible for ensuring that their products are compatible with assistive technology. Examples include a sip-and-puff switch used by people who are quadriplegic, or a screen magnifier, screen reader software, braille display, or Opticon[6] used by people who have visual impairments. Voting stations in use at polling places are considered to be “self-contained, closed products” in that they are intended to be used without requiring the user to install specific accessibility-related software or the attachment of an assistive device.[7] This does not preclude the option for users to provide some device such as a mouth stick for pointing or their personal audio headset. It should be noted that some potential voting solutions being proposed such as telephone-based or Internet voting systems would not be designed as self-contained, closed products.

2.2.5 Accessibility versus Usability
Although the general definition of accessibility includes both availability and usability by people with disabilities, in this report we will treat accessibility as the degree to which a system is available to people with disabilities. Access alone does not guarantee usability. The usability of a product by people with disabilities will be considered a subset of the general concept of usability. This view facilitates the development of our recommendations.

2.2.6 Usability in Practice (an Example)
As an example of applying these definitions, consider the typical credit card scanner at the grocery checkout line. Grocery stores, in general, are not very accessible to some disabled populations, so it is not surprising to see that the credit card scanners are not accessible. The scanners are too high to reach and see from a wheel chair, and their combination user interface of push buttons and screen are not readable by people with visual impairments. One could imagine some design guidelines on placement and audio capability that would make the scanner accessible.

Even assuming that customers can access the scanner, there are some obvious usability issues, especially with first- and second-generation scanner designs. The first difficulty is determining how to insert the credit card. Often the diagram next to the slot is horizontal, but the customer must insert the card vertically and must figure out how to match the diagram. To ensure both efficiency and satisfaction, the cashier may take the card and insert it for the customer if the customer tries a few times and still doesn’t orient the card properly.

Next there are several options to choose from following instructions on a small screen. The wording and orientation of the instructions on the screen and buttons can be confusing. The dollar total is displayed along with a question of yes or no. But the displayed “yes” and “no” are often located at the top of the screen, quite near to a set of buttons just above the screen for credit or debit. As a result, even with the color coding used on the “yes” and “no” buttons it is not uncommon to see users hesitate or even hit the credit or debit buttons instead of the “yes” and “no” buttons at the bottom, below the screen.

Effectiveness for this product could be measured by different criteria including counting the number of errors (e.g., the number of incorrect insertions of the credit cards or the number of incorrect button presses) or by counting the number of help incidents (e.g., the number of times the cashier intervenes). Efficiency can also be measured several ways including timing the transaction or by counting the number of discrete steps. It can also be measured either only in the “happy path” (e.g., with no errors affecting effectiveness) or as an average over actual use (e.g., including errors affecting effectiveness). Finally, satisfaction can be measured by various means such as through a questionnaire that asks how the customers perceived the process and the outcome (e.g., did they feel frustrated or embarrassed by needing help or by holding up the line). Sufficient usability testing can be conducted to produce measures of the usability of such a product once suitable criteria are established. The resulting measurements can be used as benchmarks for testing new designs or comparing across products of equivalent capability.

2.3 Product Requirements, Usability, and Testing Methods

2.3.1 Type
The two basic types are performance requirements and design requirements (Hemenway, 1980). Performance requirements specify what functions and sub-functions a system is capable of supporting (i.e. the system is analyzed in terms of what operations can be performed), and design requirements specify how the mechanism is designed (physical components and sub components).

Performance requirements can be further subdivided into purely functional requirements and those specifying the degree of performance. As an example, suppose we wish to require that an automobile allow the driver to open the trunk from within the passenger compartment: A design requirement might specify that there be a trunk-release handle at least 4 inches long located no farther than 7 inches to the left of the driver’s left knee and that the driver must pull the handle towards the back of the car in order to operate it. A functional (performance) requirement might simply state that there must be a way for the driver to open the trunk without leaving his/her seat. A degree-of-performance requirement might add that the operation must be able to be performed in four seconds or less (on average) once a driver is shown how to operate the mechanism. Note that although this particular example involves human use, the performance/design distinction applies just as well to requirements for autonomous products. See the next section (2.3.2) for more discussion on the human interaction.

Generally speaking, design requirements are appropriate when the purpose of the requirement is interoperability, since this often requires an “exact fit” between system components. While there may be some variation in the concrete implementation of a design requirement (e.g. the color and exact shape of the trunk release handle), the thrust is to constrain the product.

Performance requirements are usually preferable when quality is the goal, since they directly describe the behavior of the system and not the supporting mechanism. Thus, performance requirements allow for innovative solutions, and they enable comparison of multiple competing designs since they are “technology agnostic”. Design requirements usually invite direct examination as the most suitable mode of testing. Examination may be as simple as observing the presence of some required part, or may involve very precise measurement and analysis (e.g. analyzing a file to see if it conforms to a format requirement; checking “legal HTML” for a web page is a form of examination).

Performance requirements, on the other hand, are more suited to testing by operation. Since the requirement describes how a system should behave, we operate the system and see whether it works as expected.

2.3.2 Human Interaction
The next distinction is between requirements that have direct and significant effects on interaction between artifacts and human beings, and those concerned with more or less autonomous objects or behavior. In the example above, we saw several different ways in which the ability (of a human) to open the trunk of a car could be mandated by a requirement. Even though these specifications were of different types (design, functional, degree-of-performance), they all had implications for how a human and machine would interact.

Conversely, requirements limiting automobile emissions or describing the format of a DVD do not have direct implications for the way in which humans use automobiles or DVD players.

Interaction requirements can be physical or psychological. Physical interaction requirements are related to such things as the product weight; size; the spacing of knobs, controls, and buttons; the force required to interact with knobs, controls, and buttons; the user’s reach envelope (what parts of the product the user can physically reach); and the user’s field of view (what parts of the product the user can visually “reach”). Psychological interaction requirements include the user’s understanding of the system’s displays, labels, and messages, the user’s understanding of the processes and procedures required to use the product, and the user’s ability to understand the outcome of the interaction (including awareness that goals were met).

Interaction requirements can be formulated in functional terms (e.g. users must be able carry a product) or as design specifications (e.g. the unit’s maximum weight and size). It is presumed that meeting the design specification will ensure the usability of the resultant product. However, design requirements are nearly always based on an “average” or “typical” range of users and do not necessarily apply to all individual users.

Conformance tests for requirements that do not involve human interactions are usually susceptible to automation, since only the intrinsic structure or behavior of some artifact is being tested. When a requirement does involve human interaction, the way in which it is to be tested depends on its type, as defined in Section 2.3.1.

2.3.2.1 Testing Design Requirements for Interactive Products
A test for a design specification (such as the configuration of a trunk release lever) can be done by examination, because even though we expect the mechanism to be used by a human, the requirement has, for better or worse, dictated the design of the “machine” side of the interaction. As long as the mechanism follows the design, it conforms to the requirement, whether or not the design itself is well-suited for the general purpose.

2.3.2.2 Testing Functional Requirements for Interactive Products
A test for a simple “functional capability” specification would normally involve operation by an expert, presumably by following the instructions for the product in question. The expert would check for the presence and general workability of a mechanism supporting the capability. For example, if the examiner can indeed open the trunk from the driver’s seat, the test is passed. As with design specifications, passing the test is no guarantee of real usability. A product could actually include a mechanism to accomplish some task, and yet have poor usability if the mechanism is hard to understand or difficult to operate. See Section 3.2.4 for an extended example of how usability can be affected by various designs for the same function on a voting product.

2.3.2.3 Testing Performance Requirements for Interactive Products
Finally, a direct test of a degree-of-performance specification would normally involve some use of human subjects. For example, if the requirement specifies how long it takes a certain class of drivers to open the trunk after being instructed, the only real way to tell if a given automobile conforms is to have some users try it out. Of course, much hinges on the exact wording of the requirement (possible metrics include: the time users are expected to take to perform the activities on the system, the throughput of the system such as the number of forms that are processed in a given amount of time, the number of acceptable errors a user can perform in typical interaction) and how precise we wish the test to be. A casual test might substitute expert judgment for actual experimentation. Nonetheless, if the requirement mandates a certain degree of success by a designated class of users, a usability test of some sort is the most direct and reliable way of measuring conformance.

A special case of a performance specification would be to mandate a certain degree of user satisfaction with the use of the product. Within the usability and human factors engineering community, user satisfaction is often considered a requirement for a product, but it is rarely stated specifically. However, this trend is changing. Because satisfaction is one dimension of the system's usability that is purely subjective, the requirements for satisfaction are difficult, but not impossible, to include in product testing. Testing might involve questionnaires or interviews with users to determine their subjective reaction to the use of the product. Note that a number of validated subjective satisfaction questionnaires exist, such as SUMI (TUhttp://sumi.ucc.ie/UT), SUS (http://www.cee.hw.ac.uk/~ph/sus.html), and QUIS (http://www.cs.umd.edu/hcil/quis/).

2.3.3 Levels
The various specifications within a requirement can address the entire system in question (high-level) or major components or functions of the system (mid-level) or small components and functions (low-level). These are not precise distinctions. For example, the acceleration of an automobile is a high-level function; the ability to open the trunk from the engine compartment is a mid-to-low level function. Low-level requirements are often lengthy and embody a detailed description of the object in question. Design requirements tend to be low-level although performance requirements can also be low-level. Higher-level requirements embody more abstract requirements; the basic functions or mechanisms are specified without dictating the details of how this is accomplished. Note, in particular, that high-level performance requirements directly address the broad “bottom-line” requirements of a system, without constraining the means by which these are achieved.

The testing implications are straightforward: numerous low-level specifications typically require numerous tests (although each test is likely to be small and simple). Higher-level specifications might require somewhat more complex and holistic tests, but there are likely to be fewer of them.

2.3.4 Specificity
Requirements can be couched in terms that range from the very specific to the general. For example, at one extreme, a requirement might simply specify in general that an automobile “provide good visibility” for the driver; at the other, the requirement might specifically mandate the precise angular height and width of an unobstructed view for any driver between a minimum and maximum height.

Specific requirements are not the only mechanism by which quality can be encouraged. In cases where it is impossible to state precise requirements, general specifications can provide useful guidance. Indeed, some usability “requirements” (e.g., use legible fonts) are in fact simply checklists of general design considerations to be taken into account by developers.

Specificity is important when it is necessary to test objectively whether a given system conforms to a requirement. General specifications lead to tests that are either subjective (e.g. an expert uses his/her judgment to decide whether the visibility is “good”) or somewhat arbitrary (the test procedure, rather than the requirement, adopts a more precise definition of what constitutes good visibility). Specific requirements support test procedures that are both more objective and more directly justified by the text of the requirement.

2.4 Standards and Conformance Testing
Standards are a ubiquitous part of the “invisible infrastructure” that helps to assure that functions such as commerce, transportation, and communication take place smoothly and integrate appropriately. Standards can be formulated and applied in various ways. The following is a brief overview of the basic concepts of standardization.

2.4.1 Terminology of Standards
A standard usually has one of three basic purposes:

• To provide a metric for the accurate measurement of some property, such as the unit for mass (maintained by NIST), namely the kilogram. The standard is used for comparison so that all measures of a given unit are equivalent.
• To assure the interoperation of manufactured components of a system, such as the format for a compact disk to be read by a CD player.
• To establish a level of quality to be met by a product, such as emission standards for automobiles. In this report, our main concerns are the standards of quality for usability and accessibility in voting and interoperability standards for some accessibility issues. Standards for these latter two purposes are really just examples of detailed, officially formulated product requirements, as described above in Section 2.3. (Standards for metrics are a different case and not of concern in this report.) As such, all the remarks above concerning the properties of requirements (type, level, etc.) and the implications for testing apply directly to interoperability standards and quality standards.

2.4.2 Pragmatic Issues for the Application of Standards
Once a standard is approved, three other issues emerge:

• How to resolve disputes about the meaning of the standard: the interpretation problem,
• How to tell whether a given entity conforms to its requirements: the testing problem, and
• What happens if a product doesn’t conform: the enforcement problem.

2.4.2.1 Interpretation
As with any written requirement, standards can become the subject of dispute, especially when the conformance of a product depends on the precise interpretation of the standard’s wording. This is a good argument for writing clear and specific requirements in the first place. Nonetheless, disputes do arise, and some authoritative body must decide the issue. This “judicial” function has been carried out in various ways. Usually, the body that developed the standard also takes the responsibility for providing interpretations, but sometimes a third party will be given this task.

2.4.2.2 Conformance Testing
If a standard is to be something more than a mere document, there must be some procedure for applying the standard to the entities within its scope. This procedure must be designed and written (test development) and then enacted (test operation). The development of a good test suite can often involve more effort than the formulation of the standard itself. As we have seen, the test methodology depends strongly on all four properties (type, human interaction, level, specificity) of the standard.

Note that there are many types of testing that do not deal directly with conformance – examples include:

• Exploratory testing in the early design stage of development (usually called formative testing in the usability field),
• Debugging (diagnostic testing for defects), and
• Comparative testing of competing products. In particular, note that although conformance tests may often have some diagnostic value, their main purpose is to detect aspects of the system that do and do not meet the requirements of the standard, not to find the cause of the failure.

Finally, there is the issue of test operation. Conformance test suites are sometimes executed by the vendor (self-testing), or by a potential purchaser. It has also become common practice for a third party, such as an accredited laboratory to perform the testing. As mentioned earlier, such third-party testers are referred to as Independent Testing Authorities (ITAs).

2.4.2.3 Enforcement
There are two points of decision for enforcement, the first internal to the standard, the second external.

In the first case, the standard itself may make distinctions between its binding specifications (often denoted by saying that something “shall” be the case) and non-binding specifications (denoted by “should”).

Second, the standard as a whole may be enforced in several ways:

• It may be enforced directly by the Government,
• It may be enforced by potential buyers who refuse to purchase non-conforming products,
• There may be a “labeling” policy, such that a product cannot claim to be of a certain type, unless it meets the standard, or
• It may be completely voluntary. The enforcement approach taken is a policy issue, and normally has no direct technical implications.

3. Usability and Accessibility Requirements of Voting Systems

Interaction requirements can also be identified for voting products some of which exist in current or draft standards. These include specification of the typical reach envelope, minimum font size, and other specific design details. As with all design requirements, there is some question as to the effect these requirements actually have on the usability of the product.

User performance requirements for voting products also exist. These are generally not enforced and appear to be provided only as guidelines. In one state, for example, there is a requirement that the act of voting by an individual voter take no more than five minutes. However, there does not appear to be any currently defined requirements related to user error rates, though there appears to be an implied requirement related to DRE systems that sets the error rate for overvoting to zero. However, there appears to be no standard for the number of anticipated user errors, or the number of calls for assistance that would be considered acceptable when dealing with a large user population such as is the case with voting. Such errors might include the number of times a voter inadvertently attempts to overvote, unintentionally undervotes a ballot, or is unsure of the next step in a process, whether these conditions are corrected or not before the vote is cast. User satisfaction requirements do not appear to be defined for voting products though they have been the subject of many articles on voting.

The question that remains to be answered is whether or not existing standards are necessary and/or sufficient to ensure a high degree of usability for voting products. Finally, it is important to note that there is considerable variation in the implementation and design of voting products, which makes it a challenging task to create standards that are testable, span this range of design, and ensure some level of good usability and accessibility.

3.1 Implementation Examples of Functional Requirements for Voting

3.1.1 Implementation Variations
As described above, the specific implementation developed by a vendor defines the interaction characteristics of the product. And, it is these interaction characteristics that determine the usability and accessibility of a product. Multiple implementations are nearly always possible for a set of functional requirements and the implementation variations are often based on variations in the medium or technology used. In the case of voting products, examples are the selection of paper versus electronic, touch screen versus selection wheel, and QWERTY versus alphabetic keyboards for write-ins. The choice of technology is up to the designer, though much of it may be limited by technological capabilities, cost, or some other factor. The choice of the interface design, including many elements of the presentation layout and screen flow is also up to the designer, though it is constrained somewhat by standards, conventions, and industry best practices. Assuming all functional requirements are supported, variations in the specific implementations will cause different interaction challenges.

Since multiple designs may be in use across the country, across the state, or even across a district, the usability of the products will vary. Further, within the U.S., vendors are free to create unique voting products. This is due, in part, to our culture of both voluntary standards and free competition. Contrast this with the approach taken by the Brazilian government, which contracted with two research companies to design a single product for voting (Caltech–MIT, 2001). Separate contracts were made with a number of companies to manufacture the product to the design specifications. In this situation, provided the product is both accessible and usable, all voters will experience the same system and therefore the same level of usability and accessibility will be seen across the entire country.

Although this single design approach is a possibility for the U.S, the variations in State requirements and the nature of the relationship between the Federal government and State governments make it a highly unlikely solution. Nevertheless, it is still necessary to ensure that all designs from all vendors achieve a minimum level of usability and accessibility. The current VSS approach assumes that appropriate standards can be put in place to ensure the usability and accessibility of voting products. However, design standards can ensure a specific level of usability and accessibility only if they completely specify the interface design. This can restrict both the incorporation of new advances in technology as well as creativity on the part of the designers to develop novel solutions. Alternatively, the usability and accessibility of each product can be independently determined and compared to a fixed standard for these aspects of the product design. It is for this reason that this report focuses on performance-based standards for both usability and accessibility and minimizes the dependence on design standards.

3.1.2 Example of Voting Product Design Variations
One functional requirement from the HAVA requires that voting systems allow the voters to change their votes in any of the contests before casting their ballots. In a review of the existing voting products at the 2003 International Association of Clerks, Recorders, Election Officials and Treasurers (IACREOT) Trade Show in Denver, we observed at least three different implementations of this requirement. All three were touch screen-based, DRE products and all three technically met the current VSS standards. Yet significant variations in the designs and implementations existed. Note that the variations themselves do not necessarily indicate a problem with existing standards or the DREs themselves, but do indicate the importance of measuring usability. Further, although we have focused on DREs, similar issues can be identified for all systems. (For example, paper ballots do not prevent overvoting and lever machines, in some cases, had labels too high for small individuals.)

In this section, we will describe these designs in terms of their interaction characteristics[8] in a winner-takes-all type election and discuss the resulting usability issues.

3.1.2.1 Product A – No Change Feedback
In one design, the voter selects a name from a list of candidates by pressing on the name on the screen through the system’s touch screen interface. If he touches a different name within the same contest, the first choice is changed to the new selection. There are no messages (auditory or visual) associated with this change action beyond the highlight of the new choice.

3.1.2.2 Product B – Yes/No on Change
In a second design, the voter selects a name using the same touch screen approach but, if the voter presses on the second name, a message is displayed asking her if she is attempting to change her vote. The voter can select “yes”, in which case the system removes the message from the screen, removes the mark from the first candidate, and marks the second candidate as the selected choice. If the voter selects “no”, the system removes the message from the screen, but no other action is taken.

3.1.2.3 Product C – Deselect/Select to Change
In a third design, the voter makes a selection for the first candidate as he would do with the other two designs but, if he presses on the second name, nothing happens. There is no change in his vote and no message displayed. In this design, the voter must reselect the first candidate again to remove the selection mark before he can select a new name.

3.1.2.4 Interpretation from a Usability Perspective
These three designs represent different approaches to satisfying the same functional requirement using basically equivalent technology (e.g., a touch screen-based, DRE interface). Other technologies are possible as well. For example, another voting product also reviewed at the trade show had the same interaction characteristics as Product C but used a non-touch screen interface for selection. In this case, the navigation and selection was accomplished through a rotating wheel and a selection button and no other means of navigation and selection were provided.

Product A appears to be the simplest to use since it contains the fewest number of steps. User action is responded to directly by the system. However, this design includes the possibility of the voter inadvertently changing his vote and not detecting this. If, while moving his hand across the screen, the voter accidentally touches the name of an alternate candidate that candidate will be selected. If the voter fails to notice this change, and continues the voting process, he won’t necessarily notice this error during the review. If he does notice this error, he must return to the selection and change his vote. Even if the voter is able to perform this pass without difficulty (he is able to determine how to return to the contest and make the correction), it will increase the time on task for this voter. The potential also exists for the voter to fail to notice this change, even during the review process, and cast his votes with the inadvertent error.

Product B has a specific feature apparently designed to prevent this very error. The voter must specifically acknowledge the change in vote before it takes effect.

Product C appears also to prevent inadvertent selection of an alternate vote, but does so in a fashion that requires the voter to determine why the system failed to respond. He must determine that the vote has to be removed before the intended vote can be cast. There is some question as to whether or not the voter would realize this by himself. Some voters might have no difficulty in making this determination. Others might ask for help to understand how to make this change. Some voters might perceive (incorrectly) that the system does not allow them to make the change once a candidate selection is shown on the screen. All three designs support the functional requirement but provide separate usability challenges for the voters in terms of what they must physically do and mentally understand. As a result, the error rates for each of these designs (in terms of voter confusion, calls for help, error rate, time on task, or voter acceptance) would likely vary, though all three errors rates may very well be within acceptable limits. The actual error rate of these designs cannot be determined without adequate testing with actual voters and a determination of what are “acceptable” limits.

3.2 Potential Usability Problems in Voting Products

There are a number of factors that determine the nature and frequency of usability problems encountered with any product. Users must be able to (1) deduce the interaction required (or be trained and able to accurately recall the interaction) and (2) be physically and mentally able to perform the interaction. If users cannot achieve (1) or (2), they will not be able to use the system. However, since we are talking about human involvement, perfect usability is rarely if ever realized. Instead, usability varies between perfect success (a perfect match between designer expectation and user interaction and accomplishment of the goal within an allotted or acceptable period of time) and total failure (the inability to reach the goal or to reach the goal accurately within an allotted or acceptable period of time).

Hence, there are three classes of usability problems:

• Usability problems prior to success
• Usability problems prior to partial failure
• Usability problems leading to total failure

3.2.1 Usability Problems Prior to Success
Usability problems can affect voters during the process but not affect their ability to accomplish the goal of casting a valid vote as intended. These problems will result in changes in the overall task time. Problems might also show up as voters try to perform inappropriate actions such as attempting to move or scroll beyond the limits of a page, accidentally changing votes, attempting to overvote a contest, or unintentionally undervoting an election. Provided the voter corrects each of these errors before completing the goal of casting a ballot, the results can still be considered successful, even though it took additional time or used more cognitive or physical effort than it would have if the voter’s actions had taken place with an alternate design. (Note that if large numbers of voters take extra time, this could result in longer waiting lines at the polling place discouraging some voters from staying to vote.) In addition, the problems experienced in attempting the task of voting may be severe enough to require assistance (either on line or live). Finally, they can result in changes in the user’s level of satisfaction.[9] When dealing with a large number of voters, some will inevitably struggle mentally or physically with the interface before determining the correct interaction required but will ultimately succeed. Others will make and correct one or more process errors before succeeding.

In a voting product, these usability problems might manifest themselves by increases in the time it takes one user to vote. These changes can be detected only by measuring the actual time required to vote or by directly observing voter behaviors (e.g., physical hesitation while voting). From an individual perspective, the task is still completed so the issues might not be serious enough to address, except where problems affect the user’s subjective ratings including confidence in the final result. Although individual performance might not be sufficiently affected to warrant concern over the design, additional delays in lines and added frustration on the part of those waiting can be severe enough to affect the overall system performance (i.e., the collection of all voters in a given polling location). Voters waiting longer in line may perform worse than those that have to wait less (potentially leading to more frequent or more severe usability problems) or might even leave before voting.

3.2.2 Usability Problems Leading to Partial Failure
More critical usability problems can result in the user being able to accomplish only some of the tasks associated with a goal or exceeding acceptable time limits. These usability problems might also be manifested in changes in individual user behavior, time on task, and user satisfaction, but they would also be show up as changes in the quality of the final product (e.g., the ballots may show more undervotes, null voters within specific elections, or rolloff voting behaviors). Since the tasks that are accomplished are correct, the final result of these problems would be classified as usability problems prior to partial failure. However, it should be noted that the presence of undervotes, null votes within specific elections, or rolloff voting behaviors cannot be assumed to be the results of usability problems. The voter may have intentionally decided to perform these actions.

More disturbing than the presence of problems leading to partial failure, is the fact that voters might not even be aware of the existence of the problems. For example, a voter may unintentionally cast a ballot that shows signs of rolloff voting behavior believing that they voted all levels of an election when, in fact, they did not. Note also that usability problems may result in added pressure to complete the ballot, thus resulting in a conscious decision not to vote in some races even though this was not the original intention (a usability problem leading to partial failure). This would also be classified as a usability problem leading to partial failure, but the voter would not necessarily view this as a failure. (This is one reason why exit polling and post test surveys can lead to false impressions about the nature and extent of usability problems in a product.)

3.2.3 Usability Problems Leading to Total Failure
A usability problem that resulted in the total inability to perform the tasks or to perform the task within an allotted or acceptable period of time would be considered a usability problem leading to total failure. In a voting product, one presumes this is a rare case since total failure appears to be a detectable event and the voter would likely be assisted by a poll worker to complete the voting process. However, such usability problems may not actually be that rare, just rarely observed, for they can be manifested by a voter prematurely casting a ballot (which was reported in Maryland) or the voter leaving the voting booth without casting the ballot (which was reported in New York). Voters can also become so frustrated that they abandon the voting process without completing a ballot.

3.2.4 Examples of Potential Usability Problems
Using the examples of the three touch screen-based, DRE Products A, B, and C described earlier, we can look at the variations in the designs and speculate about the usability problems (which represent a potential for error), and even predict relative rates (though it is not possible to predict actual rates without a controlled study of actual voters). We assume for the purpose of these illustrations that the voters are physically able to interact with a touch screen. (In general, this should not be the only way that people with disabilities can interact with the product.)

This analysis is based on the physical and psychological interaction required. However, other factors can also affect the nature and probability of error. Variations in the visual display or spacing could also change how errors occur for any of these designs. The sensitivity or internal technology of the different touch screen devices could also change the error profile. Without actual usability testing, we cannot know if any of these designs or potential errors described would cause usability problems prior to success, usability problems leading to partial failure or a false sense of success, or even usability problems leading to failure.

3.2.4.1 Example 1 – Changing a Vote
Using Product A from Section 3.1.2.1 described above, there is a potential for the voter to inadvertently touch the screen and change her vote while moving her hand across the screen. If she detected and corrected the mistake, this would represent a “usability problem prior to success.”

There is also a possibility that this event would not be detected by the voter at the time it occurs and therefore, the error would not be corrected at that point.[10] If the event is not detected at the time it occurred, then she would have another opportunity to detect the event during the ballot review (as mandated by HAVA) before casting her vote. However, there is still a possibility that she might not detect the event and cast her ballot with the unintended error. This illustrates a usability problem leading to a false sense of success.

Product B has an additional design feature that appears to specifically preclude the inadvertent selection error and thus would likely have fewer incidents of this error going undetected (since the inadvertent contact with the screen results in a message). However, it introduces something new that the voter must understand and interact with correctly. There is the possibility that he might select “yes” instead of “no” or vice versa, which could be influenced by the arrangement of the buttons, their prior experience with similar messages on computer systems, or the wording of the message.[11] [12] Further, such an error message should be “modal”, that is, it does not allow the voter to interact with any other part of the system until he completes the interaction with this message. If the error message is not “modal”, there is a possibility that the message might accidentally become hidden from view and leave the system in an indeterminate state without actually casting a vote. It is unclear in this specific case if the product would allow the user to cast a ballot with a non-modal message open. If it did, then this illustrates a usability problem leading to total failure.

Product C, where the voter must reselect the first candidate’s name to remove the selection mark before selecting a new name, represents a design that has a lower or even zero probability that an inadvertent vote change will occur since it would require inadvertently touching the screen twice – one to remove the existing vote and once to inadvertently select the new vote. However, this same design must support the voter’s attempt to change her vote. This design appears to be the most difficult for voters to understand since it lacks any feedback or guidance telling a voter how to change her vote. Thus, if she selects a new candidate without de-selecting the first candidate’s name (assuming that this is the correct action required), there is no feedback during the event to alert her to a problem. Rather than assuming her action was inappropriate to accomplish her vote change, she might assume that this design does not allow her to make a new selection once one is made. This would make it a usability problem resulting in partial failure. Even if the voters assume the system should allow this type of change (or were told that it did), they might struggle with this design or seek help from a poll worker.

3.2.4.2 Example 2 – Voting a Multi-Seat Contest
All three of the example products support the two interaction requirements that the voter be able to select from multiple names within a multi-seat contest and change his vote before casting his ballot. In all three cases, he selects candidates using the same touch screen selection method that is used for selecting a candidate in a single-seat contest. The visual display of each of these designs is consistent across the three products as well as between the single-seat and multi-seat contest. An un-selected candidate is represented as a closed box and a selected candidate is represented as a closed box with a check mark.

In other words, two different types of contests are represented by the same visual design. Readers familiar with computer programs with graphical user interfaces will note that these different types of behaviors (single selection from a group and multiple selections from a group) are generally represented by two different visual elements – a round circle (called a radio button) for single selections from a group and a square for multiple selections from a group (called a check box). These different visual designs are intended to aid the user in visually identifying the capabilities (and their inherent interaction requirements) associated with the type of each group. The design of the example products can appear to voters familiar with computers as internally inconsistent (or even as “coded wrong”) and thus might represent a usability problem prior to success. There is also a chance that a voter could mistake the box shape as a computer type check box and attempt to overvote a single-seat election. With the new DRE products this would result in a usability problem prior to success since nearly all of them preclude overvoting. Finally, some users may mistake a multi-seat contest as a single-seat contest and inadvertently undervote the contest – a usability problem leading to partial failure.[13]

3.3 Potential Accessibility Problems in Voting Products

To satisfy the goal of accessibility, barriers to access by people with disabilities and language difficulties must be removed or an alternate means of access provided. These barriers are often represented by physical barriers such as the inability to enter a building, to reach controls or read displays from a seated position, to interact with controls that require visual feedback (e.g., touch screens) or to use a mouse or a touch screen due to lack of fine motor control. Difficulty in communication can also be a barrier to access. Products that provide information via audible feedback exclusively may be difficult or impossible to use by persons who are deaf or have hearing loss. The issue is present not only in the primary display but also in the feedback used to indicate progress or selection. Many products provide auditory feedback for the user to indicate the end of a page or the last page in a multi-page form, which is fine if the feedback is redundant and also available as visual information. Touch screen products often use auditory feedback to aid the user in knowing that a selection has been made (though this is nearly always redundant with visual feedback). Visual displays cannot be accessed by many users who have visual impairments. Again, a touch screen product, even if not used for data display, relies heavily on visual feedback for proper operation. Once the barriers to access are removed by adding redundancy, a second condition must be satisfied – the product must be usable by these populations.

There is some interaction between usability and accessibility, since the means of providing access represents interaction challenges for the user. Some environmental factors may have changed from those used by non-disabled users. The product may have a different keyboard or entry device for disabled users. In the case of touch screen-based, DRE products for example, an alternate set of keys typically are provided for movement and selection. There might also be differences in the medium used for data display and feedback (e.g., audio instead of video, text instead of audio, alternate entry device instead of a touch screen). Interestingly, specific accessibility features, if used by non-disabled users, may reduce some usability problems. The usability problems noted in the DRE interface design that allowed users to change votes with only a visual indication of the event risks inadvertent activation. However, a user who is blind, using an audio interface, is provided with the name of the new selection even if the selection was inadvertent. This would increase the probability of detection of the event. However, accessible interfaces are often provided as an alternative and are not integrated. Sighted users might not have access to the audio when they are using the touch screen interface. (Note that voters with vision or cognitive problems can benefit from the audio together with a touch screen to confirm that they are reading and interpreting the screens correctly.) In addition, there are special issues of usability for disabled users. The design of the ballot, the length of the ballot, the number of candidates, and the number of races might be obvious to a sighted user, but not to a voter who is blind or visually impaired unless a feature is included that provides this information.

Though access may be provided, additional requirements for product usability by people with disabilities exist. For both touch screen and non-touch screen-based DRE products reviewed for this report, audio is the primary alternate medium provided for users who are blind or visually impaired. There are many good reasons for this decision on the part of vendors, but problems remain. Audio may be provided as recorded speech or synthetic speech, each with its own benefits and disadvantages. Audio feedback takes longer than reading unless the user can and is able to understand audio playback at high speed. In any case, audio data is transient, so users who are blind or visually impaired rely on short term memory to a larger extent and for more data then non-disabled users. Browsing an audio display is significantly harder and more time consuming than browsing visual displays. Some voters who are deaf might take a longer time to vote. Deaf individuals, particularly those who are congenitally deaf, read at a lower reading level than non-disabled users.[14] Data entry via an alternate input device may be more difficult, take more steps, or have other differences than the primary input.

At a minimum, a fully accessible user interface is anticipated to have an average longer time on task for a person interacting with an audio-based interface than the time on task using a visual display. For standardized tests, it is presumed that there is a 50% increase in task time. However, this estimate is based on completing a standardized test, at a desk, using a familiar alternative interface. Personal correspondences by the authors of this report with individuals with visual disabilities place the estimate on the order of 3 to 4 times longer for some users who are blind. Furthermore, the nature and frequency of usability problems encountered are almost certain to be different.

4. Current Usability and Accessibility Related Standards

4.1 Current (and Proposed) Voting Systems Standards related to Human Factors, Usability, and Accessibility

• Help America Vote Act (HAVA)
• Federal Election Commission’s (FEC) Voting System Standards (VSS)
• Institute of Electrical and Electronic Engineers (IEEE).

4.1.1 Requirements of HAVA
Title III of HAVA directly imposes certain requirements by 2006 on voting systems used for elections for Federal offices. Those related to usability include:

• Voter verification of the ballot, in private, before final submission
• Voter opportunity to correct a ballot in private before final submission
• Notification of a potential overvote before casting and allowance for correction in private
• Accessibility for voters with disabilities
• Availability of alternative languages for data presentation
• Public availability of certain voting information, including a sample ballot, and instructions on how to cast a vote

These represent both high-level and mid-level functional requirements.

4.1.2 Current FEC Process: the VSS
The voluntary Voting System Standards (VSS) issued by the Federal Election Commission (FEC) have been in effect since 1990 and were last updated on April 30, 2002. According to HAVA, they continue to be the official Federal standards for voting until superseded by guidelines issued by the newly created Election Assistance Commission (EAC). The VSS is a lengthy document comprising an overview and two volumes, the first containing the standards themselves and the second covering test methods. The current voting system standards contain specifications that are functional, for the most part, but also include some design specifications.

Significantly, the VSS define a voting system as the devices that allow users to vote. Voters are not considered part of the system. This is in contrast to the definition provided in this report. There is a consequent emphasis on mechanical and electronic performance of the device. Usability is presently covered only in an advisory Appendix, although there are plans to add it as an official specification. Also, no coverage is included for mail-in or absentee balloting or for Internet voting. Both of these areas may present significant issues in the area of security and the potential for fraud or misuse, but the usability aspects of these areas could be addressed independently. Conversely, there is considerable attention given to telecommunications, again demonstrating the present emphasis on the technical aspects of voting.

In general, the VSS document does not clearly define human factors, usability, accessibility, or the associated conformance testing that should be applied to these areas.

In the following sections, we describe the VSS sections on Human Factors, Usability, and Accessibility in more detail.

4.1.2.1 VSS: Human Factors and Usability
The VSS notes that human factors issues are covered mainly in Appendix C to Volume I. ITAs are given "wide latitude to develop [emphasis added] and perform appropriate tests". Human error rates are acknowledged, but are mentioned only in relationship to system error rates as follows:

"…the term 'error rate' applies to errors introduced by the system and not by a voter's action, such as the failure to mark a ballot in accordance with instructions. ... Further research on human interface and usability issues is needed to enable the development of Standards for error rates that account for human error" (page 5).

In the following subsections, we identify features of the VSS that are pertinent to our discussion of how to develop and test usability and accessibility standards. Note that this discussion is by no means a thorough analysis of the VSS and is not intended to diminish the tremendous and valuable efforts of the FEC and NASED over the past 25 years to develop these standards.

4.1.2.2 Pertinent Features of VSS: Volume I, Performance Standards
In Volume I of the VSS a broad range of levels and specificity is represented in the standards (see 2.3 of this report). Some specifications are quite precise; others very general. Some specifications are low-level; others are very high-level performance and functional requirements. Some of the standards are technology-specific, and others are generic. Generally speaking, the standards have a "bottom-up" empirical feel to them--as if they were composed in response to various technical issues and problems as they arose. Some sections read more like articles on good practice than true standards in a technical sense. As we will see below, there are some ISO standards that are also essentially advice and checklists.

VSS Section 1.1 emphasizes a non-process oriented approach taken within the standards:

"For the most part, the Standards address what a voting system should reliably do, not how system components should be configured to meet these requirements. It is not the intent of the Standards to impede the design and development of new, innovative equipment by vendors."

This is broadly true since most of the specifications are functional requirements in the form, "The system must be able to do X".

VSS Section 1.5.1 contains the definition of a voting system:

"A voting system is a combination of mechanical, electromechanical, or electronic equipment. It includes the software required to program, control, and support the equipment that is used to define ballots; to cast and count votes; to report and/or display election results; and to maintain and produce all audit trail information. A voting system may also include the transmission of results over telecommunication networks." VSS Section 1.6: clarifies the types of testing used for voting systems:

“...voting systems are subject to the following three testing phases prior to being purchased or leased”

• Qualification tests [performed by the ITA],
• State certification tests, and
• State and/or local acceptance tests.”

VSS Section 2 is central to understanding the Voting System Standards. Here the general functionality expected of a voting system is defined. Even though most of it does not directly address usability, it does convey the general approach of the VSS to standardization. VSS Section 2.1 commits to functional-style standards: "This section sets out precisely what it is that a voting system is required to do." Indeed, most of the requirements are functional, but there are a few low-level design specifications as well.

Accessibility is covered in VSS Section 2.2.7. Here some very specific design standards are given. For example: "Where any operable control is 10 inches or less behind the reference plane, [the system shall] have a height that is between 15 inches and 54 inches above the floor." This section has a wide variety of types of standards, ranging from broad functional statements to narrow and technology-specific design requirements, many of which can be subject to broad interpretation. For example, in the VSS:

• Section 2.3.1 Ballot Preparation states: "Ensuring that vote response fields, selection buttons, or switches properly align with the specific candidate names and/or issues printed on the ballot display".
• Section 2.4.3.1 notes that all systems shall: “...provide text that is at least 3 millimeters high and provide the capability to adjust or magnify the text to an apparent size of 6.3 millimeters."
• Section 2.4.3.2.1 states: "All paper-based systems shall… ...allow the voter to easily identify the voting field that is associated with each candidate or ballot measure response."
• Section 2.4.3.2.2 states that all “paper-based precinct count systems shall… ...provide feedback to the voter that identifies specific contests or ballot issues for which an overvote or undervote is detected;"
• Section 2.4.3.3 (DRE System Standards) states the system shall "enable the voter to easily identify the selection button or switch, or the active area of the ballot display that is associated with each candidate or ballot measure response;" This section also requires feedback for over- and undervoting and the ability to delete or change choices. And the system must "…Provide sufficient computational performance to provide responses back to each voter entry in no more than three seconds;"

Beyond VSS Section 2, a few more usability related issues are mentioned in various places:

Section 3.2.4.1 states that "all systems” shall provide “...privacy for the voter, and be designed in such a way as to prevent observation of the ballot by any person other than the voter".

Section 3.2.4.2.2 states that: "punching devices” shall “...facilitate the clear and accurate recording of each vote intended by the voter".

In contrast to the surrounding material, there is a short subsection, VSS Section 3.4.9, on Human Engineering – Controls and Display that contains explicit functional and design requirements for usability. It begins:

“All voting systems and components shall be designed and constructed so as to simplify and facilitate the functions required, and to eliminate the likelihood of erroneous stimuli and responses on the part of the voter or operator.” [Emphasis added.]

This section goes on to state that: “Appendix C provides additional advisory guidance on the application of human engineering principles to the interface between the voter and the voting system.”

VSS Section 9 on Qualification Testing (a term equivalent to conformance testing as it is described in Section 2.4 of this report) describes a general approach; however, no criteria or procedures for usability and accessibility testing are specified:

"Qualification testing encompasses the examination of software; tests of hardware under conditions simulating the intended storage, operation, transportation, and maintenance environments; the inspection and evaluation of system documentation; and operational tests to validate system performance and function under normal and abnormal conditions. The testing also evaluates the completeness of the vendor's developmental test program, including the sufficiency of vendor tests conducted to demonstrate compliance with stated system design and performance specifications, and the vendor's documented quality assurance and configuration management practices. The tests address individual system components or elements, as well as the integrated system as a whole…

Qualification testing is distinct from all other forms of testing, [emphasis added] including developmental testing by the vendor, certification testing by a state election organization, and system acceptance testing by a purchasing jurisdiction:

Qualification testing follows the vendor’s developmental testing;

Qualification testing provides an assurance to state election officials and local jurisdictions of the conformance of a voting system to the Standards as input to state certification of a voting system and acceptance testing by a purchasing jurisdiction; and Qualification testing may precede state certification testing, or may be conducted in parallel as established by the certification program of individual states.”

VSS Section 9.4.1.4 notes that:

"The interface between the voting system and its users, both voters and election officials, is a key element of effective system operation and confidence in the system. At this time, general standards for the usability of voting systems by the average voter and election officials have not been defined, but are to be addressed in the next update of the Standards. However, standards for usability by individual voters with disabilities have been defined in Section 2.7 [sic: should be 2.2.7] based on Section 508 of the Rehabilitation Act of 1973, as amended in 1998. Voting systems are tested to ensure that an accessible voting station is included in the system configuration and that its design and operation conforms with these standards." [emphasis added]

Appendix C is the VSS’s preliminary statement on requirements for usability. The requirements are a mixture of functional and design specifications, with the latter being somewhat predominant. The level and specificity of the requirements vary greatly. For example:

"...the cursor should be automatically positioned in the first data entry field and when the voter hits the 'enter/return' key, the cursor should automatically move to the next data entry field;"

"...fields where voters have to enter identifying information, if any, should be clearly labeled and the place where the information is to go should be clearly visible;" "The display should provide orientation and landmark features to support the voter in determining where they [sic] are in the ballot;"

Section C.1 emphasizes formative rather than quantitative/summative testing. For example, this section states:

"Results from the tests and evaluations can be used to correct any design deficiencies before the system are [sic] actually used for voting."

4.1.2.3 Pertinent Features of VSS: Volume II, Testing Standards
Volume II provides a set of guidelines for conducting conformance tests. As with Volume I, some of the guidance is very specific and some is very general. Overall, Volume II seems to be a requirements document for the ITAs rather than the description of a specific test suite. However, the test methodology is left up to the ITA. VSS Section 1.5 gives ITAs the power to develop tests as needed, even if the areas are not directly covered by the VSS:

“Taking advantage of the experience gained in examining other voting systems, ITAs will design tests specifically for the system design, configuration, and documentation provided by the vendor. Additionally, new threats may be identified that are not directly addressed by the Standards or the system. As new threats to a voting system are discovered,... ITAs shall expand the tests used for system security to address the threats that are applicable to a particular design of voting system."

This latitude in designing and conducting tests across voting products may be appropriate to allow the ITAs to develop specific tests based on the nature of the technology used, but would not ensure uniform testing of the independent quality of usability and accessibility across all voting products.

VSS Section A.4.3.5 (System-level test case design) tells the ITAs to simulate typical voter errors and gauge the robustness of the system, but this is not the equivalent of actual user interaction:

"Usability tests: These tests are designed to exercise characteristics of software such as response to input control or text syntax errors, error message content ..."

VSS Section B.5 authorizes ITAs to use their own judgment to decide, in some cases, whether a system is accepted or rejected:

"Of note, any uncorrected deficiency that does not involve the loss or corruption of voting data shall not necessarily be cause for rejection. Deficiencies of this type may include failure to fully achieve the levels of performance specified in Volume I, Sections 3 and 4 of the Standards, or failure to fully implement formal programs for quality assurance and configuration management described in Volume I, Sections 7 and 8. The nature of the deficiency is described in detail sufficient to support the recommendation either to accept or to reject the system, and the recommendation is based on consideration of the probable effect the deficiency will have on safe and efficient system operation during all phases of election use."

4.1.2.4 Current Conformance Testing
Currently, there is one laboratory accredited for hardware testing and two for software. The testing process is intensive and time-consuming. A “smooth” qualification process for both hardware and software testing generally takes several months. (A detailed description of the testing process may be found at: TUhttp://www.nased.org/ITA_process.htmUT.) Design guidelines supporting accessibility in the VSS are tested as part of this process (mainly by inspection). However, this current process does not include any significant usability testing or other conformance testing pertaining directly to usability. The VSS Appendix C is advisory and not mandatory.

4.1.2.5 Recent FEC Efforts in Support of Usability
The FEC organized an Advisory Board on Usability and Human Interface Standards in the Fall of 2002 to investigate how concerns about usability and interface standards could be incorporated into the VSS. More recently a series of usability guides for voting systems (FEC Developing, 2003; FEC Procuring, 2003; FEC Usability Testing, 2003) have been issued to assist state election officials and voting system vendors in the design, development, and procurement of usable voting products. While these brochures have high educational value as short tutorials, they do not provide specific details, procedures, or criteria necessary to determine if a product is usable.

4.1.3 IEEE Effort
The Institute of Electrical and Electronics Engineers (IEEE) has undertaken Project P1583 on Voting Equipment Standards under the Standards Coordinating Committee 38 (SCC 38) on Voting Standards to formulate standards for DRE voting equipment. IEEE charged this project as follows:

“Project P1583 is charged with development of a standard of requirements and evaluation methods for election voting equipment. The standard will provide technical specifications for electronic, mechanical, and human factors that can be used by manufacturers of voting machines or by those purchasing such machines.”

Details can be found at http://grouper.ieee.org/groups/scc38/1583/index.htm.

It appears that paper-based systems, such as optical scan systems, are not covered by their effort. The standard is currently (as of October 2003) in draft form and is undergoing review and editing. Within the standards, IEEE Section 5.3 addresses usability and accessibility issues The specifications are a mix of high and low-level, and performance and design requirements. Low-level design specifications predominate in the standard. Only voting equipment is covered in the body of the standard, but there is an informative annex offering guidance for ballot design.

IEEE Section 6.3, which covers testing, classifies testing methodologies into two categories: Standards Compliance and Usability Testing. In the Standards Compliance section, four different methods are used to determine whether or not a voting system conforms to each applicable usability/accessibility standard: inspection, expert –based evaluation, and tests and usability tests.

Inspection (I) the design is inspected to determine whether it possessed a feature or function specified in the standards...

Expert-based analytical evaluation (E) a human factors or usability subject-matter expert performs a comprehensive review ... to determine whether the applicable standards are being met.

Test (T) tests are specified to determine whether the applicable standards are met, e.g., a measure of letter height or sound intensity.

Usability testing (UT) evaluates a voting system by having a representative sample of voters perform voting tasks under realistic but simulated conditions. User performance and user opinion regarding their interactions with the system are measured and compared against usability and accessibility goals and requirements.

The various requirements in IEEE Section 5.3 are mapped onto one of these methodologies as the preferred way to verify that the system under test conforms. Usability testing is mainly reserved for general usability requirements that cannot be tested by the other compliance testing methods.

4.2 Generic Usability and Accessibility Standards

These generic standards contain examples of the various kinds of requirements as described above in Section 2.3 (performance vs. design, specific vs. general, etc.). One further distinction is worth making: some of these standards apply to products in the conventional sense. For others, it is the development process itself that is specified. We refer to the latter as “process-oriented” standards.

Several of the standards are ISO standards for usability. The U.S. does not have anything equivalent as a national standard except for ANSI/HFES 100-1988 (HFES-100, 1988), which covers only ergonomic (physical) requirements of workstations in an office setting; it does not cover software interface design issues or processes. Several Military standards exist that address human factors engineering concerns for equipment (including software) and facilities, as well as for specific military applications (such as helicopter cockpit design), for specific items (such as labeling), as well as planning and process requirements. Many of these documents are no longer supported. One notable exception for process standards development is the Department of Defense, which has standards such as MIL-STD-1472 (MIL-STD-1472) and MIL-H-46855 (MIL-H-46855), covering human factors engineering. But, process documents like MIL-H-46855 and others are no longer being maintained and have been rescinded. Companies in the U.S. tend to rely on industry standards and best practices for their guidance with such documents as the Windows Style Guide (which changes with each release) and other commercially available books on interface design. Another exception is ANSI/INCITS 354-2001 (ANSI/INCITS 354, 2001) a standard developed by NIST for documenting summative usability test results. It is currently in the process of internationalization. Some of the generic standards that are applicable to voting are described below.[15]

4.2.1 Section 508 of the Rehabilitation Act of 1973, as Amended in 1998
The U. S. Access Board is an independent Federal agency devoted to accessibility for people with disabilities. Part of its mission is to develop and maintain accessibility requirements for the environment, transit vehicles, telecommunications equipment, electronic and information technology, and to provide technical assistance and training on these guidelines and standards. It also enforces compliance for federal buildings under the Architectural Barriers Act. The Access Board developed the ADA Accessibility Guidelines for Buildings and Facilities (ADAAG, 2002) that apply to any building where a poll is located. The physical and environment accessibility requirements in the current VSS are based on these standards for clearances and reach, etc. Of particular interest for DREs are the Section 508 Electronic and Information Technology Accessibility Standards (Section 508, 2000). The Standard provides information on accessibility for operating systems and computer applications (including web sites) and it covers both self-contained, closed products and “open architecture" products. The VSS and the IEEE draft standards based their DRE accessibility standards on the Access Board Section 508 Standards.

As stated in the official Section 508 website (see: http://www.section508.gov):

“In 1998, Congress amended the Rehabilitation Act to require Federal agencies to make their electronic and information technology accessible to people with disabilities. Inaccessible technology interferes with an individual's ability to obtain and use information quickly and easily. Section 508 was enacted to eliminate barriers in [Federal] information technology, to make available new opportunities for people with disabilities, and to encourage development of technologies that will help achieve these goals….Under Section 508 (29 U.S.C. ‘ 794d), agencies must give disabled employees and members of the public access to information that is comparable to the access available to others.”

Subpart B contains the technical specifications that apply to a wide variety of IT products including software, web-based applications, multimedia and PCs. From the 508 summary page:

“This section provides technical specifications and performance-based requirements, which focus on the functional capabilities of covered technologies. This dual approach recognizes the dynamic and continually evolving nature of the technology involved as well as the need for clear and specific standards to facilitate compliance. Certain provisions are designed to ensure compatibility with adaptive equipment people with disabilities commonly use for information and communication access, such as screen readers, braille displays, and TTYs.”

Thus, we see that the Standards encompass both design and performance specifications, and both quality-oriented and interoperability requirements. As with other standards we have examined, some of the requirements are very low-level and specific, others are very general.

4.2.2 ISO 9241: Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs)
The 17 parts of this standard cover many aspects of working with VDTs. It is one of the largest and most detailed usability standards in effect today. The various parts of 9241 exemplify the extremes of standardization. Some parts are highly technical and require very specific quantifiable properties. Other parts offer general guidance on how to approach certain tasks.

As an example of a technical specification, consider Part 7: Requirements for Display with Reflections. Section 4 carefully defines technical concepts and metrics and how they are related mathematically. This allows precise characterization of hardware performance (e.g. luminosity at various angles). Section 5 states that the purpose is to assure that VDTs be "legible and comfortable in use". Section 6 lays out the precise requirements to be met. Section 7 then describes the approved test method (lighting, optical instrumentation etc.) for ascertaining conformance. It is notable that the standard itself defines the test method as well as the requirements.

The next notable point is that the standard anticipates (as a future technique) a completely different test method based on the performance of human subjects, namely their ability to read text from the screen under various lighting conditions. This test method more directly addresses the purpose of the standard as set out in Section 5, but is less closely tied to the technical requirements of Section 6. Thus, there are really two approaches described within the standard:

• Specifying the physical characteristics of the light coming off the screen, as measured by optical instruments (in our terminology, a design, specific, low-level standard), and
• Specifying the resulting legibility of text and graphics, as measured by human performance testing (a performance, specific, high-level standard).

Later in the same document, in contrast with the highly technical nature of Part 7, Part 10: Dialogue Principles is a very general, non-binding standard. As its name implies, it simply describes some design principles that should be taken into consideration when developing a system that interacts with human users via a VDT. It is a good overview and tutorial, but is not a standard in the strict sense. Likewise, Part 11: Guidance on Usability defines and discusses basic concepts of usability (effectiveness, users, tasks, etc). The discussion is very thorough - comparable to a long introductory chapter in a textbook.

4.2.3 ISO 13407: Human-centered Design Processes for Interactive Systems
This standard "provides guidance" to project managers on how to incorporate human-centered design into their development processes. The tone and content is more like a survey article, rather than a true standard (note the use throughout of the non-binding "should"). Like the Common Industry Format for Usability Test Reports (see below), this is a process-oriented standard.

In the conformance section, the standard indicates that the project manager must generate documentation showing that the procedures of 13407 were followed. The "level of detail" of the documentation is to be negotiated by the "involved parties". Annex C provides templates for such documentation, which is to be evaluated by an "assessor". Thus, the test procedures fall into the category of subjective inspection. ISO 13407 contains a good deal of useful information -- it provides a checklist of possible techniques to be used by the conscientious project manager who wishes to improve the usability of a product.

4.2.4 ISO 16982: Ergonomics of Human-System interaction -- Usability Methods Supporting Human-Centered Design
This document supplements ISO 13407. As a technical report (TR) it is "entirely informative" and is therefore not a standard. Its purpose is to describe several usability methods and recommend when they are most applicable. Thus, like ISO 13407, this document is basically advice for the project manager. Twelve methods are discussed, eight of them involving the use of human subjects. Overall the TR presents a useful survey of available methods and techniques. As a consensus document, it is not "cutting edge" but rather collects, organizes and presents the common wisdom on applicability of usability methods.

4.2.5 ISO 10075: Ergonomic principles related to mental workload
This standard has two parts: General terms and definitions, and Design principles. Part 1 contains definitions and general terms. Part 2 contains a discussion of general design considerations and approaches. Included topics are: ambiguity of goals (more ambiguity implies more stress), complexity of tasks, and redundancy of information. Possible solutions include frequent rest periods, better illumination, and job rotation. It is not really a standard in the strict sense, but rather a general discussion of some high-level problems and goals related to usability.

4.2.6 ANSI/INCITS 354-2001: Common Industry Format (CIF) for Usability Test Reports
The ANSI/INCITS 354-2001 defines a format to be used by someone who has conducted a usability test on a product (vendor or third-party) so that the report of the results of that test is reported in a standard and interchangeable way, thereby encouraging buyers to take these results into account as part of the total cost of ownership when procuring software. This is not a specific design standard, but one oriented towards the process of evaluating usability.

5. Current Human Factors Engineering, Usability, and Accessibility Research

5.1 Background: Basic Research

In addition to issues of applicability based on the research design parameters used, there is the issue of the interaction effect of variables. By isolating a single variable of interest, research can make conclusions about this variable. However, in a real world situation, this variable may interact with other variables. For example, the current VSS standards include information on minimal size for text to ensure the text is readable. This value is based on research with font size as an isolated variable (i.e., with lighting, contrast ratio, color, font style, and other factors held constant). In a real world situation, differences in contrast ratio, font style, lighting condition, display density, or even user fatigue could change the minimal font size requirements. Under some conditions, the minimal requirements for font size might need to be higher than stated and under other combinations, lower font sizes might suffice.

Usability Research Related to the Design and Testing Process

The user-centered design (UCD) process, and its derivative forms, is an approach that includes interaction with users throughout the product’s design and development cycle to gather data and test design assumptions. The basic concept is to ensure that usability is incorporated into a product’s design from the beginning of the design process and evaluated throughout the development process. Methods of incorporating usability include the use of user profiles (or personas), the development of use case models, usability walkthroughs, heuristic reviews, expert review, and user-based testing. Story boards, mock-ups, and prototypes can each be evaluated to test design assumptions and interaction effects. These activities serve to provide formative or diagnostic data on a product from conception through deployment.