Guidelines Development Committee (TGDC)
Our next telecon is scheduled for: Thursday, April 26, 2007 at 1 PM ET
Attendees: Alexis Scott-Morrison, Alice Miller, Allan Eustis, John Crickenberger (NVLAP), John Cugini, John Wack, Matt Masterson (EAC), Nelson Hastings, Philip Pearce, Secretary John Gale, Sharon Laskowski, Steve Freeman (NVLAP), Wendy Havens, Whitney Quesenbery
Discussion of Test Methods:
This meeting was dedicated to a discussion with NVLAP regarding test methods. The goal of HFP was to take a look at every requirement and try and define an appropriate method for testing it. The requirements fit into three categories: high-level requirements that are specific (tested mostly through performance tests); high-level requirements that are vague (no specific features that map to testing, done via expert judgment testing); and low-level requirements that are specific (these are easy to test and write test methods).
Some examples were given. For color blindness, a low level, specific requirement would be that any information displayed in color must also be explained in text. For plain language, the requirement is a high-level, vague saying everything must be written in plain language and avoid passive language unless it is appropriate - this is subjective and can only be tested by experts in this field.
John Gale inquired if something over time could change from one testing category to another? John Cugini pointed out that if something passes the high level tests it may not be necessary to run through the low level requirements. As technology changes and requirements change, the test methods will change.
One issue for NVLAP is how to choose expert testers. They need to understand the type of expert knowledge that is required so that they can find the correct experts. Also there is the issue of who designs the tests. Up until now the test labs have not only done the testing, but designed the tests. The goal here is to centralize the tests - such as when the Access Board develops usability tests for standards and they should be used in each of the test labs therefore making consistency more attainable in test results. In the past VVSGs, we've only provided requirements, this time we will also be providing test methods.
What can the subcommittee do to help NVLAP? What issues does NVLAP have? Some of the standards in the current VVSG are ambiguous. (That will hopefully be fixed in the next version of the VVSG - we give some specific failure criteria and good descriptions about what is being done and what is being checked.)
[NOTE FROM NVLAP: NIST/TGDC members who are interested in our test methods are welcome to attend assessment visits to laboratories.]
NVLAP would like to require a cross reference matrix saying which test method is testing which requirement. This has never been made an official requirement, and legally it currently can't be forced. Whitney pointed out that we could write within the VVSG what test method went with which requirement.
Another problem with testing is that the labs want to test everything on a voting system in one test scenario. This is not practical. NVLAP was glad to see the language about not testing the 20,000 different criteria in one particular scenario. You can not catch every requirement by running one session. The scenarios are trying to come up with realistic variations on the use including people with different disabilities and different things that could happen.
Another issue of testing was the testing method for alternative languages. The testing does not specifically have to test every language; it should test the variation of technology (logic, storage, and presentation) that generates the languages. The testing needs to include the way languages are stored and the way they are presented.
Other issues that tNVLAP has come across is in the mobility requirements. This includes height requirements for tables. We assume that systems will be used as specified by the vendor, but not always the case. Machines can be tested in the lab but not "fit" appropriately in the polling place because vendor specifications are not followed completely.
Reporting requirements for testing has been an issue. The VVSG will include a general testing document that will cover general issues, including reporting issues making it clear who has to report what, when, and in what detail.
Testing by the VSTLs cannot cover everything. It was reminded that they can cover high level requirement testing, but the state testing and certification must also be done so that machines are certified in more detail with state and local laws. Whitney wants to make sure that it is clear who is to test what so that no gaps in testing occur.
There appears to be an issue that is caused by allowing "straight party" votes. Not sure what the fix of this is. We do have a requirement that states that electronic systems must prevent over voting. The problem is we can't tell how "straight party" voting interacts with individual votes also cast.
(Expert) Testers have to go through specific training, which will be documented. This is another place where test methods are invaluable because you can't qualify a person to undefined test methods. Also they will help NVLAP do a better job in accrediting labs, and we will have more consistency across the tests.
Expert testers will be covered by having a test method that is documented and validated, that the tester can perform it correctly because they have been qualified to perform that particular test method, and only people that are qualified are allowed to do the test method for final reporting and certification.
adjourned at 12:00 p.m.
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the HFP subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
policy / security notice / accessibility statement