Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
TGDC Subcommittee Work - Historical Meetings - CRT Meetings - 2007
|
Joint CRT and STS Teleconference Agenda:
Attendees: Alan Goldfine, Allan Eustis, David Flater, Jon Crickenberger (NVLAP), Michael Koo, Nelson Hastings, Wendy Havens Administrative Updates (Allan Eustis):
CRT's Plenary Presentation (David Flater): CRT only plans one presentation for the meeting, being given by David Flater.
This is the last scheduled meeting before the plenary. The need for future teleconferences will be decided after the July 3rd plenary. Meeting adjourned at 11:20 a.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
|
************
|
Joint CRT and STS Teleconference Draft Agenda:
Attendees: Alan Goldfine, Alicia Clay, Allan Eustis, Angela Orbaugh, Bill Burr, David Flater, David Wagner, John Wack, Jon Crickenberger (NVLAP), Michael Koo, Nelson Hastings, Paul Miller, Philip Pearce, Ron Rivest, Thelma Allen, Wendy Havens, Rene Peralta No Administrative Items. CRT Issues (David Flater) The issues log being maintained by CRT contains over 273 issues, with 27 remaining open as of today. Many of the open items are regarding terminology waiting on feedback from Paul Miller. In order to make the deadline for final draft, the open issues need to be closed by Friday, June 15.
Next CRT meeting scheduled for Thursday, June 21. (Possibility it will be cancelled).
Next STS Meeting Tuesday, June 19th. Meeting adjourned at 12:30 p.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
*********** |
|
CRT Teleconference Draft Agenda:
Attendees: Alan Goldfine, Allan Eustis, David Flater, John Wack, Jon Crickenberger (NVLAP), Mat Masterson (EAC), Thelma Allen, Wendy Havens Administrative Updates:
Issues Received (David Flater): At the last meeting, it was decided that issues with terminology would be discussed with Paul Miller, CRT is awaiting feedback. Issues discussed today:
The issue log is currently about 165 pages, with 1 page per issue. David Flater has incorporated many comments and given tentative resolutions for many. To summarize the issues, most are editorial, lots of glossary terms, David Wagner had a lot on coding standards which were not serious modifications but clarified a lot of ambiguities. A lot of comments were received from Ron Rivest and David Wagner, we are awaiting comments from other TGDC members. Next meeting is scheduled for Thursday, June 14, 2007. Meeting adjourned at 11:30 a.m.
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
|
************
|
CRT Teleconference Draft Agenda:
Attendees: Alan Goldfine, Allan Eustis, Brit Williams, David Flater, John Wack, Jon Crickenberger (NVLAP), Michael Koo, Nelson Hastings, Sharon Laskowski Administrative Updates:
Specific Issues on CRT Chapters Received from TGDC Members and Public (David Flater): Open Issues:
Closed Issues:
Next CRT meeting scheduled for Thursday, June 8, 2007. Meeting adjourned at 11:50 a.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Teleconference Draft Agenda:
Attendees: Alan Goldfine, Allan Eustis, Brit Williams, David Flater, Jon Crickenberger (NVLAP), Michael Koo, Paul Miller, Wendy Havens, Administrative Updates (Allan Eustis):
CRT's Plenary Presentation (Alan Goldfine and David Flater): David and Alan gave a quick overview of the material to be presented at the plenary meeting (presentation emailed with agenda):
Plenary Meeting (Allan Eustis): Will have a better idea how day # 2 will run after today's meeting with subcommittee chairs. Bill Jeffrey would like to approve sections of the VVSG by subcommittee after their presentations. Significant portions of CRT material ready for approval. Schedule of Upcoming CRT Telecons (Alan Goldfine): We will continue to schedule CRT telecons for Thursday at 11:00, starting again on May 31 through the end of June. Email will be sent out to subcommittee regarding dates. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Teleconference Draft Agenda:
Attendees: Alan Goldfine, Allan Eustis, Brit Williams, Dan Schutzer, David Flater, Michael Koo, Paul Miller Administrative Updates (Allan Eustis):
EMC Requirements (Alan Goldfine): The paper as discussed at the last CRT teleconference has been submitted to the editors for formatting and incorporation into VVSG draft. There was some tweaking to the informative language but nothing significant. Reliability and Accuracy Benchmarks (David Flater): Two small points: 1) We clarified the issue of how the device class control tabulator only listed benchmarks for non-user serviceable failures. The amendment made was under failure type, instead of non-user serviceable, it was changed to "all" with a footnote saying that apart from misfeeds (and jams) which are handled by a separate benchmark, TGDC experience is that central tabulator failures are never user serviceable. 2) The second issue was in regards to the misfeed rate. The benchmark was set on a 99% curve, meaning the benchmark was set so that a conforming device should leave no more than 1% chance of blowing the benchmark in the example election for the volume positive. In case of misfeeds, the benchmark was set to the rate quoted, 1 in 500. Draft VVSG (Allan Eustis for John Wack): The VVSG draft will go up on the web Friday and will be substantially the version that is printed for discussion at the May 21-22 plenary meeting. It is a near final draft, however some security sections are still preliminary. The plenary will focus mainly on issues of major concerns. On day 2, there will be discussion of the next steps and future TGDC meetings. We will also discuss the VVSG final review process. We are completing this document under the laws established by HAVA. TGDC Logistics (Allan Eustis): The web page is up as well with logistic information and draft agenda. TGDC members will receive a CD will all the information on it. Final VVSG will go on the web on Friday and also burned to CD sent to TGDC. Other:
Meeting adjourned at 11:30 a.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Teleconference Draft Agenda:
Attendees: Alan Goldfine, Allan Eustis, Dan Schutzer, John Wack, Jon Crickenberger (NVLAP), Mat Masterson (EAC), Michael Kass, Michael Koo, Nelson Hastings, Philip Pearce, Sharon Laskowski, Steve Freeman (NVLAP) Administrative Updates (Allan Eustis):
EMC Requirements: The latest draft requirements were posted. This is first version since the one discussed at the TGDC plenary meeting. It includes draft requirements for radiated emissions and telecommunication interference. Still in rough format - we are working on informative text, organization, some references, and testing procedures. It is complete in that it covers everything we intended to include. QA/CM Requirements: The complete version was discussed two weeks ago. As a result minor word-smithing changes were incorporated. This version has been sent to be included in current VVSG draft. E-poll books (John Wack): John summarized the joint STS/CRT/HFP subcommittees meeting. John Kelsey had discussed possible threats to privacy, being concerned over personally identifiable information being contained on one system that is used to activate ballots. The risk is that someone could piece together the information to figure out who voted for whom. There was also discussion about externally networking epoll books and ballot activation, concerns being for security and reliability. The decision at the meeting was to allow epoll books to activate ballots. The activation mechanism needs some additional requirements. David Flater has written some additional general applicable requirements. These need to be specific and strengthened. The decision was also made if epoll books were externally networked; they were not to be used for ballot activation. Some alternative options were discussed and these will be presented to the entire TGDC at the May plenary meeting. Next CRT meeting scheduled for May 3rd may be canceled or given to another subcommittee. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Teleconference Draft Agenda:
Attendees: Alan Goldfine, Allan Eustis, David Flater, John Crickenberger (NVLAP), John Wack, Mat Masterson (EAC), Michael Kass, Michael Koo, Nelson Hastings, Paul Miller, Sharon Laskowski, Wendy Havens Administrative Updates (Allan Eustis):
Reliability and Accuracy Benchmarks (David Flater): The document posted on the web (see above) reflects the changes that were discussed at last weeks meeting. David went over several items in the paper. It was decided to keep the 120,000 voting sessions for EBM ballots. There will be a 1 in 500 allowance for paper feed errors/jams. It was decided that there would be no more than 6 errors (failures) allowed of any sort pending Paul Miller's final review. David was planning to finalize document at end of day so any comments were due immediately. E-PollBooks (John Wack): John Wack brought up the topic of e-pollbooks. The EAC requested the TGDC to take a look at writing requirements for e-pollbooks - at a minimum in regards to the use of them for ballot activation. Due to the short time left until the next iteration of the VVSG is due out, it was decided that we could not provide more than high-level requirements. That is what David Flater has done - he has added a device class in the VVSG for ballot activators. The specific questions today were in regards to allowing networked devices, such as epollbooks, to also activate ballots. (The requirement in the VVSG currently states that any part of the voting system can not be externally networked and if epollbooks are used as ballot activators, they are part of the voting system.) Some states are currently using the networked epollbooks to activate ballots. The main concerns are security, availability and privacy. Paul Miller understands the availability issue being a major concern, but also thinks there are possibilities of handling this by having backup ballot activation techniques. This topic will be discussed by all three TGDC subcommittees on April 24th.
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Teleconference Agenda:
Note: The QA/CM paper (revision 3) is complete, pending CRT review. Track Changes (in the .doc file) documents all changes made since the plenary version. Attendees: Alan Eustis, Alan Goldfine, Alicia Clay, David Flater, John Crickenberger, John Wack, Mat Masterson (EAC), Michael Kass, Paul Miller, Sharon Laskowski, Wendy Havens Administrative Updates:
Latest News on Reliability and Accuracy Benchmarks (David Flater): There are no modifications to this week's document from last week. David is currently waiting on feedback from the vetting Paul Miller is doing with NASED. Paul said the feedback he is receiving is pretty consistent. There might be small modifications, but nothing substantial. Based on this input, David will work to finalize document and harmonize to the official draft. He will make final changes to the draft in the time frame May 7 - 9. There will still be some opportunity for change before the July deadline. Paul Miller wanted to know if David planned on making any changes to the background assumption material that he had provided; only editorial changes, nothing significant. Some of the language Paul used will be used to help clear up some definitions. Quality Assurance/Configuration Management (Alan Goldfine): Please see the document referenced to in the email - the copy on the web does not include the track changes that were specifically left in. There were quite a few editorial and formatting changes, but only a few major changes that are pointed out. First, on page 6, requirement 2.1-a regarding the quality assurance manual's delivery deadline was changed to remove the statement "delivered before the start of the design and development process". There is no time specification. Second, on page 8, the list of critical parts and components was changed based on conversation with Philip Pearce. We added to conditions for critical parts and components, 1) components whose failure would violate a voter's ability to vote independently and 2) would impede the usability of the voting system by all voters. These two components support usability and accessibility. John Wack felt there were a few requirements that STS should review, such as physical identification of the system and the requirements about voting system configuration log. Ongoing Maintenance of Draft (David Flater): Nothing new - made changes discussed last week. Other Issues: John Wack suggested a meeting to over issues of concern in the VVSG for each subcommittee before the plenary meeting on May 21-22. There will be an STS meeting in two weeks on e-poll books. CRT and HFP are invited to participate in the meeting. Meeting adjourned at 11:30 a.m.
[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Teleconference Draft Agenda:
Attendees: Allan Eustis, Brit Williams, David Flater, John Crickenberger, Mat Masterson (EAC), Nelson Hastings, Paul Miller, Sharon Laskowski Administrative Updates:
Reliability and Accuracy Benchmarks (David Flater): Current state of the Benchmarks research paper by Paul Miller and David Flater was forwarded to the subcommittee. This was prepared in response to the direction of the TGDC at the last plenary meeting - namely to attempt to gather what data/guestimates we could about different volumes of usage of different devices so as to come up with an estimate for what the benchmark for reliability and accuracy should be in at least the correct order of magnitude. From NASED feedback, no given number is going to be correct for the usage in every possible jurisdiction. Part of the direction that was taken was that different severities would be assigned to different types of failures in a manner that was not entirely different than what was done in the 1990 voting system standards. [NOTE: David Flater asked why the scoring standard in the 1990 standard was taken out of the 2002 revision that described what a relative failure is and assigned different weights depending on what kind of failure it was. Brit Williams said there was not a major show stopping reason it was deleted. He noted that Appendices were not par of the 1990 VSS.] Paul Miller is going to talk to other members of NASED to confirm numbers and approach. Assuming the process comes out favorably, the draft will be changed to expand the single benchmark into a collection of benchmarks that are tailored to individual types of equipment. After that, there is an attempt to answer what constitutes a typical volume for these different types of equipment, understanding that each jurisdiction will understand different what "typical" means. There's a breakdown of how many errors/failures a precinct can tolerate on Election Day. Failures are classified differently depending on whether personnel present can easily remedy situation versus calling in specialized assistance. Next Step: Paul Miller will confirm or modify numbers based on discussion with NASED members. David will get material prepped for inclusion into draft VVSG. Ongoing Maintenance of Draft (David Flater): David is still catching up with changes discussed at plenary meeting. Including such changes as:
Meeting adjourned at 11:45 a.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Teleconference Draft Agenda:
Attendees: Alan Goldfine, Allan Eustis, Dan Schutzer, David Flater, John Wack, Lynne Rosenthal, Mat Masterson (EAC), Nelson Hastings, Paul Miller, Sharon Laskowski, Steve Berger Administrative Updates:
Discussion of QA/CM Research Paper: This paper has been posted on the web for a week. The material in this paper has not been included in the VVSG yet, it should go to coordinator of paper in the next week. Alan Goldfine discussed the plenary presentations. The CRT subcommittee plans to cover 5 topics: 1) electromagnetic compatibility, 2) quality assurance/configuration management requirement, 3) review of CRT changes, 4) discussion of benchmarks, and 5) general discussion. For electromagnetic compatibility, the requirements are still in the process of being developed. These requirements have been divided into three parts: conducted disturbance (should be completed next week), radiated disturbances (draft set of requirements by March 30), and telecommunications (incomplete, 1st draft by March 30). (Steve Berger forwarded these to IEEE for comments which are forthcoming.) QA/CM requirements raise an issue. The requirements are being done as a result of a TGDC resolution directing the review of QA/CM. At the December 2006 meeting, it was decided that the ISO 9000/9001 standard would be used for quality assurance. This is just the beginning of the process . Details are needed to state what conformance means. Vendor requirements are more specific - required documentation, data delivered to test labs and EAC. The draft requirements state two different alternatives which were discussed in great detail. The first alternative requires that vendors provide their quality assurance procedures ( quality manual) early in the process, before design/development of system begins. The other alternative requires that the quality assurance procedures be done, but there is no specification. The problem identified : if a system's quality assurance procedures are not approved, and the design/development is well on its way, a system may not be approved and there would be wasted time and effort. Opinions were expressed that this did not seem to be a problem, that the real question was if the system passed certification, the manufacturing process must produce systems of the same quality. This topic did not appear to be resolved. It will be brought up at TGDC plenary meeting for discussion and possible consensus. It was pointed out that there was no time specified in the EAC manual regarding the registration process. Mat Masterson will bring this up to the EAC. David Flater then discussed what he would be presenting at the plenary meeting. The first topic would be a review of the changes as discussed at the last CRT subcommittee meeting. David will also be presenting information regarding benchmarks. CRT has received feedback from NASED regarding the question about resetting benchmarks on reliability and accuracy in the next VVSG. Defensible numbers need to used for the benchmarks. The response from NASED in response to reliability said that no failures that lead to unrecoverable votes are acceptable, in other cases, our tolerance for failure depends on how hard it is to recover from those failures, there is no typical volume on which to base a benchmark. We currently do not have a particular volume to base a benchmark; however, if test labs are going to advise rejection of systems that perform unreliably during testing, there needs to be a benchmark for what constitutes an unacceptable rate of failure. With respect to accuracy, we have a real requirement that does not map to a particular benchmark. The acceptable number of errors is 1 less than the vote margin between first and second place. The current benchmark is 1 in 10M ballot positions - this was set as a compromise based on costs of testing. NASED acknowledged the need to review test methods and also expressed concern that 1 in 10M is probably not achievable for real ballots. Since we want volume testing to produce realistic ballots, the benchmark should meet everyone's requirements but also be attainable. Should the benchmark be relaxed so that it is attainable? The route we went through did not produce the data we need to produce defensible benchmark numbers. There is not a lot of time left. At the TGDC meeting, we plan to ask our customers for input about what this number should be. Paul Miller informed the group that NASED had concerns about putting into writing indicating an acceptable rate of failure. There was also a concern about the definition of terms. We have to look at what are "failures". If poll workers can't set up systems, that's a failure. If a printer jams, that's a failure. However, there are manageable failures and those should be identified. David stated that failures were defined as equipment breakdown, including software, so that continuous service is worrisome or impossible. Complete testing of systems in voting environment cannot be tested as we do not know how many types of machines a precinct will have and how many as backup, if any. Failures fall anywhere in a large spectrum. If we ask test labs to try and evaluate the severity of failures, we will get ambiguous results. Any comments/questions should be sent via email. The meeting adjourned at 12:10 p.m. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary, pre-decisional and do not necessarily reflect the views of NIST or the TGDC.]
************ |
|
CRT Subcommittee Teleconference* Agenda:
Administrative Updates (Allan Eustis):
Letters to NASS and NASED Regarding Benchmarks (David Flater): It was determined at the end of the last TGDC meeting that we needed election community buy-in of the benchmarks in terms of accuracy and reliability. Letters were sent to NASS and NASED asking for comments/consensus on these issues. NASS has declined to take a position, we have not heard back from NASED. (It was noted that due diligence was taken to get feedback.) Comm. Davidson suggested contacting State Election Directors to find out what they are doing. CRT had been advised by NIST General Counsel not to do this, but it was suggested that EAC might be able to. Comm. Davidson will check with EAC General Counsel.
Discussion of Quality Assurance (QA)/Configuration Management (CM) Research Paper: This paper was written after consensus at the December meeting that the ISO 9000 and 9001 standards should provide the framework for the VVSG 07 requirements dealing with QA/CM. This is a first draft of a set of requirements that attempted to do that. ISO 9000 is a general set of requirements - the real key is to design specific requirements that work for voting systems - that is what this paper tries to do. Alan asked for comments/feedback - did we address key issues, is it outside our scope, did we address everything within our scope? Steve Berger commented that this was going in the direction he had in mind. Steve also wanted to know if a vendor changed hardware/software, did it show up as a version change? Alan noted this specific issue was probably covered in configuration management. It is a valid concern and he will look into it.
Re 1.2.8.6, Philip Pearce asked about the critical parts list of what constitutes a failure of a part or a component - should there be something added regarding diminished accessibility or usability. Alan agreed and asked if Philip would draft up some appropriate bullets.
[NOTE: In 2005, TGDC requested process model - CRT's working draft contains informative process model.] EMC Draft with Outline: Discussion was held at a previous meeting about revisions of electrical requirements within environmental requirements section of the VVSG. It was suggested CRT reach beyond the voting team to experts in the field of electrical equipment testing. We've talked to experts (at NIST) regarding radiated interference and conducted electromagnetic interference. Based on these discussions, we have outlined work and some draft requirements for this area. Alan inquired if there were any general opinions or concerns regarding this outline. Technical questions should be sent via email to Alan Goldfine. Steve Berger questioned if there was any field data about problems currently with these issues? The answer was no, but that should not stop the group from looking into it. Steve also mentioned that there were a lot of "to be determined" specs; including requirements question marks. Alan pointed out that this represented discussion about whether certain topics need to be developed within volume 5 of the VVSG. Are these testing standards or are they specifically testing scenarios. This is a work in progress - all the TBDs will be determined. Comm. Davidson suggested getting comments from the vendors about costs after Alan receives Steve's questions via email. John Wack, Allan Eustis, and Alan Goldfine will discuss offline about bringing this up at the next vendor's meeting. Discussion Draft (Summary of Changes): Revised set of draft requirements are available on the web. (Specifically this is David Flater's material which has been submitted to the editorial team for review and formatting.) The change log file contains a count of things that were changed between drafts. David went over a few of the changes. Biggest change was in the introductory informative text to coding conventions (C language with use of COTs add ons becomes structured language) , and in the follow on changes to the conventions themselves. Other changes (subject to revision) included: data to be provided, functional testing, conformity assessment and use of components. The question was asked if for the definition of EBMs (electronically assisted ballot markers), was consideration given to blindness or visually impaired. David noted that he clarified the definition of EBMs but that the issue with blindness or visually impaired was being looked at specifically by HFP. Meeting adjourned at 12:00.
************ |
|
CRT Subcommittee Teleconference* Agenda:
Participants: Alan Goldfine, Allan Eustis, Brit Williams, David Flater, Commissioner Donetta Davidson, John Wack, Nelson Hastings, Philip Pearce, Sharon Laskowski, Wendy Havens, Dan Schutzer Administrative Updates (Allan Eustis):
Report from the EAC Advisory Board (John Wack):
Other Issues: Commissioner Davidson mentioned a question brought up to her at a meeting about problems with sending results over analog lines. Allan had heard the same problem in WY.
Future CRT phone meetings are scheduled for: Thursday, February 15 at 11:00 a.m. (Topics of discussion to include technical aspects proposed for quality assurance and configuration management as well as electromagnetic requirements.) Thursday, March 1 at 11:00 a.m. Thursday, March 15 at 11:00 a.m.
************ |
|
CRT Teleconference* Agenda:
Participants: Alan Goldfine, Allan Eustis, Brit Williams, Dan Schuzter, David Flater, Max Etschmaier, Nelson Hastings, Paul Miller, Sharon Laskowski, Stephen Berger, Philip Pearce, Wendy Havens Administrative Updates (Allan Eustis):
Volume testing, reliability, and accuracy testing discussion paper (David Flater): This write up is is a realization of things that were discussed at TGDC meeting. First major addition is requirements to conduct volume testing similar to CA volume testing reliability protocol. Specific parameters were modeled after this protocol -- one exception is with respect to central tabulators. The other part has to do with test methods for accessing conformance to the benchmarks for reliability accuracy and probability of misfeeds for paper based tabulators. Following that are specific requirements for reliability, accuracy, and probability for misfeeds. Discussion regarding this paper among subcommittee members followed. After discussion, Allan asked Steve Berger to summarize his concerns so that David Flater would know which direction to proceed. Steve: Looking at the testing resource required by the draft (Table 7 below), we should look at it two ways. First, determine what resource is being assumed and explicitly state that this current draft would require "x" number of man hours of testing. Second, if the minimum level of testing is performed, what's the confidence that can be stated; if there's not enough testing to make a credible statement, then we need to look at why we're doing the test in the first place. Second point, do the tests, as currently prescribed, isolate problems being observed in the field. For example, the calibration issue. Do the tests we currently have screen out future systems that may require too frequent calibrations. Do we understand the causal factor? If we don't, then we need to do research.
David Flater provided a high level preliminary answer. We have not specified a particular test suite. We have specified a test method. Specific resources and results at end of conformity assessment depend in large part on the test suite used, which is currently developed by the test labs and approved by the EAC. With respect to isolation of specific issues, those issues are targeted by a series of environmental tests, from operational and non-operating points of view. We can always do research on causal issues. Allan Eustis pointed out that some of these issues would be examined at an upcoming workshop by EAC on cost of testing. Also, relevant policy-related issues need to be referred to the EAC. Steve Berger stated that the fundamental question had to be asked "would problems in the field today be prevented by the VVSG 2005 (as published), if not have we rectified the concerns in order that issues would be effectively identified and handled under VVSG 2007?" We can use this methodology on a number of field issues. We should take a systems view. (Max's next paper on Quality Assurance will address some of these issues and will be released in the next week or so. It will be a discussion item at an upcoming CRT meeting.) Election Management Systems (EMS) Discussion Paper (David Flater): There is confusion over the definition of the Election Management System (EMS) as defined in the 2002 VSS. The current definition covers both pre and post voting functions. The question posed is whether the definition should be revised, making a distinction between pre and post requirements. (Brit Williams pointed out that originally (VSS 1990) the definition referred to firmware and software, and that pre and post election were added later. Some vendors group both functions into one software package, others separate it by function. EMS is run on standard desktop/laptop computers.) Consensus at the meeting was that the current definition was on track, and that follow up concurrence would be asked for from ITAA. Data Collection for Reliability and Accuracy Benchmarks (David Flater): In order to set benchmarks for reliability and accuracy, we need to know what the customer wants to see in terms of reliability and accuracy. Since se have controversy over testing, it puts benchmarks on hold. The way we do this data collection is currently being reviewed by the legal office. The question arose about availability. How is availability defined? Turn on a machine and if it works it means it's available. Concerns were expressed over the fact that logic and accuracy testing was not performed at the precincts before use. It was also discussed that some issues/problems thought to deal with calibration problems turned out to be usability problems. Some of these issues are being looked at by the HFP subcommittee. Concerns expressed by CRT participants: Does a system go out of calibration during transport? Are the environmental tests being performed sufficient to determine this? What about older technology problems such as memory packs that are either old, or the technology on them is old? Separate component issues were raised. We need to evaluate the components of the system, as well as the system as a whole. Manufacturers of components need to be contacted about expectations of individual components. We need to know if touch screen systems are being deployed in a manner that goes against manufacturers' advice or if they are being subjected to expectations beyond what manufacturers say are reasonable. We then need to look at the process to detect when this is happening. We also need to make sure that vendors are in agreement with component manufacturers about what to expect. Vendors do not want products used in unacceptable operating conditions. Next teleconference is scheduled for February 1, 2007, at 11:00 a.m. EST. [* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion served the purposes of the CRT subcommittee of the TGDC in directing NIST staff and coordinating its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]
************ |
