Joint Technical Guidelines Development Committee (TGDC)
Human Factors and Privacy Subcommittee (HFP) &
Security and Transparency (STS) Teleconference *

February 23, 2007, 11:00 a.m.
Draft Minutes


1. Software Independence and Implications to Usability of Audits
2. Review of Software Independence Approaches Chart

Attendees: Alexis Scott-Morrison, Allan Eustis, Barbara Guttman, Bill Burr, David Baquis (U.S. Access Board), David Flater, John Cugini, John Kelsey, John Wack, Mat Masterson (EAC), Mohammad Mareuf , Nelson Hastings, Philip Pearce, Ron Rivest, Sharon Laskowski, Sharon Turner-Buie, Wendy Havens, Whitney Quesenbery,

Administrative Items:

  • Allan introduced Mat Masterson, who will be working with Commissioner Davidson as a liaison between EAC and the TGDC.
  • Sharon Laskowski expressed a desire to have a consensus at the end of the discussion that this is the paper we want to release as an explanatory document at the next plenary and have some action item in the form of requirements in the accessibility section acknowledging how to address SI in accessibility.

Ron Rivest reported that a high level issue here with software independence- we have to be able to do audits - audits are how we detect errors. The auditing process itself is part of the whole package and must be usable by poll workers in a reasonable way and this must be tested. Ron and STS are looking to HFP for guidance on how best to do this from a usability stand point.

This topic about usability of SI and usability of audits was discussed in great detail.

  • Concerns were expressed over audio playback being slow, but it was pointed out that when auditing records, this was usually handled by one poll worker reading to another.
  • The record itself (the ballot being audited) can't be something that is degraded by being handled - this may be a core requirement issue.
  • A description of the envisioned auditing process should be provided by the vendor, as well as a time analysis.
  • For VVSG 07, what are the requirements, what kind of systems would fail?
  • Alexis expressed concerns that TGDC was stepping out bounds beyond writing requirements for systems and telling election administrators that they needed to conduct audits. It was clarified that TGDC is not requiring audits, what it is requiring is that if a state requires an audit, the voting system must be capable of performing the audit.
  • From a usability stand point, we're saying that "the accompanying audit procedure for the system must be usable." The vendor must clearly expose the process they envision for an election official/jurisdiction to use to conduct an audit, and an assessment of the time required to conduct the audit.
  • We can not do benchmarks or design guidelines at this point.
  • What would make a system fail? Without design guidelines, how do you decide what fails a system?
  • What would make an audit process unacceptable to an election official? Whatever the process is, it can not degrade the ballots/information.
  • One problem with a recent election was that the print out record did not match the electronic record in the type of information that was provided. This concurrence would be a necessary requirement.
  • John Wack mentioned the possibility of including bar codes that contained specific records and also keeping a bitmap file of the ballot.
  • Whitney pointed out that we need to keep two things in mind: we should not write requirements that cause harm, and we need to make sure that a certified system is a good (usable) system.
  • John Cugini pointed out that the paper being discussed covered high level requirements and potential ways to meet them.
  • John Wack inquired if we should put a statement in about "more study necessary", and this is "a safe way to do it for now"? John Cugini and Whitney agreed that this is always a good idea to have further research.
  • Reminder this is a discussion paper about what we should include. It would be nice to have feedback from EAC and other election officials.
  • David Flater suggested looking at 1990 VSS requirements regarding paper ballot stock and degradation requirements for wording.
  • A requirement must be keeping a durable (non-degraded) image of the vote for 22 months, and must survive numerous trips through auditing/voting system.

ACTION: Write high level requirement that states systems should be auditable and vendor must document procedures. We should look at specifics, such as re-reading ballots. Vendors must provide process so test labs can perform necessary tests. We must document what makes an audit technically possible and usable. Write definition of durability for surviving 22 months as required for voting records.

Reactions to SI Approaches Paper:

John Kelsey was happy that the procedural defense was added to the paper and feels that this provided the SI requirement - this is the one where users without disabilities are asked to test the accessible systems for voters with disabilities. Whitney feels that it will not pass through the Holt Bill.

Whitney does not believe we have an existing accessible system that will receive high marks in the 3 columns of the evaluation process noted in the paper.

John Wack mentioned that David Wagner felt that with IDV systems, there were no accessible systems that were SI.

Ron was asked to clarify the definition of SI - errors are detectable in principle, either during voting stage by the voter, or later from evidence results of the audit. It does not require the voter paying attention to what's going on, i.e. it doesn't require that the audit be used but the auditing equipment should be trustworthy.

John Wack felt that we need verification for audits and forensics for recounts. For example, we would preserve a recorded record of the audio verification provided to blind voters.

David Baquis indicated the Access Board was in favor of having all voters be able to verify their votes at the time they vote. David felt that the audio recording of the verification could be played back for the voter as a confirmation that what they just heard was what was recorded for later auditing if necessary.

ACTION: Simplify table in paper and add an extra column. Review section 3.2 to implement/consider David Baquis' /Access Board comments. Ron Rivest will provide comments via email. Barbara pointed out that comments should include the section, whether its yes or no, and how to make it yes if feasible.

Next joint HFP and STS meeting, Thursday, March 1, 2007.


[* Pursuant to the Help America Vote Act of 2002, the TGDC is charged with directing NIST in performing voting systems research so that the TGDC can fulfill its role of recommending technical standards for voting equipment to the EAC. This teleconference discussion serves the purposes of the STS and HFP subcommittees of the TGDC to direct NIST staff and coordinate its voting-related research relevant to the VVSG 2007. Discussions on this telecon are preliminary and do not necessarily reflect the views of NIST or the TGDC.]

Teleconferences from 2004, 2005, 2006 and upcoming in 2006.


Link to NIST HAVA Page

Last updated: July 25, 2007
Point of Contact

Privacy policy / security notice / accessibility statement
Disclaimer / FOIA
NIST is an agency of the U.S. Commerce Department