Test Assertions for VVSG 1.1, Volume 2, Section 2.1, September 6, 2016

Requirement 2.1

NOTE: This section contains requirements on the content of the quality assurance and configuration management documentation that manufacturers must supply.

VVSG 1.1 Requirement 2.1:

1. All voting system manufacturers shall develop and present to the Certification Authority a complete Quality and Configuration Management Manual. The Manual shall detail the manufacturer's Quality and Configuration Management processes and procedures required by the VVSG. These processes and procedures shall conform to all requirements of the VVSG and the standards listed in Volume I Section 8.1.
2. The Manual shall declare that meeting the requirements of the entire VVSG is a binding commitment for the entire manufacturer organization. 
3. The Manual shall provide for the formulation of a project plan for the design and development of a voting system. It shall require the project plan to be clearly and unambiguously documented. The project plan should be consistent with the Design and Development Planning requirements, as specified in ISO 9001:2000, Quality management systems – Requirements Section 8.3.1. 
4. The Manual shall require the project plan to include, at a minimum, one quality check at the end of the design phase, and one quality check at the end of the development phase. The project plan shall define the progress that is required before each quality check can be passed. A "quality check" is the sum of the activities Design and Development Review, Design and Development Verification, and Design and Development Validation, as defined in ISO 9001:2000 Sections 7.3.4. through 7.3.6. 
5. The Manual shall require the manufacturer to maintain a log in which all difficulties encountered during the design and development phase for a voting system are required to be recorded. Any remedial action taken to correct a difficulty shall also be recorded. The log shall be available for inspection by the Certification Authority or the VSTL. "Difficulties" are any occasions when it is recognized that changes in past design decisions or in the project plan (see Requirement c) are necessary to complete the project. 
6. The Manual shall specify rules that define what parts, components, and assemblies of the voting system are to be considered as critical. As used here, "components" include, but are not limited to, software modules. A part, component, or assembly shall be defined as critical if its failure may:

1. Cause a faulty display of options;
2. Cause an uncertainty if voter's choice has been recorded;
3. Cause a false recording of vote cast;
4. Cause the change of stored votes;
5. Cause the false transmission for polling station totals;
6. Cause injury to voters or staff;
7. Provide an opening for tampering;
8. Violate a voter's privacy;
9. Cause a false accumulation of polling station totals;
10. Cause a false transmission for regional totals;
11. Give the appearance of irregularity;
12. Violate a voter's ability to vote independently; and
13.  Impede the usability of the polling station for all voters. 

1. The Manual shall require that the design and development process of a voting system produce statements for every part, component, and assembly, whether to be manufactured by the manufacturer or obtained elsewhere, that impacts conformity to the VVSG. These statements shall define verifiable requirements against which the part, component, or assembly can be tested at the end of its manufacturing process, or upon delivery, as appropriate. The requirements shall be defined in such a way that any part, component, or assembly that meets the requirements will provide the functionality and reliability required of it for the voting system to meet the overall functionality and reliability requirements specified in the VVSG. 
2. The Manual shall require that the design and development process define or identify processes by which all parts, components, and assemblies, defined as critical, of a voting system can be tested for compliance with requirements developed under Requirement g. 
3. The Manual shall require that the design and development process of a voting system produce a statement that defines verifiable requirements against which any voting system can be tested at the end of its manufacturing and assembly process in such a way that passing the test provides assurance that the voting system meets all requirements defined in the VVSG.
4. The Manual shall require that all purchased parts, components and assemblies, defined as critical, are tested according to the testing requirements developed under Requirement g and the processes developed under Requirement h before they are incorporated into a voting system. The records shall be maintained until such time as the certification of the voting system model expires or is revoked. 
5. The Manual shall require that all manufactured parts, components, and assemblies, defined as critical, are tested according to the testing requirements developed under Requirement g and the processes developed under Requirement h before they are incorporated into a voting system. The records shall be maintained until such time as the certification of the voting system model expires or is revoked. 
6. The Manual shall require that for each part, component, or assembly, whether purchased or manufactured by the manufacturer, that has been defined as critical (Requirement f), records shall be kept that document the complete history of the part, component, or assembly. These records shall be available for inspection. The records shall document:

1. The source of raw materials;
2. The processes used in the manufacture;
3. The time when critical manufacturing steps were taken;
4. The organization or person that performed each critical manufacturing step, and
5. The persons who performed the required inspections.
6. Any failures, discrepancies or anomalies that occurred during manufacture;
7. Any actions taken to correct the failure, discrepancy or anomaly; and
8. The final determination that the problem has been corrected.

1. The Manual shall require the manufacturer to identify and maintain the technical capability to monitor the in-service performance of each voting system sold throughout the life cycle of the voting system's model. For the purpose of this and subsequent requirements in this section, the term life cycle of a voting system model is defined as the time period from the delivery of the first voting system of that model to the time when the certification of the model expires or is revoked.
2. The Manual shall require the manufacturer to identify and maintain the technical capability to develop and implement remedies that are suitable to correct any defects that lead to in-service difficulties in all voting systems sold, throughout the life cycle of the voting system model. 
3. The Manual shall require the manufacturer to identify and maintain the financial capability to provide product support, as defined in Requirements m and n, throughout the life cycle of the voting system model.

Test Assertions

TA21a-1: Voting system manufacturers SHALL develop a complete Quality and Configuration Management Manual.

TA21a-2: The Quality and Configuration Management Manual SHALL detail the manufacturer's Quality and Configuration Management processes required by the VVSG.

TA21a-3: The Quality and Configuration Management Manual SHALL detail the manufacturer's Quality and Configuration Management procedures required by the VVSG.

TA21a-4: These processes and procedures SHALL conform to all requirements of the VVSG.

TA21a-4-1: Processes SHOULD discuss WHAT needs to be done and procedures SHOULD discuss HOW that is to be accomplished.

TA21a-5: These processes SHALL NOT conflict with any requirements of the VVSG.

TA21a-6: These procedures SHALL NOT conflict with any of the standards listed in Volume I Section 8.1.

TA21a-7: These processes SHALL NOT conflict with any of the standards listed in Volume I Section 8.1.

TA21b-1: The Quality and Configuration Management Manual SHALL state that meeting the requirements of the entire VVSG is a binding commitment for the entire manufacturer organization.

TA21c-1: The Quality and Configuration Management Manual SHALL describe, in detail, the development of a project plan for the design and development of a voting system.

TA21c-2: The project plan SHOULD be consistent with the Design and Development Planning requirements, as specified in ISO 9001:2000, Quality management systems – Requirements Section 8.3.1. 

TA21c-3: The Quality and Configuration Management Manual SHALL require the project plan to be clearly documented.

TA21c-4: The Quality and Configuration Management Manual SHALL require the project plan to be unambiguously documented.

TA21d-1: The Quality and Configuration Management Manual SHALL require the project plan to include, but not be limited to, one quality check at the end of the design phase, AND one quality check at the end of the development phase.

TA21d-2: The project plan SHALL define the progress that is required before each quality check can be passed. 

TA21e-1: The Quality and Configuration Management Manual SHALL require the manufacturer to maintain a log.

TA21e-1-1: This log SHALL record all difficulties encountered during the design and development phase of the voting system.

TA21e-1-2: This log SHALL record all remedial actions taken to correct each difficulty.

TA21e-1-3: The log SHALL be available for inspection by the Certification Authority OR inspection by the VSTL.

TA21f-1: The Quality and Configuration Management Manual SHALL specify rules that define what parts of the voting system are to be considered as critical.

TA21f-2: The Quality and Configuration Management Manual SHALL specify rules that define what components of the voting system are to be considered as critical.

TA21f-2-1: "Components" SHALL include, but are not limited to, software modules.

TA21f-3: The Quality and Configuration Management Manual SHALL specify rules that define what assemblies of the voting system are to be considered as critical.

TA21f-4: A part, component, or assembly SHALL be considered critical if its failure may lead to any of the following:

1. Cause a faulty display of options;
2. Cause an uncertainty if voter's choice has been recorded;
3. Cause a false recording of vote cast;
4. Cause the change of stored votes;
5. Cause the false transmission for polling station totals;
6. Cause injury to voters or staff;
7. Provide an opening for tampering;
8. Violate a voter's privacy;
9. Cause a false accumulation of polling station totals;
10. Cause a false transmission for regional totals;
11. Give the appearance of irregularity;
12. Violate a voter's ability to vote independently; and
13.  Impede the usability of the polling station for all voters. 

TA21g-1: The manufacturer SHALL develop a test plan.

TA21g-1-1: The test plan SHALL conform to the test recommendations in Notice of Clarification (NOC) 09-001: Clarification of the Requirements for Voting System Test Laboratories (VSTLs) Development and Submission of Test Plans (ref: http://www.eac.gov/assets/1/Page/Requirements%20for%20Test%20Lab%20Development%20and%20Submission%20of%20Test%20Plans.pdf).

TA21g-1-1-1: The manufacturer’s requirements MAY be derived from the test plan.

TA21g-2: The Quality and Configuration Management Manual SHALL require that the design and development process of a voting system produce statements for every part that impacts conformity to the VVSG.

TA21g-2-1: Statements SHALL be produced for parts manufactured by the manufacturer.

TA21g-2-2: Statements SHALL be produced for parts obtained elsewhere.

TA21g-2-3: These statements SHALL define verifiable requirements against which the part can be tested at the end of its manufacturing process, OR upon delivery, as appropriate.

TA21g-2-3-1: The requirements SHALL be defined in such a way that any part that meets the requirements will provide the functionality and reliability required of it for the voting system to meet the overall functionality and reliability requirements specified in the VVSG. 

TA21g-3: The Quality and Configuration Management Manual SHALL require that the design and development process of a voting system produce statements for every component that impacts conformity to the VVSG.

TA21g-3-1: Statements SHALL be produced for components manufactured by the manufacturer.

TA21g-3-2: Statements SHALL be produced for components obtained elsewhere.

TA21g-3-3: These statements SHALL define verifiable requirements against which the component can be tested at the end of its manufacturing process, OR upon delivery, as appropriate.

TA21g-3-3-1: The requirements SHALL be defined in such a way that any component that meets the requirements will provide the functionality and reliability required of it for the voting system to meet the overall functionality and reliability requirements specified in the VVSG. 

TA21g-4: The Quality and Configuration Management Manual SHALL require that the design and development process of a voting system produce statements for every assembly that impacts conformity to the VVSG.

TA21g-4-1: Statements SHALL be produced for assemblies manufactured by the manufacturer.

TA21g-4-2: Statements SHALL be produced for assemblies obtained elsewhere.

TA21g-4-3: These statements SHALL define verifiable requirements against which the assembly can be tested at the end of its manufacturing process, OR upon delivery, as appropriate.

TA21g-4-3-1: The requirements SHALL be defined in such a way that any assembly that meets the requirements will provide the functionality and reliability required of it for the voting system to meet the overall functionality and reliability requirements specified in the VVSG. 

TA21h-1: The Quality and Configuration Management Manual SHALL require that the       design and development process define OR that the design and development process identify one or more processes, by which all parts of a voting system that are defined as            critical, can be tested for compliance with requirements developed under Requirement g. 

TA21h-2: The Quality and Configuration Management Manual SHALL require that the design and development process define OR that the design and development process identify one or more processes, by which all components of a voting system that are defined as critical, can be tested for compliance with requirements developed under Requirement g. 

TA21h-3: The Quality and Configuration Management Manual SHALL require that the design and development process define OR that the design and development process identify one or more processes, by which all assemblies of a voting system that are defined as critical, can be tested for compliance with requirements developed under Requirement g. 

TA21i-1: The Quality and Configuration Management Manual SHALL require that the design and development process of a voting system produce a statement that defines all verifiable requirements against which the voting system can be tested at the end of its manufacturing and assembly process

TA2li-1-1: The design and development process that produces the statement that defines verifiable requirements SHOULD include:

1. Going through all VVSG requirements.
2. Selecting those requirements that apply to the given voting system.
3. For each applicable requirement, ensuring that proper options/features have been selected.

TA2j-1: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that all purchased system elements were categorized as one of the following: “part”, “component”, or “assembly”.

TA2j-2: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that all such purchased system parts, components, and assemblies were tested.

TA2j-2-1: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that those tests were performed according to the testing requirements developed under Requirement 2.1g.

TA2j-2-2: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that those tests were performed according to the testing processes developed under Requirement 2.1h.

TA2j-2-3: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that those tests were carried out on a component basis (i.e., prior to their assembly of the individual system elements into full voting system form).

TA2j-2-4: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that records of those tests are being maintained until the voting system model’s certification expires or is revoked.

TA2lj-3: The manufacturer MAY provide this information in the form of a spreadsheet (e.g., matrix or checklist form).

TA2k-1: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that all manufactured system elements were categorized as one of the following: “part”, “component”, or “assembly”.

TA2k-2: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that all such manufactured system parts, components, and assemblies were tested.

TA2k-2-1: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that those tests were performed according to the testing requirements developed under Requirement 2.1g.

TA2k-2-2: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that those tests were performed according to the testing processes developed under Requirement 2.1h.

TA2k-2-3: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that those tests were carried out on a component basis (i.e., prior to their assembly of the individual system elements into full voting system form).

TA2k-2-4: The manufacturer SHOULD provide information/evidence in the Quality and Configuration Management Manual that records of those tests are being maintained until the voting system model’s certification expires or is revoked.

TA2lk-3: The manufacturer MAY provide this information in the form of a spreadsheet (e.g., matrix or checklist form).

TA21l-1: The Quality and Configuration Management Manual SHALL require that for each part, purchased by the manufacturer, that has been defined as critical (Requirement f), records shall be kept that document the complete history of the part.

TA21l-2: The Quality and Configuration Management Manual SHALL require that for each component, purchased by the manufacturer, that has been defined as critical (Requirement f), records shall be kept that document the complete history of the component.

TA21l-3: The Quality and Configuration Management Manual SHALL require that for each assembly, purchased by the manufacturer, that has been defined as critical (Requirement f), records shall be kept that document the complete history of the assembly.

TA21l-4: The Quality and Configuration Management Manual SHALL require that for each part, manufactured by the manufacturer, that has been defined as critical (Requirement f), records shall be kept that document the complete history of the part.

TA21l-5: The Quality and Configuration Management Manual SHALL require that for each component, manufactured by the manufacturer, that has been defined as critical (Requirement f), records shall be kept that document the complete history of the component.

TA21l-6: The Quality and Configuration Management Manual SHALL require that for each assembly, manufactured by the manufacturer, that has been defined as critical (Requirement f), records shall be kept that document the complete history of the assembly.

TA21l-7: The records, described in TA21l-1–TA21l-6 SHALL be available for inspection.

TA21li-1: The records, described in TA21l-1–TA21l-6, SHALL document the source of raw materials;

TA21lii-1: The records, described in TA21l-1–TA21l-6, SHALL document the processes used in the manufacture;

TA21liii-1: The records, described in TA21l-1–TA21l-6, SHALL document the time when critical manufacturing steps were taken;

TA21liv-1: The records, described in TA21l-1–TA21l-6, SHALL document the organization that performed each critical manufacturing step;

TA21liv-2: The records, described in TA21l-1–TA21l-6, SHALL document the person(s) that performed each critical manufacturing step;

TA21lv-1: The records, described in TA21l-1–TA21l-6, SHALL document the persons who performed the required inspections;

TA21lvi-1: The records, described in TA21l-1–TA21l-6, SHALL document all failures that occurred during manufacture;

TA21lvi-2: The records, described in TA21l-1–TA21l-6, SHALL document all discrepancies that occurred during manufacture;

TA21lvi-3: The records, described in TA21l-1–TA21l-6, SHALL document all anomalies that occurred during manufacture;

TA21lvii-1: The records, described in TA21l-1–TA21l-6, SHALL document all actions taken to correct the failure;

TA21lvii-2: The records, described in TA21l-1–TA21l-6, SHALL document all actions taken to correct the discrepancy;

TA21lvii-3: The records, described in TA21l-1–TA21l-6, SHALL document all actions taken to correct the anomaly;

TA21lviii-1: The records, described in TA21l-1–TA21l-6, SHALL document the final determination that the problem has been corrected.

TA21m-1: The Quality and Configuration Management Manual SHALL require the manufacturer to identify the technical capability to monitor the in-service performance of each voting system sold throughout the life cycle of the voting system's model.

TA21m-1-1: The technical capability SHOULD include both the people and the processes, and tools necessary to monitor the in-service performance of the voting system throughout its life-cycle.

TA21m-2: The Quality and Configuration Management Manual SHALL require the manufacturer to maintain the technical capability to monitor the in-service performance of each voting system sold throughout the life cycle of the voting system's model.

TA21n-1: The Quality and Configuration Management Manual SHALL require the manufacturer to identify the technical capability to develop remedies that are suitable to correct any defects that lead to in-service difficulties in all voting systems sold, throughout the life cycle of the voting system model. 

TA21n-1-1: The technical capability SHOULD include both the people and the processes, and tools necessary to develop and implement remedies that are suitable to correct defects during the in-service performance of the voting system throughout its life-cycle.

TA21n-2: The Quality and Configuration Management Manual SHALL require the manufacturer to maintain the technical capability to develop remedies that are suitable to correct any defects that lead to in-service difficulties in all voting systems sold, throughout the life cycle of the voting system model.

TA21n-3: The Quality and Configuration Management Manual SHALL require the manufacturer to identify the technical capability to implement remedies that are suitable to correct any defects that lead to in-service difficulties in all voting systems sold, throughout the life cycle of the voting system model. 

TA21n-4: The Quality and Configuration Management Manual SHALL require the manufacturer to maintain the technical capability to implement remedies that are suitable to correct any defects that lead to in-service difficulties in all voting systems sold, throughout the life cycle of the voting system model.

TA21o-1: The Quality and Configuration Management Manual SHALL require the manufacturer to identify the financial capability to provide product support, as defined in Requirement m, throughout the life cycle of the voting system model.

TA21o-2: The Quality and Configuration Management Manual SHALL require the manufacturer to identify the financial capability to provide product support, as defined in Requirements n, throughout the life cycle of the voting system model.

TA21o-3: The Quality and Configuration Management Manual SHALL require the manufacturer to maintain the financial capability to provide product support, as defined in Requirement m, throughout the life cycle of the voting system model.

TA21o-4: The Quality and Configuration Management Manual SHALL require the manufacturer to maintain the financial capability to provide product support, as defined in Requirement n, throughout the life cycle of the voting system model.

TA21o-5: The financial capability SHOULD include the proper resources including appropriate funding and personnel necessary to provide product support during the in-service performance of the voting system throughout its life-cycle.

Operational Definitions

Critical – A part, component, or assembly shall be defined as critical if its failure may:

1. Cause a faulty display of options;
2. Cause an uncertainty if voter's choice has been recorded;
3. Cause a false recording of vote cast;
4. Cause the change of stored votes;
5. Cause the false transmission for polling station totals;
6. Cause injury to voters or staff;
7. Provide an opening for tampering;
8. Violate a voter's privacy;
9. Cause a false accumulation of polling station totals;
10. Cause a false transmission for regional totals;
11. Give the appearance of irregularity;
12. Violate a voter's ability to vote independently; and
13.  Impede the usability of the polling station for all voters. 

(ref: Requirement 2.1f, Volume 2, VVSG 1.1)

Difficulties – "Difficulties" are any occasions when it is recognized that changes in past design decisions or in the project plan (see Requirement c) are necessary to complete the project.  (ref: Requirement 2.1e, Volume 2, VVSG 1.1)

Life Cycle – The “life cycle” of a voting system model is defined as the time period from the delivery of the first voting system of that model to the time when the certification of the model expires or is revoked.

Process – A process defines “what” needs to be done and which roles are involved. (ref: http://sei.cmu.edu/library/assets/process-pro.pdf)

Procedure – A procedure defines “how” to do the task and usually only applies to a single role. (ref: http://sei.cmu.edu/library/assets/process-pro.pdf)

Quality Check – A "quality check" is the sum of the activities Design and Development Review, Design and Development Verification, and Design and Development Validation, as defined in ISO 9001:2000 Sections 7.3.4. through 7.3.6. (ref: Requirement 2.1d, Volume 2, VVSG 1.1)