Conformance Testing 101

Test Development for Voting Systems

Conformance Testing 101

Conformance testing captures the technical description of a specification and measures whether an implementation (i.e., software product or system) faithfully implements the specification. The goal of conformance testing is to provide a level of assurance that the requirements imposed by a specification are being met by implementations claiming conformance to that specification. 

Conformance testing is bound in scope by the specification - i.e., if a requirement or behavior is not defined in the specification, there can not be a conformance test for that requirement or behavior. Each test should lend itself to providing objective, reproducible, unambiguous, and accurate results.

Conformance test suites are designed by carefully choosing different input values, trying to design test cases that will invoke every functional requirement in the specification, at least once. Subsequent test cases may be designed to test combinations of requirements. Due to the combinatorial explosion of possible input values as well as the number of combinations of requirements, exhaustive testing is impossible. The objective is to produce test for as many of the requirements in the specification that are feasible and use these test to find errors in the implementation. The outcome of a test is either a pass or a fail result. If the implementation fails a test, then it is not a conforming implementation. However, passing all the tests is no guarantee that an implementation really does conform, since it might well violate the specification in untested areas. This approach, called falsification testing, can prove with certainty that an implementation does not conform. The converse is not true. Nonetheless, it can give one a level of confidence that an implementation has the required capabilities and that they are implemented correctly.