Conformance Testing

Conformance Testing

Martha Gray, Alan Goldfine, Lynne Rosenthal, Lisa Carnahan
National Institute of Standards and Technology

With any standard or specification, eventually the discussion turns to "how will we know if an implementation or application conforms to our standard or specification?" The following discussion defines conformance and conformance testing as well as describes the components of a conformance testing program.

1. Definitions - what is conformance (to a standard)?

Conformance is usually defined as testing to see if an implementation faithfully meets the requirements of a standard or specification. There are many types of testing including testing for performance, robustness, behavior, functions and interoperability. Although conformance testing may include some of these kinds of tests, it has one fundamental difference -- the requirements or criteria for conformance must be specified in the standard or specification. This is usually in a conformance clause or conformance statement, but sometimes some of the criteria can be found in the body of the specification. Some standards have subsequent documentation for the test methodology and assertions to be tested. If the criteria or requirements for conformance are not specified, there can be no conformance testing.

The general definition for conformance has changed over time and been refined for specific standards. In 1991, ISO/IEC DIS 10641 defined conformance testing as "test to evaluate the adherence or non-adherence of a candidate implementation to a standard." ISO/IEC TR 13233 defined conformance and conformity as "fulfillment by a product, process or service of all relevant specified conformance requirements." In recent years, the term conformity has gained international use and has generally replaced the term conformance in ISO documents.

In 1996 ISO/IEC Guide 2 defined the three major terms used in this field.

• conformity - fulfillment of a product, process or service of specified requirements
• conformity assessment - any activity concerned with determining directly or indirectly that relevant requirements are fulfilled.
• conformity testing - conformity assessment by means of testing.

ISO/IEC Guide 2 also mentions that "Typical examples of conformity assessment activities are sampling, testing and inspection; evaluation, verification and assurance of conformity (supplier's declaration, certification); registration, accreditation and approval as well as their combinations."

Conformance tests should be used by implementers early-on in the development process, to improve the quality of their implementations and by industry associations wishing to administer a testing and certification program. Conformance tests are meant to provide the users of conforming products some assurance or confidence that the product behaves as expected, performs functions in a known manner, or has an interface or format that is known. Conformance testing is NOT a way to judge if one product is better than another. It is a neutral mechanism to judge a product against the criteria of a standard or specification.

2. Testing for conformance?

Ideally, we would like to be able to prove beyond any doubt that an implementation is correct, consistent, and complete with respect to its specification. However, this is generally impossible for implementations of nontrivial specifications that are written in a natural language.

The alternative is falsification testing, which subjects an implementation to various combinations of legal and illegal inputs, and compares the resulting output to a set of corresponding "expected results." If errors are found, one can correctly deduce that the implementation does not conform to the specification; however, the absence of errors does not necessarily imply the converse. Falsification testing can only demonstrate non-conformance. Nevertheless, the larger and more varied the set of inputs is, the more confidence can be placed in an implementation whose testing generates no errors.

3. Conformance clause and specification wording

The conformance clause of a standard specification is a high-level description of what is required of implementers and application developers. It, in turn, refers to other parts of the standard. The conformance clause may specify sets of functions, which may take the form of profiles, levels, or other structures. The conformance clause may specify minimal requirements for certain functions and minimal requirements for implementation-dependent values. Additionally it may specify the permissibility of extensions, options, and alternative approaches and how they are to be handled.

3.1 Profiles and/or levels

Applications often do not require all the features within a standard. It is also possible that implementations may not be able to implement all the features. In these cases, it may be desirable to partition the specifications into subsets of functionality.

A profile is a subset of the overall specifications that includes all of the functionality necessary to satisfy the requirements of a particular community of users.

Levels are nested subsets of the specifications. Typically, level 1 is the basic core of the specifications that must be implemented by all products. Level 2 includes all of level 1 and also additional functionality. This nesting continues until level n, which consists of the entire specification.

It is possible for a standard to have both profiles and levels defined for the entire specification (i.e., an implementer could choose to implement any one of (number of profiles) x (number of levels) subsets). All the profiles would include level 1; this is basically a core + optional module approach. Alternatively, each profile could have its own set of levels.

3.2 Extensions to the specifications

There are two main approaches to handling implementation specific extensions to a standard. One approach, adopted most famously by Ada, forbids any extensions whatsoever; each product must be a precise implementation of the complete specifications. This is called strict conformance.

The other approach adopts the weaker overall requirement that, while extensions are allowed, an implementation must perform all the functionality in the specifications exactly as specified. This more common approach usually includes some additional, more specific, requirements in the conformance clause, along the lines of:

• Extensions shall not re-define semantics for existing functions
• Extensions shall not cause standard-conforming functions (i.e., functions that do not use the extensions) to execute incorrectly
• The mechanism for determining application conformance and the extensions shall be clearly described in the documentation, and the extensions shall be marked as such
• Extensions shall follow the principles and guidelines of the standard they extend, that is, the specifications must be extended in a standard manner.

One approach that has been used successfully is to have a register for extensions. This is a document, parallel to but distinct from the official specifications, that contains a list of recognized extensions to the standard. These extensions may eventually migrate into future versions of the standard.

3.3 Options

A standard may classify features as "mandatory" vs. "optional," and provide a table listing features so classified. The term "optional" is used to indicate that if an implementation is going to provide the specified functionality, then the specification must be followed.

3.4 Implementer defined values

Specifications sometimes need to address:

• Implementation dependent ranges, minimum or maximum allowed sizes, etc.
• Values that may be different for different conforming implementations of the standard
• The color model(s), if any, supported by the standard
• Features reserved for registration.

3.5 Alternate approaches

Specifications may describe several different ways to accomplish its operation (e.g., a choice of file formats or codes). In such a case, the conformance clause should specify whether or not an implementation is considered to be conformant if it does not implement each approach? (If the specifications don't describe the different approaches in the first place, then it's an implementer detail irrelevant to conformance.)

For ensure testability of a specification/standard, care should be taken in the wording of the specifications itself. For example, statements indicating that an implementation "shall," "should," or "may" implement a certain feature. The meaning of the words "shall," "should," and "may," in the context of the given standard, could be defined in the conformance clause or be as defined by organizations such as ISO.

Some recent standards include, as official parts of the standard, lists of assertions. These assertions are statements of functionality or behavior derived from the standard, and which are true for conforming implementations. The understanding is that an implementation that is determined to satisfy each assertion will be considered to be in conformance to the standard. Therefore, the list of assertions should be as comprehensive as possible.

4. Conformance Testing Program

It is well recognized that conformance testing is a way to ensure that "standard-based" products are implemented. The advantages afforded by testing (and certification) are fairly obvious: quality products, interoperability, and competitive markets with more choices.

Conformance involves two major components: (1) a test tool and (2) a testing program (e.g., certification or branding). A testing program cannot exist without a test tool or test suite, but a test tool/suite can exist without a testing program. Not all specifications or standards need a testing program. Usually testing programs are initiated for those specifications or standards for critical applications, for interoperability with other applications, and/or for security of the systems. The decision to establish a program is based on the risk of nonconformance versus the costs of creating and running a program.

A conformance testing program usually includes:

• Standard or specification
• Test tool (e.g., tool, suite, and/or reference implementation
• Procedures for testing
• Organization(s) to do testing, issue certificates of validation, and arbitrate disputes

A specification/standard that includes a conformance clause and a test tools are essential to defining and measuring conformance.

Generically, a conformance test suite is a collection of combinations of legal and illegal inputs to the implementation being tested, together with a corresponding collection of "expected results." If such a list is provided, the starting point for the development of the test suite is the list of assertions for the standard. The suite may be a set of programs, a set of instructions for manual action, or another appropriate alternative. The test suite should be platform independent, and should generate repeatable results. Development of the test suite may well be the costliest part of the conformance program.

A reference implementation is an implementation of a standard that is by definition conformant to that standard. Such an implementation provides a proof of concept of the standard and also provides a tool for the developers of the conformance test suite (by generating expected values, testing the test suite, etc.) A reference implementation has maximum value in the early stages of a conformance program.

The conformance testing policies and procedures should be spelled out before testing begins. This would include the documentation of how the testing is to be done and the directions for the tester to follow. Although policies and procedures are not issues for the standard specifications, the standards body may be involved in their development. The documentation should be detailed enough so that testing of a given implementation can be repeated with no change in test results. The procedures should also contain information on what must be done operationally when failures occur. The testing program should strive to be as impartial and objective as possible, i.e., to remove subjectivity of both the procedures and the testing tools.

Finally, the testing policy and procedure should identify and define the actions of the organization(s) responsible for conducting the tests, handling disputes, and issuing certificates (or brands) of validation.

Who does the testing? Some standards have no official testing organizations. They rely on self-assessment by the implementer (1st party testing) and acceptance testing by buyers. Others have qualified third party testing laboratories that apply the test suite according to the established policies and procedures. A testing laboratory can be an organization or individual, and can either be accredited from a formal accreditation organization such as NIST's National Voluntary Laboratory Accreditation Program (NVLAP) or recognized by the buyer, seller, and certificate issuer, as qualified to perform the testing.

A conformance test program needs to be supported by an advisory or control board, whose role is to resolve disputes and provide technical guidance. This board should be a body of impartial experts. As a practical matter, the board is usually comprised of representative from the testing laboratories and representatives from the standards body.

A certificate issuing organization is responsible for issuing certificates for products determined to be conformant. The decision to issue a certificate is based on the testing results and established criteria for issuing certificates. These criteria may or may not require implementations to pass 100% of the specifications. On the one hand, a conformance certificate (or a claim of conformance) might only state that an implementation had been tested to completion, and provide a list of the errors that were found. It would then be up to a purchaser to decide the criteria (how many or what kinds of errors) it wishes to use to make implementations eligible for purchase. On the other hand, the policy might be that a certificate is issued (or a claim of conformance is made) only if no errors are found. Often a hybrid of these is appropriate -- i.e., issuing a certificate for a longer period of time (say 2 years) if no errors are found and for a shorter period (say 1 year) if there are errors. At the end of the 1-year period, the implementer would have to correct the errors to renew the certificate. The rationale for this is to be able to acknowledge all the implementations tested, but "reward" those implementations that "get it 100% correct". Another issue it whether or not a certificate expires (e.g., good for 2 years or never expire). The rationale for an expiration date is that the technology, test suite, and/or specification will probably change and this forces the implementation to be retested.

While actual testing (1st or 3rd party testing) and certification (branding) can be carried out by various organizations, it is essential that there be a centralized sponsor or owner of the conformance testing program. The sponsor has a fundamental interest in ensuring the success of the program. Typically, the sponsor establishes and maintains the program. It assumes responsibility for insuring that the components of the program are in place and becomes the centralized source for information about the program.

--------------------------------------------------------------------------------

Lisa J. Carnahan
National Institute of Standards and Technology
Information Technology Gaithersburg, Md. 20899
USA

Email: lisa.carnahan [at] nist.gov (lisa[dot]carnahan[at]nist[dot]gov)
Tel: (301) 975-3362
FAX: (301) 948-6213

--------------------------------------------------------------------------------

Prepared by Robin Cover for the The SGML/XML Web Page archive. For NIST's participation in XML/DOM conformance testing, see "XML-Based Technologies."

--------------------------------------------------------------------------------