What is this thing called Conformance?

What is this thing called Conformance?

Introduction

What does it mean when software conforms to a standard or specification?  How do you know that the software conforms? And just as important, why is this important?

According to a study by the Gartner Group[1], the private sector spends over $2.2 trillion on information technology (IT) annually.  Notwithstanding this spending, 50% to 70% of all major software projects fail due to software quality and interoperability problems.  Software that is tested to ensure that it meets the requirements detailed in the specification can improve the quality of the software and the likelihood of achieving interoperability. This type of testing is called conformance testing.  Determining whether a product faithfully implements a standard or specification is essential to creating robust, interoperable solutions.  This bulletin presents an overview of conformance and addresses conformance to the World Wide Web Consortium (W3C) Recommendations for the Extensible Markup Language (XML) family of specifications. 

Unscrambling the Terminology

ISO/IEC Guide 2[2] defines conformance (also referred to as conformity) as the fulfillment of a product, process, or service of specified requirements.  These requirements are specified in a standard or specification as either part of a conformance clause or in the body of the specification. A conformance clause is a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance to the specification.  Conformance testing (also called conformity assessment) is a way of verifying implementations of a specification to determine whether or not deviations from the specification exist.  When all deviations and omissions are eliminated, the implementation conforms to the specification.  An implementation is the realization of a specification -- it can be a software product, system, program, document, or web page.  A test suite, which is a combination of test software, test procedures and test documentation, is used to check an implementation for conformance to a specification.  The test software, consisting of a set of test files (i.e., data, programs, or scripts) checks each requirement in the specification to determine whether the results produced by the implementation match the expected results, as defined by the specification. The test procedures define the administrative as well as the technical process for testing an implementation.  The test documentation describes how the testing is to be done and the directions for the tester to follow.  Additionally, the documentation should be detailed enough so that testing of a given implementation can be repeated with no change in test results. 

Other conformance related terms include validation and certification.  Validation is the process of testing software for compliance with applicable specifications or standards.  Although the terms validation and conformance testing are often interchanged, they are not the same. The validation process consists of the steps necessary to perform the conformance testing by using an official test suite in a prescribed manner.  Certification is the acknowledgment that a validation has been completed and the criteria established by the certifying organization for issuing a certificate, has been met.  Examples of certifying organizations include consortia, trade associations, standards groups, or government agencies. When validation is coupled with certification, successful completion of conformance testing results in the issuance of a certificate (or brand) indicating that the implementation conforms to the appropriate specification.  Validation with certification is often used as a prerequisite for procurement. It is important to note that certification cannot exist without validation, but validation can exist without certification. Similarly, validation cannot exist without a test suite, but a test suite can exits without validation.

Degree of Formality

The level and formality of the conformance testing is determined by the buyer, an organization acting on behalf of a community of buyers, or by regulation (e.g., safety, health, national security concerns).  Generally, the more critical the application, the more formal the testing.  For example, some testing programs may require a very formal testing and certification approach using an independent (i.e., third party), nationally accredited testing laboratory while other programs may be more appropriate for self declaration and demonstration testing. Typically, the decision to establish a testing program is based on weighing the benefits achieved by obtaining conformance versus the costs of creating and running a validation and certification program.

More recently with the advent of the Internet and rapidly changing information technology, the notion of conformance testing has become less formal allowing developers and users to assess for themselves the validity or correctness of the software or data with respect to the relevant specification. They can then make any necessary corrections to allow the implementation to conform. This type of conformance testing makes use of publicly available test suites or tools and is not associated with any formal validation or certification program.

Benefits of conformance testing

Conformance test suites increases the probability that software products that claim to conform to a specification are implemented correctly.  Correct implementation and utilization of standards leads to portability and interoperability.  Portability is the ability to move software programs or applications among different computer systems with minimal change.  Interoperability is defined as the capability of two or more systems to exchange and use information.  Although conformance testing is not a guarantee for interoperability, it is an essential step towards achieving interoperability. Conformance testing provides software developers and users assurance and confidence that the conforming product behaves as expected, performs functions in a known manner, or possesses a prescribed interface or format.  

In order to test that a specification is faithfully adhered to, one must carefully examine the specification and compare the implementation's result against the syntax and semantics contained in the specification.  Often times, errors or ambiguities in the specification are discovered.  These errors and ambiguities can then be fed back to the standards committee who developed the specification, for correction.  For software developers, using conformance test suites early-on in the development process can help improve the quality of their implementations by identifying areas in which they conform as well as those areas in which they do not.   In the marketplace, conformance test suites and consequently conformance testing provides a vehicle for exchanging information between buyer and seller.  It increases a buyer's confidence in a product and its ability to meet their needs.  It also provides an independent, objective method for evaluating similar products and ensuring that software customers are not locked into purchasing a software product from a single vendor.

Testing Methodologies

Depending on the type of specification (e.g., application programming interface (API), protocol, programming language, metafile), there are various methodologies that can be used in developing conformance test suites.  However, these methodologies all follow the same basic strategy - falsification testing; and the same basic process - analyze the specification for test assertions or test requirements, write a test purpose for each test, and create and document a test case which realizes each test purpose.

Ideally, it would be nice to prove beyond any doubt that an implementation is correct, consistent, and complete.  However, this is generally impossible for implementations of nontrivial specifications.  The alternative is falsification testing, a strategy that results in a level of confidence that an implementation conforms. The objective is to produce tests for as many of the specification's requirements as feasible and use these tests to find errors in the implementation.  Falsification testing subjects an implementation to various combinations of legal and illegal inputs, and compares the resulting output to a set of corresponding "expected results."  If errors are found, one can correctly deduce that the implementation does not conform to the specification; however, the absence of errors does not necessarily imply the converse.  Falsification testing can only demonstrate non-conformance.  Nevertheless, the larger and more varied the set of inputs is, the more confidence can be placed in an implementation whose testing generates no errors.

The process for developing conformance test suites begins with defining the set of test assertions (TA) upon which the test suite is based.  Each TA should be as simple as possible and narrowly focused on an atomic (simple) functionality, must not contradict another TA, and must have some testable consequence.   Using this set of TAs, a set of tests that will exercise each of the requirements is developed.  Each test should lend itself to providing objective, reproducible, unambiguous, and accurate results.  Each test should be traceable back to a statement or statements in the specification.  Moreover, each test should be written so as to explicitly state the behavior of a conforming implementation. Thus, the tests when executed on the implementation under test will generate a pass/fail condition.  Failure to pass a test implies failure to conform.

Conformance Test Suites for XML Technologies

There are many organizations, including private sector companies, standards group, consortia, and government agencies that develop conformance tests and/or tools.  Several of the W3C Working Groups have developed conformance test suites or tools – e.g., SVG, CSS, and HTML.  The National Institute of Standards and Technology (NIST) and the Organization for the Advancement of Structured Information Standards (OASIS) are addressing XML-related interoperability issues by jointly developing test suites that are intended to complement the W3C Recommendations. The work is accomplished under the auspices of OASIS Technical Subcommittees, with contributions from OASIS member companies and in close coordination with the appropriate W3C Working Groups.  Three of the test suites that are being developed are:

XML Test Suite - The XML test suite for XML processors contains over 2000 test files and an associated test report. Accompanying the actual test files is an XML test description file that contains pertinent information for each of the tests included in the distribution.  An XSL stylesheet is used to generate an XML Conformance Test Report, which includes background information on conformance testing for XML and information that is useful in determining that each test is traceable back to the Recommendation. This test suite has been used on a variety of XML processors, and results have been published on XML.com.  XML Test Suite is available from the NIST web site, http://xw2k.nist.gov/xml/index.html

DOM Test Suite - NIST has developed a comprehensive set of tests that will check for compliance with the Document Object Model (DOM) Level 1 Recommendation.  Two sets of test suites were built, one for DOM with an ECMAScript binding and the other with a Java™ binding.  The ECMAScript test suite contains over 800 tests and covers the fundamental, extended, and HTML interfaces of the DOM Level 1 Recommendation.  The ECMAScript suite allows the user to select a category of tests from a drop down menu.  The tests are then executed and the results displayed.  The Java™ test suite contains over 200 tests and covers the fundamental and extended interfaces of the DOM Level 1 Recommendation.  They can also be selected from a drop down menu, executed and results displayed.  Alternatively, the Java™ test suite can be downloaded and run locally.

The test suites are available at http://xw2k.nist.gov/xml/index.html 

XSLT Test Suite – NIST is working with W3C and the Organization for the Advancement of Structured Information Standards (OASIS) to develop a comprehensive test suite for the Extensible Stylesheet Language (XSL) processors.  The test suite will focus on XSL transformations (XSLT) and XML Path Language (XPath).  The XSLT test suite will allow users to customize the test suite to filter out optional category tests. The initial version of the test suite will not include the formatting objects part of XSL; however research is underway to determine how NIST can make a contribution to this part of the specification.  The final test suite is expected to be complete by March 2001.

 Additional information about these test suites and the W3C Recommendations are available from:

NIST – http://xw2k.nist.gov/xml/index.html

OASIS – http://www.oasis-open.org/committees

W3C – http://www.w3c.org

Disclaimer

Any mention of commercial products or reference to commercial organizations is for information only; it does not imply recommendation or endorsement by NIST nor does it imply that the products mentioned are necessarily the best available for the purpose.

™Java and all Java based marks are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States and other countries.

[1] Gartner Group, a Stamford, CT based consulting company

[2] ISO/IEC Guide 2:1996, Standardization and Related Activities – General Vocabulary