Conformance Resources and Information

Conformance Resources and Information

Conformance is usually defined as a way to determine if an implementation faithfully meets the requirements of a standard or specification. Determining whether a product faithfully implements a standard is essential to creating robust, interoperable solutions. The NIST/ITL Conformance Pages provide an online reference to documents, presentations, test suites related to conformance, conformance issues, and conformance-related topics.

General Conformance

• QA Framework:Specification Guidelines
  W3C Recommendation on writing better specifications. (This link takes you off the NIST site) [Aug 2005].
  
  
• Principles for writing conformance requirements [PDF]
  List of principles to help craft conformance requirements (aka test assertions) [Rosenthal, Lyon; Feb 2005].
  
  
• What is this thing called conformance?
  An overview of conformance including a discussion of terminology, methodology, and benefits. Additionally, various test suites for XML technologies are presented. [Rosenthal, Brady; NIST/ITL Bulletin, January 2001].
  
  
• Presentation: What is this thing called conformance? [ppt] Presentation on conformance including what it is, how it works, and what the benefits are. Additionally specific examples from available conformance test suites (XML, DOM, XSLT) are presented. [Rosenthal, Brady, XML 2000 Conference, December 2000].
  
  
• Conformance [ppt]
  Presentation given at the ebXML Tokyo meeting, November 2000. An overview of conformance is given, including the topics and issues that need to be addressed when including a conformance clause in a specification. [Skall, Rosenthal, November 2000].
  
  
• Conformance Testing
  With any standard, eventually the discussion turns to "how will we know if an implementation conforms to the standard?" This paper defines conformance and conformance testing as well as describes the components of a conformance testing program. [Gray, Goldfine, Rosenthal, Carnahan].
  
  
• Overview of Conformance Testing
  Discussion of conformance terms, assessment program, types of testing, and factors for success. [Gray].

Validation and Certification

Many of the documents and presentations in the General Conformance section include a discussion of validation and certification programs

• Conformance Testing and Certification Framework [pdf]
  This paper presents general concepts, components, and issues related to establishing and administering a conformance testing program. Although it addresses testing for ebXML specifications, it can be applied to testing any specifications. [Rosenthal, Skall, Carnahan, White Paper for ebXML, April 2001].
  
  
• Conformance Testing and Certification Model for Software Specifications
  This paper identifies the key roles, activities, and products involved in creating conformance testing and certification programs. [Carnahan, Rosenthal, Skall, ISACC '98 Conference, March 1998].

Test Methodologies

• DOM Test Suite Methodology [pdf]
  This paper provides a description of the Document Object Model (DOM) test suite methodology for how conformance tests to the W3C DOM specification was developed using automated test generation techniques. [NIST GCR04-860, 2004].
  
  
• Interactive Conformance Testing for VRML
  Paper describing the test development strategy and design issues in developing the NIST Virtual Reality Modeling Language (VRML) conformance test suite. [Brady, Rosenthal].
  
  
• Conformance Testing of Object-Oriented Components specified by State/Transition Classes [pdf]
  This paper provides a conformance testing technique for new components to be integrated into an existing software system. The approach relies on finate state machines, database modeling and processing techniques, and algorithms for analysis and traversal of directed graphs. [Gallagher, 2000].
  
  
• Formal Specification Languages in Conformance Testing
  Report assessing the potential the advantages of applying formal specification languages rather than natural language, to the problems of conformance testing.[Liburdy, Gray, Rosenthal, Software Quality Week, 1998]. (currently not available)

Test Suites

• XML - A Conformance Test Suite, jointly developed by NIST and W3C for the W3C XML and Namespace Recommendations.
  
  
• DOM Conformance Test Suites, developed by NIST and W3C, for DOM Level 1 and Level 2, CORE and HTML modules.
  
  
• XSL - Conformance Test Suites, developed by NIST, for both Formatting Objects (XSL-FO) and XSL Transformations (XSLT/XPath).
  
  
• XML Schema - Conformance Test Suites, developed by NIST, for XML Schema Part 2: Datatypes.
  
  
• XML Query - Conformance Tests, developed by NIST, for Functions and Operators, Working Draft.
  
  
• ebXML - Test Framework and Conformance Tests for ebXML Messaging.