The Standards and Testing Landscape

Figure 1 illustrates the relationships between standards, conformance, implementation, validation, and assessment. As an example, we describe the aspects of each component in the context of an HL7 v2 implementation guide. However, the concepts apply broadly.

Point 1 and 2: Within a standard or Implementation Guide, a Conformance Clause defines the essential, or 'normative', requirements that systems must meet. This clause details which conformance profiles, such as the ability to create and send laboratory results messages, or specific profile components are necessary for implementation. Some profiles (set of requirements) are designated as required, while others are optional. Furthermore, the Conformance Clause can specify varying 'levels of support' that systems may provide.

Point 2: The requirements specified in the Standard and Conformance Clause determines the Validation Method of the expected testing scope.

Points 3 and 5: Implementers make a Conformance Claim to some or all of the requirements indicated in the Conformance Clause. The Conformance Claim is a declaration of the implementer’s system capabilities with respect to the standard. The Implementer develops a system based on their Conformance Claim (Point 5).

Credit: Robert Snelick

Figure 1: Standards and Testing Landscape

Points 2, 4, 6, and 7: The Validation Method assembles the detailed requirements in the standard based on the implementer’s Conformance Claim with respect to the Conformance Clause (Points 2, 4, and 7). This defines the scope of the testing requirements for the Validation Method. Next, the Implementation is tested against the requirements (Point 6). This step is conformance testing, conducted to ascertain if requirements in the standard have been implemented correctly.

Points 8: The test evaluator performs an assessment of the testing results. Interpretation or results and application of actions to be taken differ depending on the setting and the aim of the testing.

Point 9 and 10: Conformance testing from a high-level perspective can be categorized as capability testing and production testing. Capability testing assesses a system’s ability to provide defined functionality. The setting is in a controlled environment in which directed test cases are designed to exhaustively examine a system. Capability testing can be performed by healthcare information technology (HIT) vendors internally or by established testing programs. Capability testing provides assurances to stakeholders that a HIT software product meets the requirements indicated in the specification.

Conformance testing can also be conducted on live systems in a production environment. In this setting, the aim is to provide real-time feedback such that processing choices can be made.

Point 9: A testing program can issue certificates of recognition that can be used by vendors for requests for proposals and marketing material. This designation indicates that the HIT system possesses certain capabilities. This is important because it enables sellers of HIT products to substantiate their claims and buyers to have increased confidence in a product. The Integrating the Healthcare Enterprises (IHE) consortium, which holds connect-a-thon events, is an example of a testing program that issues product recognition.

Another level of recognition is product certification. An example of product certification is the Assistant Secretary for Technology Policy (ASTP) Office of the National Coordinator (ONC) for Health IT Certification Program that ensures that Certified HIT meets the technological capability, functionality, and security requirements adopted by the U.S. Department of Health and Human Services (HHS). Developers certify their Health IT Modules by demonstrating conformance to these certification criteria, using test procedures and/or test tools (such as those created by the National Institute of Standards and technology (NIST)). ASTP/ONC works with Authorized Testing Labs (ATLs) and Authorized Certification Bodies (ACBs) to perform the testing and to certify the products.

Point 10: Interpretation and subsequent actions taken as a result of the assessment will differ in production environments. In a testing program (Point 9), a negative test outcome results in failure, whereas in production environments, testing outcomes can inform business decisions. These decisions can vary from installation to installation. Content with errors may still provide usable and critical data. In the production environment setting, the assessment is used as a tool. The outcomes can inform the system to accept the data entirely, accept the data in part and notify the initiating system of the issues, or reject the data entirely and respond to the initiating system, accordingly.