OMB'S Information Quality Bulletin for Peer Review
Based on the review it has conducted, NIST believes that it does not currently produce or sponsor the distribution of influential scientific information (including highly influential scientific assessments) within the definitions promulgated by OMB. As a result, at this time NIST has no agenda of forthcoming influential scientific disseminations to post on its website in accordance with OMB's Information Quality Bulletin for Peer Review.
NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY
PART I: BACKGROUND, MISSION, DEFINITIONS, AND SCOPE
Section 515 of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554), hereinafter "Section 515," directs the Office of Management and Budget (OMB) to issue government-wide guidelines that "provide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies." OMB complied by issuing guidelines which direct each Federal agency to (A) issue its own guidelines ensuring and maximizing the quality, objectivity, utility, and integrity of information disseminated by the agency; (B) establish administrative mechanisms allowing affected persons to seek and obtain correction of information that does not comply with the OMB 515 Guidelines (Federal Register: February 22, 2002, Volume 67, Number 36, pp. 8452-8460, hereinafter "OMB Guidelines") and the agency guidelines; and (C) report periodically to the Director of OMB on the number and nature of complaints received by the agency regarding the accuracy of information disseminated by the agency and how such complaints were handled by the agency.
In compliance with OMB directives, the Department of Commerce (DOC) has issued Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Disseminated Information (http://ocio.os.doc.gov/ITPolicyandPrograms/Information_Quality/DEV01_002680).
This document implements Section 515 and fulfills the OMB and DOC information quality guidelines. It may be revised periodically, based on experience, evolving requirements in the National Institute of Standards and Technology (NIST), and concerns expressed by the public. Covered information disseminated by NIST will comply with all applicable OMB, DOC, and NIST Information Quality Guidelines and Standards. NIST is committed to maintaining a high level of quality in the information it disseminates.
In implementing these guidelines and standards, NIST acknowledges that ensuring the quality of information is an important management objective that takes its place alongside other NIST objectives, such as ensuring the success of the NIST mission, observing budget and resource priorities and restraints, and providing useful information to the public. NIST intends to implement these guidelines and standards in a way that will achieve all these objectives in a harmonious way.
NIST's mission is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life.
The definitions in this section apply throughout these Guidelines and Standards.
Quality is an encompassing term comprising utility, objectivity, and integrity. Therefore, the guidelines sometimes refer to these four statutory terms, collectively, as "quality."
Utility refers to the usefulness of the information to its intended users, including the public. In assessing the usefulness of information that the agency disseminates to the public, NIST considers the uses of the information not only from its own perspective but also from the perspective of the public. As a result, when transparency of information is relevant for assessing the information's usefulness from the public's perspective, NIST takes care to ensure that transparency has been addressed in its review of the information.
Objectivity consists of two distinct elements: presentation and substance. The presentation element includes whether disseminated information is presented in an accurate, clear, complete, and unbiased manner and in a proper context. The substance element involves a focus on ensuring accurate, reliable, and unbiased information. In a scientific, financial, or statistical context, the original and supporting data will be generated, and the analytic results will be developed, using sound statistical and research methods.
Integrity refers to security – the protection of information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification.
Information means any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a Web page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agency's presentation makes it clear that what is being offered is someone's opinion rather than fact or the agency's views.
Government information means information created, collected, processed, disseminated, or disposed of by or for the Federal Government.
Information dissemination product means any books, paper, map, machine-readable material, audiovisual production, or other documentary material, regardless of physical form or characteristic, an agency disseminates to the public. This definition includes any electronic document, CD-ROM, or Web page.
Dissemination means agency initiated or sponsored distribution of information to the public. Dissemination does not include distribution limited to government employees or agency contractors or grantees; intra- or inter-agency use or sharing of government information; and responses to requests for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act or other similar law. This definition also does not include distribution limited to correspondence with individuals or persons, press releases, archival records, public filings, subpoenas or adjudicative processes.
Agency initiated distribution of information to the public refers to information that the Agency distributes or releases which reflects, represents, or forms any part of the support of the policies of the Agency. In addition, if the Agency, as an institution, distributes or releases information prepared by an outside party in a manner that reasonably suggests that the Agency agrees with the information, this would be considered Agency initiated distribution and hence Agency dissemination because of the appearance of having the information represent Agency views.
Agency sponsored distribution of information to the public refers to situations where the Agency has directed a third party to distribute or release information, or where the Agency has the authority to review and approve the information before release. By contrast, if the Agency simply provides funding to support research, and if the researcher (not the Agency) decides whether to distribute the results and - if the results are to be released - determines the content and presentation of the distribution, then the Agency has not "sponsored" the dissemination even though it has funded the research and even if the Agency retains ownership or other intellectual property rights because the Federal government paid for the research.
Influential, when used in the phrase "influential scientific, financial, or statistical information,'' means that the agency can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policy and private sector decisions.
Scientific information means factual inputs, data, models, analyses, technical information, or scientific assessments based on the behavioral and social sciences, public health and medical sciences, life and earth sciences, engineering, or physical sciences. This includes any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative or audiovisual forms that involves a field identified in the preceding sentence. This definition includes scientific information that an agency disseminates from a web page, but does not include the provision of hyperlinks to scientific information that others disseminate. This definition does not include opinions, where the agency's presentation makes clear that what is being offered is someone's opinion rather than fact or the agency's views.
Influential scientific information means scientific information the agency reasonably can determine will have or does have a clear and substantial impact on important public policies or private sector decisions.
Scientific assessment means an evaluation of a body of scientific or technical knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information. These assessments include, but are not limited to, state of science reports; technology assessments; weight of evidence analyses; meta analyses; health, safety, or ecological risk assessments; toxicological characterizations of substances; integrated assessment models; hazard determinations; or exposure assessments.
The term highly influential scientific assessment means influential scientific information that the agency or the Administrator of the Office of Information and Regulatory Affairs in the Office of Management and Budget determines to be a scientific assessment that: (i) could have a potential impact of more than $500 million in any year, or (ii) is novel, controversial, or precedent setting or has significant interagency interest.
Reproducibility means that the information is capable of being substantially reproduced, subject to an acceptable degree of imprecision. For information judged to have more (less) important impacts, the degree of imprecision that is tolerated is reduced (increased). With respect to analytic results, "capable of being substantially reproduced'' means that independent analysis of the original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error.
Transparency is not defined in the OMB Guidelines, but the Supplementary Information to the OMB Guidelines indicates (p. 8456) that "transparency" is at the heart of the reproducibility standard. The Guidelines state that "The purpose of the reproducibility standard is to cultivate a consistent agency commitment to transparency about how analytic results are generated: the specific data used, the various assumptions employed, the specific analytic methods applied, and the statistical procedures employed. If sufficient transparency is achieved on each of these matters, then an analytic result should meet the reproducibility standard." In others words, transparency – and ultimately reproducibility – is a matter of showing how you got the results being disseminated.
These guidelines cover information disseminated by NIST on or after October 1, 2002, regardless of when the information was first disseminated, except that pre-dissemination review procedures shall apply only to information first disseminated on or after October 1, 2002.
Information Disseminated by NIST and Covered by these Guidelines and Standards
Information quality is an integral part of the pre-dissemination review of information disseminated by NIST. Information quality is also integral to information collections conducted by NIST, and is incorporated into the clearance process required by the Paperwork Reduction Act (PRA) to help improve the quality of information that NIST collects and disseminates to the public. NIST programs already are required to demonstrate in their PRA submissions to OMB the utility of a proposed collection of information that they plan to disseminate. Additionally, for all proposed collections of information that will be disseminated to the public, NIST programs should demonstrate in their PRA clearance submissions to OMB that the proposed collection of information will result in information that will be collected, maintained, and used in a way consistent with applicable information quality guidelines and standards.
Information Not Covered by these Guidelines
Information with distribution intended to be limited to government employees or agency contractors or grantees, including but not limited to the Manufacturing Extension Partnership (MEP) Extranet Information for Center grantees.
Information with distribution intended to be limited to intra- or inter-agency use or sharing of government information, including but not limited to programmatic performance, budget, human resources and strategic planning information submitted to the DOC.
Responses to requests for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act or other similar law.
Information relating solely to correspondence with individuals or persons, including but not limited to calibration reports, and Internal or Interagency Reports prepared and released to a single customer.
Press releases, fact sheets, press conferences or similar communications in any medium that announce, support the announcement or give public notice of information NIST has disseminated elsewhere.
Archival records, including library holdings.
Archival information disseminated by NIST before October 1, 2002, and still maintained by NIST as archival material.
Information presented to Congress as part of legislative or oversight processes, such as testimony of NIST officials, and information or drafting assistance provided to Congress in connection with proposed or pending legislation, that is not simultaneously disseminated to the public.
Information limited to adjudicative processes, such as pleadings, including information developed during the conduct of any criminal or civil action or administrative enforcement action, investigation or audit against specific parties, or information distributed in documents limited to administrative action determining the rights and liabilities of specific parties under applicable statutes and regulations.
Solicitations (e.g., program announcements, requests for proposals).
Hyperlinks to information that others disseminate, as well as paper-based information from other sources referenced, but not approved or endorsed by NIST.
Policy manuals and management information produced for the internal management and operations of NIST, and not primarily intended for public dissemination.
Documents not authored by NIST and not intended to represent NIST's views, including information authored and distributed by NIST grantees, as long as the documents are not disseminated by NIST.
Research data, findings, reports and other materials published or otherwise distributed by employees or by NIST contractors or grantees that are identified as not representing NIST views.
Opinions where the presentation makes it clear that what is being offered is not the official view of NIST.
Although information delivered to specific customers is not "disseminated" for purposes of these information quality standards, NIST's commitment to quality extends to this information. For example, NIST has entered into a Mutual Recognition Arrangement with the International Committee on Weights and Measures (CIPM), pursuant to which NIST has adopted a system for assuring quality in the results of measurements delivered to customers in calibration and measurement certificates.
In addition, NIST releases software and algorithms. Some of this information is released as a service to the public, and the release of the information is not intended to imply NIST endorsement or adoption of the information. For this information, NIST will attach a disclaimer.
PART II: INFORMATION QUALITY STANDARDS AND PRE-DISSEMINATION REVIEW
As OMB has recognized (OMB Guidelines, pp. 8452-8453), "information quality comes at a cost." In this context, OMB directed that "agencies should weigh the costs (for example, including costs attributable to agency processing effort, respondent burden, maintenance of needed privacy, and assurances of suitable confidentiality) and the benefits of higher information quality in the development of information, and the level of quality to which the information disseminated will be held." Therefore, in deciding the appropriate level of review and documentation for information disseminated by NIST, the costs and benefits of using a higher quality standard or a more extensive review process will be considered. Where necessary, other compelling interests such as privacy and confidentiality protections will be considered.
The utility and integrity standards below pertain to all information disseminated by NIST. Following the utility and integrity standards are objectivity standards for each of the specific categories of NIST-disseminated information. Because most of the standards presented in this document reflect existing practice in NIST, the present tense has been used when describing them; but regardless of tense used, a performance standard is intended.
Utility means that disseminated information is useful to its intended users. "Useful" means that the content of the information is helpful, beneficial, or serviceable to its intended users, or that the information supports the usefulness of other disseminated information by making it more accessible or easier to read, see, understand, obtain, or use. Where the usefulness of information will be enhanced by greater transparency, care is taken that sufficient background and detail is available, either with the disseminated information or through other means, to maximize the usefulness of the information. The level of such background and detail is commensurate with the importance of the particular information, balanced against the resources required, and is appropriate to the nature and timeliness of the information to be disseminated.
NIST's mission is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life. NIST maintains ongoing contact with a broad spectrum of users through a variety of means, including but not limited to public meetings, public workshops, individual contacts, and formal and informal collaborations and partnerships, to ensure that the information it disseminates continues to remain relevant. NIST attends and holds public workshops, conferences, and meetings to gather input about what types of information would be useful to industry; universities; other not-for-profit entities; and Federal, state, and local governments; and maintains memberships in many industry groups for the purpose of facilitating such discussions.
Prior to dissemination, NIST information, independent of the specific intended distribution mechanism, is safeguarded from improper access, modification, or destruction, to a degree commensurate with the risk and magnitude of harm that could result from the loss, misuse, or unauthorized access to or modification of such information.
All electronic information disseminated by NIST adheres to the standards set out in Appendix III, "Security of Automated Information Resources," OMB Circular A-130; the Computer Security Act, and the Government Information Systems Reform Act.
Confidentiality of data collected by NIST is safeguarded under legislation such as the Privacy Act and Titles 13, 15, and 22 of the U.S. Code.
Objectivity ensures that information is accurate, reliable, and unbiased, and that information products are presented in an accurate, clear, complete, and unbiased manner. In a scientific, financial, or statistical context, the original and supporting data are generated, and the analytic results are developed, using sound statistical and research methods.
Third-party Information. Third-party information from both domestic and international sources, such as states, municipalities, agencies and private entities may be included in information that NIST disseminates. Although third-party sources may not be directly subject to Section 515, information from such sources, when used by NIST to develop information products or to form the basis of a decision or policy, must be of known quality and consistent with the applicable information quality guidelines and standards. When such information is used, any limitations, assumptions, collection methods, or uncertainties concerning it are taken into account and disclosed.
Corporate and General Information. Corporate and general information disseminated by NIST is presented in a clear, complete, and unbiased manner, and in a context that enhances usability to the intended audience. The sources of the disseminated information are identified to the extent possible, consistent with confidentiality, privacy, and security considerations and protections, and taking into account timely presentation, the medium of dissemination, and the importance of the information, balanced against the resources required and the time available.
Information disseminated by NIST is reliable and accurate to an acceptable degree of error as determined by factors such as the importance of the information, its intended use, time sensitivity, expected degree of permanence, relation to the primary mission(s) of the disseminating office, and the context of the dissemination, balanced against the resources required and the time available. A body of information is considered to be reliable if experience shows it to be generally accurate. Accurate information, in the case of non-scientific, non-financial, non-statistical information, means information which is reasonably determined to be factually correct in the view of the disseminating office as of the time of dissemination.
Review of corporate and general information disseminated by NIST is incorporated into the normal process of formulating the information to take advantage of inherent quality checks that are part of the process of formulating the information. This review is at a level appropriate to the information, taking into account the information's importance, balanced against the resources required and the time available. NIST treats information quality as integral to every step in its process of developing the information, including creation, collection, maintenance, and dissemination.
Review can be accomplished in a number of ways, including but not limited to combinations of the following:
Scientific and financial information disseminated by NIST is reliable and accurate to an acceptable degree of error as determined by factors such as the importance of the information, its intended use, time sensitivity, expected degree of permanence, relation to the primary mission(s) of the disseminating office, and the context of the dissemination, balanced against the resources required and the time available. A body of information is considered to be reliable if experience shows it to be generally accurate. Accurate information, in the case of scientific and financial information, means information which is reasonably determined to be factually correct in the view of the disseminating office as of the time of dissemination.
Influential, when used in the phrase "influential scientific, financial, or statistical information," means that NIST can reasonably determine that dissemination of the information will have or does have a clear and substantial impact on important public policies or important private sector decisions.
Preparation of Scientific Information
Laboratory Notebooks: To appropriately document the preparation of scientific information, NIST maintains a policy regarding laboratory notebooks. NIST Laboratory staff engaged in measurement and in research and development activities are responsible for maintaining a thorough and accurate record of their work by keeping a laboratory or research notebook. Staff using electronic media for measurement, research, and development are responsible for maintaining a written notebook that chronologically documents the progress of their activity and indexes work files so that experimental data and results may be retrieved. The implementation of this policy provides transparency as to the sources of data and the methodologies used to prepare the information disseminated. It also provides the means for ensuring "reproducibility" as defined in the OMB guidelines:
Uncertainty Policy: NIST maintains a policy entitled "Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement Results" (NIST Technical Note 1297, 1994 Edition) (See http://physics.nist.gov/TN1297.) The policy requires that NIST measurement results be accompanied by quantitative statements of their uncertainty, and that a uniform approach to expressing measurement uncertainty be followed. The policy applies to most NIST measurement results, including results associated with international comparisons of measurement standards, basic research, applied research and engineering, calibrating client measurement standards, certifying standard reference materials, and generating standard reference data. The implementation of this policy provides additional transparency as to the accuracy of the information disseminated.
Pre-dissemination Review Process for Scientific Information
The pre-dissemination review process enables NIST to substantiate the quality of disseminated scientific information through documentation or other means appropriate to the nature and importance of the information, balanced against resources required and the time available.
Pre-dissemination review of scientific information disseminated by NIST is incorporated into the normal review processes for each type of information to take advantage of inherent quality checks that are part of the process of formulating the information. This review is at a level appropriate to the information, taking into account the information's importance, balanced against the resources required and the time available. NIST treats information quality as integral to every step in its process of developing the information, including creation, collection, maintenance, and dissemination. All scientific information disseminated by NIST receives a level of scrutiny commensurate with the critical nature of the information and its intended use.
Pre-dissemination review of scientific information can be accomplished in a number of ways, including but not limited to combinations of the following:
In addition, all NIST papers published in external scientific journals receive external peer review as provided by the individual journal. NIST does not assume a rebuttable presumption from these reviews, as allowed by the OMB guidelines; rather, NIST uses such external peer review as an additional pre-dissemination review process.
Peer Review of Influential Scientific Information: "Influential scientific information" or "highly influential scientific assessments" that the agency intends to disseminate are subject to OMB's Final Information Quality Bulletin for Peer Review (OMB Peer Review Bulletin), issued December 16, 2004 (70 FR 2664, Jan. 14, 2005). Peer review of these information products will be conducted in accordance with that Bulletin.
Pre-dissemination Review Process for Financial Information (Economic Impact Studies and Policy Analyses):
Economic impact studies prepared by NIST staff are reviewed by a NIST editorial review board; reports prepared by contractors are not. Instead, the contractor reports are reviewed by the NIST employees with oversight for the contract. In addition, economic impact studies funded by other agencies receive external peer review if required by the funding agency.
Policy studies of financial issues prepared by NIST staff receive internal peer and management review, Department of Commerce review, and possibly higher level reviews, but are not reviewed by a NIST editorial review board.
1. Burden of Proof. The burden of proof is on the requester to show both the necessity and type of correction sought. The requester has the burden of rebutting the presumption that information subjected to formal, independent peer review is objective provided that, for "influential scientific information," and "highly influential scientific assessments," the peer review fulfills the requirements of the OMB Peer Review Bulletin.
Affected person means an individual or entity that uses, benefits from, or is harmed by the disseminated information at issue.
Person means an individual, partnership, corporation, association, public or private organization, or State or local government.
Responsible office means the office within an agency that is designated to make the initial decision on a request for correction based on that agency's information quality guidelines and standards. For NIST, this is the NIST Division.
B. Procedures for Submission of Initial Requests for Correction
1. Any affected person may request, where appropriate, timely correction of disseminated information that does not comply with applicable information quality guidelines and standards. An affected person should submit a request for such action to:
2. An initial request for correction of disseminated information must be made in writing and submitted to the point of contact identified in paragraph 1. above. Any employee receiving a misdirected request should make reasonable efforts to forward the request to the point of contact identified in paragraph 1. above, but the time for response does not commence until the responsible office receives the request.
3. No initial request for correction will be considered under these procedures concerning:
Any request rejected under this provision will nevertheless be accounted for in the Department's report to OMB.
4. At a minimum, to be considered proper, initial requests must include:
6. No action will be taken regarding a request not including all the elements of paragraph III.B.4. (including a request made by a person unaffected by the dissemination of the information), or a request that does not state a claim according to paragraph III.C.1. The submitter of any such request will be notified, usually within 60 calendar days, of this disposition, and, may amend the request and resubmit it. Whether resubmitted or not, such requests will be accounted for in the Department's annual report to OMB.
7. A proper request received concerning information disseminated as part of and during the pendency of the comment period on a proposed rule or other action involving an opportunity for prior notice and public comment, including a request concerning the information forming the record of decision for such proposed rule or action, will be treated as a comment filed on that proposed rule or action, and will be addressed in issuance of any final rule or action.
C. Action by the Responsible Office on Initial Requests for Correction
1. Upon receipt of a proper request, the Division Chief will make a preliminary determination whether the request states a claim. A request for correction states a claim if it reasonably demonstrates, on the strength of the assertions made in the request alone, and assuming they are true and correct, that the information disseminated was based on a misapplication or non-application of applicable published information quality guidelines and standards. In other words, to state a claim, a request for correction must actually allege that NIST disseminated some information that does not comply with applicable information quality guidelines and standards.
A determination that a request does not state a claim will be communicated, along with an explanation of the deficiencies, to the requester, by the Director, NIST Management and Organization Office, usually within 60 calendar days of receipt. The request may be amended and resubmitted as indicated in paragraph III.B.6. above.
2. If a proper request is preliminarily determined to state a claim, the Division Chief will objectively investigate and analyze relevant material to determine whether the disseminated information complies with the applicable published information quality guidelines and standards. The Division Chief will make an initial decision, based on the request and any internal investigation and analysis, whether the information should be corrected because it does not comply with the applicable information quality guidelines and standards ("granted request") or not corrected because it does comply with the applicable information quality guidelines and standards ("initial denial"). The Division Chief will make an initial decision whether the information should be corrected and what, if any, corrective action should be taken. No opportunity for personal appearance, oral argument, or hearing is provided.
If the agency determines that corrective action is appropriate, corrective measures may be taken through a number of forms, including but not limited to: personal contacts via letter or telephone, form letters, press releases, postings on an appropriate website, or withdrawal or correction of the information in question. The form of corrective action will be determined by the nature and timeliness of the information involved and such factors as the significance of the error on the use of the information, and the magnitude of the error.
3. The Division Chief will communicate his/her initial decision or the status of the request through the NIST OU Director to the Director, NIST Management and Organization Office, who will communicate the initial decision to the requester, usually within 60 calendar days of NIST's receipt of the request.
4. The initial decision or status update will contain the name and title of the Division Chief and a notice that the requester may appeal an initial denial to the NIST Associate Director for Laboratory Programs (with the name, title, and address of that official), pursuant to paragraph III.D.1. below, within 30 calendar days of the date of the initial denial.
An initial denial will become a final decision if no appeal is filed within 30 calendar days.
D. Appeals from Initial Denial
1. An appeal from an initial denial must be made within 30 calendar days of the date of the initial decision. Such appeal must be in writing and addressed to:
An appeal of an initial denial must include:
3. The NIST Associate Director for Laboratory Programs will decide whether the information should be corrected based on all the information presented in the appeal record. No opportunity for personal appearance, oral argument, or hearing on appeal is provided. The NIST Associate Director for Laboratory Programs will communicate that decision to the requester usually within 60 calendar days after receipt of the appeal. The decision of the NIST Associate Director for Laboratory Programs will constitute a final decision by the DOC.