NIST logo

Key Performance Indicator Effectiveness


Key performance indicators (KPIs) are primary tools for optimizing factory operations, but manufacturers do not fully benefit from KPIs. As a result, industry leaders are asking: which KPIs are best for process performance monitoring, what is the effectiveness of each KPI, and of the many KPIs available, which is the handful most effective at improving the process[1]? This project will yield the measurement science contributing to automated optimal manufacturing by developing measurement techniques that quantify the effectiveness of KPIs to mitigate process and product quality deficiencies in a timely manner.



To achieve gains in automated optimal manufacturing operations by quantifying the effectiveness of KPIs to mitigate process and quality deficiencies, delivering results to standards bodies by 2014.

What is the new technical idea?

Achieving the optimized factory is hard, and there is much work to be done. For example, monitoring and analysis using the overall equipment effectiveness (OEE) KPI (an important KPI) is only done in 5.3% of machine shops[2]. Broadly speaking, optimal KPI utilization has not been realized, in part because manufacturers do not know which KPIs provide the most value to a given manufacturing process. There has recently been a request1 for a metric of KPI effectiveness, in order to focus on the subset of the most effective KPIs for optimizing a particular process. NIST will address this problem through the development of measurement science which can be used to quantify the effectiveness of a KPI for optimizing a process.

Beyond KPI effectiveness, manufacturers need to know what to fix and how to fix it when a set of KPIs reveal a problem. This requires the prior discovery of the correlation between KPI measurements and the parameters of the process effecting those measurements. NIST will address this problem through the development of measurement methods to quantify this correlation.

Knowing the correct subset of KPIs for the process and how to fix a defective process are still insufficient to enable optimization. Everything must occur in a timely fashion. KPIs are computed from performance measurements, which are all too commonly gathered manually. Automating performance measurement gathering is essential to optimizing operations. NIST will address this problem through measurement science support for the successful development and implementation of those standards enabling automation of KPI generation and use.

What is the research plan?

NIST research will focus first-year efforts on discrete parts processes (e.g., job shops with machine tools and measuring machines) and certain quality-related KPIs (e.g., OEE), which are in harmony with the expertise of project staff. If successful, outyear projects will expand the application of this approach to other manufacturing processes. In FY13 we plan to: 1) discover measurement science barriers for effectiveness of quality-related KPIs for job shop processes, 2) define effectiveness metrics for the inadequate integration costs KPI, and 3) define and execute a first round of KPI effectiveness measurement tests.

The FY13 focus will be on two quality-related performance measures: overall equipment effectiveness (OEE) and first article inspection (FAI). The misinterpretation of OEE is common in the discrete parts industry and is typified by exclusion of the quality factor when evaluating asset performance. This is due to the lack of a pervasive quality information model within the enterprise, and thus, correlation of quality feedback to asset performance is difficult and costly. A candidate quality information model is under development within the Dimensional Metrology Standards Consortium’s (DMSC) and is called Quality Information Framework (QIF). Correct, complete, unambiguous, and widely-implemented information standards enable automated optimization, since the end user enjoys data of higher quality and also gains agility. QIF is on the path to become such a standard, so will also enable automated optimization. NIST will address this problem by working with the QIF working group to define standards based part quality results (QMResults) and multi-part statistics (QMStatistics) to allow quality traceability for the correlation of quality metrics to assets. In FY13, NIST will also work with the industrial partners to incorporate QMResults into the MTConnect standard, which will allow quality traceability metrics to be tightly correlated to asset performance by factory analytic applications and result in more accurate OEE performance measurements.

FY13 will address another KPI: inadequate integration costs, with measurement science research being performed in collaboration with the NIST EL Applied Economics Office, Purdue University, and The Boeing Corporation. The new work in FY13 is to define KPI effectiveness measurement for the inadequate integration costs KPI and to archive this ongoing research.

Our research approach is: 1) define the systems under test in terms of the following components: optimization targets, manufacturing process, operator (human or computer), and KPI set, 2) determine target manufacturing sectors and operations for test, 3) determine current baseline measurement of each component of the systems under test, 4) define and apply draft KPI effectiveness metrics for all systems under test, both empirically and theoretically, 5) refine research approach and effectiveness metrics based on results from baseline study and application of KPI effectiveness metrics. Research method is expected to involve in situ performance measurements to determine baseline values, through collaboration with academic researchers.

In general, our research is expected to be heavily collaborative with various manufacturers, manufacturing associations, e.g., Advanced Manufacturing Technology (AMT), research organizations, e.g., Purdue University; and standards bodies including ISO TC 184 SC5 (KPI working group), Dimensional Metrology Standards Consortium (DMSC), the MTConnect Institute, the Automotive Industry Action Group’s (AIAG) Production Part Approval Process (PPAP) group, the Manufacturing Enterprise Solutions Association (MESA) Metrics Working Group, and the International Aerospace Quality Group (maintainers of the FAI standard: AS9102a).

Outyear focus will be to further address the correlation between KPI values and process parameters through work on traceability in quality operations, and to apply the KPI effectiveness measurement science beyond job shop operations and quality-related KPIs.

Recent Results:
  • Outcome:  MTConnect Version 1.1.0 Parts 1, 2 and 3, April 2010; NIST contributed through performance and validation testing at Boeing Commercial Aircraft and General Motors.
  • Outcome:  Quality Information Framework (QIF) Schema version 0.92 released to the Dimensional Metrology Standards Consortium (DMSC) and published on
  • Outcome: OMAC Standardized Working-Steps for CNC Milling Machines
  • Output: Reference document describing performance metrics and benchmarks necessary to perform optimization of shop floor operations, which provided functional requirement for the NIST VFT
  • Output: Archival publication describing Virtual Factory Testbed layout utilizing the Core Manufacturing Simulation Data (CMSD) XML specification
  • Output: Generated software components – compatible with MTConnect – which receive CMSD simulated/emulated data, and which will be used for shop floor operations optimization work in outyears

Standards and Codes:

  • ISO 22400 Manufacturing operations management – Key performance indicators (in draft) ISO TC184 SC5 WG9  and the MESA Metrics Working Group (liaison committee to ISO TC184 SC5 WG9)
  • Dimensional Metrology Standards Consortium (DMSC) Quality Information Framework (QIF), QMPlans, and QMResources standards
  • ISO TC184 SC4 Industrial Data (STEP) AP242 with PDES, Inc.
  • MTConnect Version 1.1.0 Parts 1, 2, and 3

[1]Darlene Schuster, Executive Director of the American Institute of Chemical Engineers, at the Smart Manufacturing Leadership Consortium meeting 20 Oct 2011 in Minneapolis, MN.

[2]Mark Albert, “Each Shop’s One Big Thing” Modern Machine Shop, 12 Jul 2011