NIST logo

Performance Measurement of Equipment Networks in Smart and Wireless Factories


Smart factories rely heavily on both wired and wireless networks for real-time control and information exchange. Users have to make critical decisions about which technologies will meet their needs: wired versus wireless and competing vendors and protocols. These decisions are difficult to make without having access to standardized performance metrics and test methods. This project will develop and deploy measurement science-based performance measurement metrics and test methods for wired and wireless factory equipment networks to enable real-time factory floor monitoring and optimization of smart manufacturing systems.



Develop the measurement science needed to assess the performance of equipment networks used in factories enabling the integration of real-time factory equipment information and the optimization of wired and wireless manufacturing systems delivering results through open-source implementations and standards bodies by 2014.

What is the new technical idea?

The new technical idea is to develop network performance metrics and test methods for smart and wireless equipment networks using models of analysis results instead of raw timing data. These models will measure the effectiveness of standards for collecting real-time production information from factory floor equipment, reducing barriers to optimizing manufacturing activities within a factory. One barrier is the cost of running wired networks, ranging from $100s per foot in factories to $5000 per foot in nuclear sites. Another barrier is the difficulty in comparing performance results between protocols. Many industrial wireless protocols have been developed for communicating with devices, such as ISA-100 and WirelessHART for wireless sensors. It is difficult to compare and quantify the benefits of applying these wireless protocols for wire replacement for sensor and equipment connectivity. In addition, it is difficult to diagnose the root causes of performance problems. The Smart Manufacturing Leadership Coalition (SMLC) recognizes that "the collection and use of engineering data in manufacturing facilities today is relatively inefficient due to the lack of standardized, easily useable data systems," and calls for better data protocols, interfaces and communications in their priority action for establishing consistent data methods for all industries[1]. This project addresses these challenges by developing model-based methodologies for performance measurement and analysis, and developing performance metrics and testing methods for smart and wireless factory equipment networks. Existing standards for performance measurements of networked devices include the Internet Engineering Task Force (IETF), the International Telecommunication Union (ITU), and the International Electrotechnical Commission (IEC). The focus of these standards is measuring the performance of common computing hardware running on wired IT networks, and do not apply to real-time wireless factory networks. In this project, we will develop performance metrics and testing methods for smart and wireless factory networks with the intent of disseminating the results to these and other standards organizations.

What is the research plan?

The project will improve and expand the current raw factory network performance metrics (e.g., jitter, latency, and bandwidth) and develop new test methods to analyze and report performance results. The value of the performance characteristics measured were recognized to be widely applicable to smart factory networks and led to the development of the Industrial Ethernet Network Performance (IENetP) test tool and the factory equipment network testing (FENT) framework. The current data analysis methodologies used (traditional Gaussian statistical analysis) are not appropriate for much of the data that has been collected due to the analog nature of the calculation versus the results have been observed. Wireless factory networks have the same requirements for timing and determinism as wired factory networks, but may need additional metrics and test methods in order to fully characterize their performance . The research challenge is to develop mathematical models based on measurement results and associate these models with root casues of performance problems. An associated research challenge is to define a portfolio of data sets, measurements, and analyses for industrial networks that can provide figures of merit for real-time networking properties, similar to vehicle fuel economy. The FENT framework provides the capability to verify the newly developed metrics and test methods in this project. Project researchers will collaborate with industry partners across the discrete manufacturing, batch, and continuous process industries to determine any additional performance measurement needs. We will publish all the measurements and analyses. Outputs from the project will be disseminated to groups like the IETF as a request for comment (RFC) or to SDOs like IEEE, ISA, ITU, or IEC as a proposed standard.

Recent Results:

FY2011 Impacts:

  • ENetP is the primary data analysis tool in the ODVA EtherNet/IP Low-Cost Performance Test Suite.
  • The IEEE 1451.0, 1451.5, and 1451.7 interface standards were adopted by ISO as international standards, ISO/IEC/IEEE 21450, 21451-5, and 21451-7 standards, respectively.

FY2012 Outputs

  • NIST Technical Note on Factory Equipment Network Testing Framework: Scope, Requirements, and Architecture
  • NIST Technical Note on Factory Equipment Network Testing Framework: Universal Client Application, Application Programming Interface
  • Software testing tool designed to allow both end users and vendors to evaluate the performance of factory equipment and networks, including the FENT framework architecture, the universal client application (UCA), the UCA application programming interface (API), and personality modules (PMs) for multiple protocols, including IEEE 1451, EtherNet/IP, and the NIST-developed sensor simulator (SensorSIM).
  • Design, construction, and testing completed of a distributed network testbed including the Virtual Local Area Network (VLAN) for testing factory equipment networks
  • Publication: Kang B. Lee, Eugene Y. Song, Peter S. Gu, “Integration of MTConnect and Standard-based Sensor Networks for Manufacturing Equipment Monitoring”, Proceedings of ASME 2012 International Manufacturing Science and Engineering Conference MSEC 2012, June 4-8, 2012 in Notre Dame, Indiana.
Standards and Codes:

This project will disseminate the results to the EtherNet/IP Interoperability Recommendations and Test Plan from ODVA (formerly the Open DeviceNet Vendor Association), the IEEE 1451 standard for networked sensors, the IEEE 1588 standard for a precision time protocol through revisions to standards that result from validation testing in the FENT testbed. In addition, the results will be disseminated to the IETF Benchmarking Methodology and IP Performance Metrics working groups, the ITU Telecommunication Standardization Sector, and the IEC TC/SC65C committee on industrial networks. Expansion of the testing framework is likely to include other standards including wireless protocols such as ISA-100.11a, WirelessHART, and ZigBee. NIST will work with industry associations and standards development organizations (SDOs), such as ay the IEEE Conformance and Assessment Program (ICAP), the International Society of Automation (ISA), the Association for Manufacturing Technology (AMT), and the OPC Foundation to establish testing and certification authorities.

[1]Smart Manufacturing Leadership Coalition, Implementing 21st Century Smart Manufacturing, Workshop Summary Report, June 24, 2011, p. 16.

Manufacturing equipment in a factory floor.

Start Date:

October 1, 2011

Lead Organizational Unit:


General Information:

Kang Lee, Project Leader

301 975 6604 Telephone
301 975 8058 Fax

100 Bureau Drive, M/S 8220
Gaithersburg, MD 20899