NIST logo

Robot Perception for Workspace Situational Awareness


Next generation robotic systems are expected to perform highly complex tasks in dynamic manufacturing environments.  To be successful in performing these tasks, they need situational awareness –the ability to detect, interpret, and anticipate the actions of people and objects in their environment.  Prototypes of these perception systems are being developed but science-based approaches to measure their performance do not exist. This project will develop the metrics and methods that underpin such approaches, with an initial focus on the ability to detect people and objects as they move about the workspace. The project will build a testbed and conduct experiments to assess that ability. Project results will provide scientific foundations for new standards that enable the use of perception systems in manufacturing applications.



By 2015, develop and deploy measurement science, including a testbed, for manufacturing perception systems, to enable robot perception for workspace situational awareness.

What is the new technical idea?

Even after 30 years of progress, robots still have very restricted capabilities in manufacturing. They can perform only repetitive tasks in highly structured and controlled environments. Their limited perception systems remain the principal obstacle to expanding these capabilities. Academic and industrial researchers are developing more capable perception systems but science-based approaches to measure their performance do not exist. This project will develop that foundation. Its primary challenges are:

  • Lack of performance measurement methods and standards to assess perception systems that enable robots to detect objects (robots, automated guided vehicles, mobile manipulators, other equipment), and people of interest in dynamic, unstructured manufacturing environments;
  • Lack of performance measurement methods and standards to assess perception systems that enable robots to adapt to new tasks rapidly, safely, and efficiently; to monitor tasks to ensure that they are executed properly; and to inspect parts and assemblies so they meet quality requirements.

The new technical idea is to develop ground truth systems that will provide real-time, high-fidelity, time-synchronized information about people and objects in a manufacturing workspace. These ground truth systems will provide a basis against which to evaluate what systems under test perceive and to characterize their performance. To support the collection, storage, and use of this information, the project will develop an ontology of sensors and the objects they detect, and world models that maintain a database of tracked objects. To support these development efforts, a perception evaluation testbed will be built to enable the validation of the ground truth systems and their applications in measurement methods.

What is the research plan?

This project will develop science-based test methods and metrics for evaluating the performance of perception systems that detect objects and people under static and dynamic conditions for the purpose of situational awareness. It will also develop a testbed that includes ground truth systems that will form the foundation for those test methods.

The testbed will be developed in cooperation with industry and academia.  Together with the other projects in the program, we will create an appropriate set of technical requirements that provide the foundation for new standards for perception systems for robots, automated guided vehicles (AGVs), and mobile manipulators. The testbed will contain a collection of state-of-the-art perception and safety technologies. The testbed will be developed incrementally, starting with emphasis on performance measurement methods for specific manufacturing test scenarios and capabilities such as human detection and tracking, kit building, bin picking, and shop floor monitoring.

Measurement science research conducted in the testbed will include developing an ontology for representing the characteristics and capabilities of perception systems for detecting people and objects. Situational awareness systems will require the integration and fusion of multiple types of sensors, each of which will be best for particular classes of objects and behaviors. By collecting and organizing the capabilities of sensor types, the ontology will enable both the design of sensor networks appropriate for the needs of a manufacturing facility, and the organization of performance procedures and metrics for the evaluation of individual sensors, perception algorithms, and sensor networks.

This project will also develop the measurement science needed for integrating a variety of existing state-of-the-art sensor systems into a suitable ground truth system for evaluating perception systems that detect people and objects.  Its initial focus will be on 3D and 6D laser tracker-based position measurement systems; optical, microwave, RFID, and ultra wide band tracking systems; 2D and 3D imaging systems; and camera networks.

Recent Results

Established a preliminary testbed with ground truth systems for pose estimation and mobile robot tracking.

Standards and Codes

The project will work with the IEEE and ASTM to develop new standards for knowledge representation and for perception systems.  The standards will be developed within ASTM’s E57 Committee on 3D Imaging Systems and the IEEE Robotics and Automation Society’s Working Group on Knowledge Representation.