Performance of Human-Robot Interaction

Summary

With an increased interest in the reshoring of manufacturing in the United States, more businesses are turning to technology to accommodate the increased demand for American-made parts and products. The progression to full automation, however, is both long and extremely expensive. To bridge the gap between current manufacturing capabilities and the future production goals, manufacturing processes must leverage both human and machine labor, often without sufficient lead time to provide adequate training for the use and maintenance of equipment. As such, a renewed interest in human-focused robotics and automation design, control, and programming has emerged. The Performance of Human-Robot Interaction project will expand on its prior work to produce novel measurement science to assess and assure the usability, performance, and trustworthiness of emerging robotic systems. Focus areas will include 1) performance and design considerations of novel interface designs for task-driven, user interfaces and experiences; 2) trust and safety in human-facing robotic solutions; 3) traceable models and test methods for the accurate and repeatable localization and tracking of humans in the shared workspace; and 4) developing high quality datasets for training artificial intelligence models of human behaviors and intention. The outputs of this project – datasets, benchmarking tools, test methods, protocols, metrics, and standards–will enable integrators and end-users to maximize the effectiveness and efficiency of collaborative human-robot teams in production processes, and will benefit all scales of manufacturing that employ robot-assisted, skilled labor.

Description

Credit: Earl Zubkoff

Objective
Deliver a suite of datasets, benchmarking tools, test methods, protocols, metrics, standards, and information models to enable effective, human-robot collaboration in manufacturing, and advance interactive robot technologies to facilitate the safe and efficient teaming of people and robots that maximally leverages the strengths and capabilities of each toward meeting production goals.

Technical Idea
To achieve the specified objective, the Performance of Human-Robot Interaction project will focus on developing a collective metrology suite consisting of datasets, benchmarking tools, test methods, metrics, systems models, software libraries, and algorithms to evaluate the capabilities and performance of human-robot interfaces, mechanisms of trust and safety in human-facing robots, and system and situation awareness models levered in human-robot interaction. This collective metrology suite will enable technology developers, integrators, and end users of collaborative robot technologies to:

• Integrate, evaluate, and optimize metrology technologies (i.e., sensors and algorithms) built into and influencing the design of interfaces intended to optimize the user experience. We will develop test methods and metrics, labeled data sets, guidelines and best practices for HRI studies and dataset generation and publication, and virtual models that can be used to assess and assure effective information sharing and situation awareness.

• Assess and optimize the actions of robot systems and presentation of robot feedback to encourage trust and establishing common ground with human operators in collaborative tasks. We will develop guidelines and documented best practices for the design of physical appearances, behaviors, and feedback mechanisms that encourage ease-of-use and integration, and minimize perceived robot intent ambiguity for users working with and around robots.

• Drive innovation in the design and testing of systems that enable safe interactions and implementations, establish and maintain trust in the capabilities, safety, and security of operator interests, and reinforce good faith relationships between stakeholders and machines. We will develop guidelines and best practices that guide safe robotic system design and implementation.

• Develop new behaviors and capabilities of interactive robot systems to support new operator training, team adaptability to process change and uncertainty, and responsiveness and utility in high-impact situations. We will develop new protocols and test methods for the evaluation of human preferences and experience, as well as mechanisms for evaluating the effectiveness of information sharing protocols for efficient communication.

• Share and receive instructions, status updates, and diagnostic data using methods intuitive to both sides of the collaborative human-robot team using this project’s resulting models, metrics and software libraries. We will develop new test methods and software tools for assessing communication efficacy through advanced interfaces, which can be used to optimize the presentation of task-relevant information to human operators.

• Assess and assure the efficiency and effectiveness of employing collaborative robots in flexible factory environments such that the costs and benefits of leveraging robots in collaborative teams are optimally balanced in favor of the end user. We will continue to develop new metrics, test methods, and standards for the evaluation of the impacts of employing collaborative robots in manufacturing environments (including the ease of use, the cost of programming, impacts on team process performance, and the impacts on risk assessments and safety protocols).

• Employ robots capable of automatically and safely adapting to user preferences, experience, attention, and actions relevant to the collaborative task. We will develop new test methods and metrics for the evaluation of human-aware collaborative robots, including metrology for the evaluation of operator intent and motions as they pertain to safe and productive human-robot collaboration.

Research Plan
This research plan focuses on four principal capabilities of robot systems that collectively contribute to human-robot integration and teaming: 1) creating models of trust and safe human-robot interaction; 2) developing test methods, metrics, and best practices for the design of interfaces; 3) design metrology tools for measuring the impacts and requirements of digital twins; and 4) developing novel metrology tools for soft robotics and flexible materials. Collectively, these focus areas comprise a collective suite of test methods, metrics, and protocols to assess and assure close-proximity human-robot interaction performance in emerging manufacturing applications. For each phase of development, the test methods, metrics, and protocols will be evaluated using the NIST human-robot interaction laboratory.

• Trust and Safety in Human-Robot Teaming in Manufacturing: As skilled human workforces are expected to leverage robotic technologies to achieve ever-increasing demands for throughput and quality, considerable trust must be placed in the robotic systems. The safety and security of related interactions are paramount to maintain this trust, and the scientific research that feeds into commercial systems must remain focused on preserving the wellbeing of workers. We will deliver protocols and best practices for the establishment of standardized guidelines for conducting human-robot interaction research, protecting operator safety and business security, and maintaining confidence and trust in daily interactions. Misapplications of artificial intelligence undermine the greater public trust in intelligent systems, so these procedures and protocols are critical for establishing and maintaining trust of the operators who must routinely interact with these robotic systems.

• Designs of Interfaces: We will deliver protocols and test methods for assessing the design and use of robots and their functionality in collaborative human-robot teaming. These test methods will quantify the impacts of the robot’s interfaces and means of interacting with human operators, and assess the effectiveness and efficiency of interfaces on the humans’ ability to program, control, and diagnose robotic applications.

• Human-Specific Localization and Tracking: We will deliver traceable models and test methods for the accurate and repeatable localization and tracking of humans in the shared workspace. These models and test methods will drive the development of new, standardized measurement science that will enable the accurate and repeatable evaluation of human position and pose.

• High Quality Datasets for Training AI: We will develop high quality datasets for training artificial intelligence models of human behaviors and intention. These datasets will be made publicly available for the benefit of the greater human-robot interaction community, and will provide bases for advancing the state of practice for human-robot teaming in manufacturing applications.

---

Recent Major Accomplishments

• NIST-developed reporting criteria have been adopted by the Association for Computing Machinery (ACM) for their annual HRI conference.
• NIST-led standards efforts on HRI through IEEE have increased growth of stakeholder interest in HRI benchmarks, research replicability, and reporting criteria.
• NIST’s work to raise awareness of metrological challenges in HRI through publications (including special issues of archival research journals; book chapters on measurement science, verification, and validation of HRI; and numerous publications on HRI test methods and metrics) have resulted in significant community support for replicable and repeatable research, HRI applications- and technology-focused events, and improved reporting criteria for human-subject HRI studies.