Perception Performance of Robotic Systems

Summary

Perception systems extract information and knowledge from raw sensor input and are essential elements of any robotic system. Sensors and data processing algorithms are continuously improving, but selecting perception systems for a given robotic implementation is challenging. Complex robotic assembly applications cannot rely solely on traditional two-dimensional (2D) information and require three-dimensional (3D) knowledge for precise part pose estimation to satisfy tight task tolerances. This project has achieved significant results over the past 5 years, and in the next 5 years the project will focus on developing measurement science for several high-priority challenges. The project will work with stakeholders from industry, academia, and other research organizations to develop metrics, procedures, datasets, artifacts, algorithms, and guidance that will support the development of standards to quantify and evaluate various aspects of 3D imaging system performance. 

Description

Objective
To develop measurement science for characterizing sensing and perception system performance to reduce the risks of adopting these technologies and to advance the agility, safety, and productivity of collaborative, industrial, and mobile robots.

Technical Idea
Over the past 5 years, the project conducted a market survey of 3D imaging systems that are (or can be) used for robotic assembly applications, organized a series of workshops to bring industry stakeholders together to discuss the challenges involved in implementing these systems, worked with industry to develop a Standards Roadmap for 3D Imaging in Robotic Assembly Applications, and established 4 standards efforts under ASTM Committee E57 on 3D Imaging Systems to address the standards needs identified by the road-mapping effort. Guided by the roadmap, in the next 5 years, the project will focus on measurement science for some of the following high-priority challenges: bin-picking performance, perception performance under varying ambient lighting conditions, resolving geometric features, perceiving flexible parts, evaluation of human tracking systems, and guidance for 3D vision system selection.

This will involve identifying existing and emerging sensing and perception technologies and related software, identifying available standards, and understanding current and potential stakeholders' (manufacturers, users, system integrators, and solution providers) use cases and performance requirements. This is crucial to understanding the product and research solutions landscape as well as the conditions under which the sensing and perception systems must operate, and to mapping the performance gaps to the development of metrics and test methods that will contribute to new industry standards.

Research Plan
The project will work with stakeholders from industry, academia, and other research organizations to develop metrics, procedures, datasets, artifacts, algorithms, and guidance that will ultimately support the development of standards to quantify and evaluate various aspects of 3D imaging system performance. The plan will focus on:

1. The performance of 3D machine vision systems used for bin-picking applications. The role of 3D machine vision systems in this application involves estimating the positions and orientations (or poses) of parts that are randomly loaded into a bin (a.k.a., tote, tray, etc.) and providing a robot with the optimum part to pick, an estimate of the part’s pose, as well as the best path for the robot to follow to achieve that pick. The goal of this type of bin-picking is to then perform another task with the part directly. Examples include inserting a bolt into an engine block, placing a gear onto a shaft, or mounting a belt around two or more pulleys.
2. The performance of 3D machine vision systems under different lighting conditions. Differences in lighting conditions (such as fluorescent vs. LED) can greatly affect the performance of 3D imaging systems since these systems often operate within the visible and infrared portions of the electromagnetic spectrum. Identifying, classifying, and measuring the lighting conditions will enable NIST to develop standards that examine the effects of lighting conditions on the performance of 3D imaging systems.
3. The ability of 3D machine vision systems to resolve geometric features. Identifying and measuring the surface characteristics of objects (such as surface finish, material, texture, shape, and reflectivity) that impact a perception systems’ performance will allow NIST to develop standards that examine the effects of these characteristics on a 3D imaging system’s ability to resolve certain geometric features.
4. The ability of 3D machine vision systems to perceive and estimate the poses of atypical parts. Atypical parts may include (but are not limited to) parts that are flexible, deformable, transparent, and translucent, or parts with unusual surface properties. Flexible parts may include belts, chains, and cable assemblies. Such parts pose unique challenges for both perception systems and robots. For example, the perception challenges include the need for a standard representation of the pose of a flexible part, which can be continuously updated as the part changes shape while being handled. Transparent and translucent parts are challenging for most perception systems because many of these systems rely on being able to see (or image) the outer surfaces of objects.
5. The selection of 3D machine vision systems for robotic bin picking applications. Finally, the project will help develop guidance for the selection of 3D machine vision systems for robotic bin picking applications by working with industry experts. This could also result in standards that industry can use for various other applications.

Major Accomplishments

The NIST Bin-Picking Testbed

• Established four working groups within ASTM Sub-committee E57.23 on Industrial 3D Machine Vision Systems to develop standards for these systems.
• Established a NIST bin-picking testbed that incorporates five commercial bin-picking vision systems, one custom-developed bin-picking system, and two collaborative robotic arms.

Past Events

• NIST at the 2023 Automate Show
• NIST and ASTM E57 at the 2022 Automate Show
• NIST at 2022 Vision Show
• NIST Workshop on Standards for 3D Perception Systems for Robotic Assembly Applications

Relevant Publications

1. Rachakonda, P. , Saidi, K. , Franaszek, M. , Qiao, H. , Sweetser, J. , Khatoonabadi, A. , Dechow, D. and Rhodes, W. (2024), Towards the development of standards and performance metrics for 3D imaging systems, Proceedings of the SPIE Defense + Commercial Sensing 2024, National Harbor, MD, US, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=957648 
2. Franaszek, M. , Rachakonda, P. and Saidi, K. (2024), Filtering Organized 3D Point Clouds for Bin Picking Applications, Applied Sciences, https://doi.org/10.3390/app14030961, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=956513 
3. Franaszek, M. , Rachakonda, P. and Saidi, K. (2023), Improving Fitting CAD to 3D Point Cloud Acquired with Line-of-Sight Sensor, 2023 IEEE International Symposium on Robotic and Sensors Environments (ROSE), Tokyo, JP, https://doi.org/10.1109/ROSE60297.2023.10410732, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=936471
4. Rachakonda, P. , Cheok, G. , Franaszek, M. and Saidi, K. (2023), Evaluating the depth resolution of 3D sensors for manufacturing automation applications, Measurement, https://doi.org/10.1016/j.measurement.2023.113307, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=935408
5. Saidi, K. and Cheok, G. (2021), A Standards Roadmap for 3D Imaging in Robotic Assembly Applications, Advanced Manufacturing Series (NIST AMS), National Institute of Standards and Technology, Gaithersburg, MD,  https://doi.org/10.6028/NIST.AMS.100-39, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=931820
6. Saidi, K. and Cheok, G. (2021), A Standards Roadmap for 3D Imaging in Robotic Assembly Applications, Advanced Manufacturing Series (NIST AMS), National Institute of Standards and Technology, Gaithersburg, MD, https://doi.org/10.6028/NIST.AMS.100-39, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=931820 
7. Marvel, J. , Eastman, R. , Cheok, G. , Saidi, K. , Hong, T. , Messina, E. , Bollinger, B. , Evans, P. , Guthrie, J. , Hershberger, E. , Martinez, C. , McNamara, K. and Wells, J. (2021), Technology Readiness Levels for Randomized Bin Picking, Performance Metrics for Intelligent Systems (PerMIS) 2012 Workshop, Special Session, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, https://doi.org/10.6028/NIST.IR.7876
8. Cheok, G. and Saidi, K. (2020), Standards for 3D Imaging Systems for Manufacturing Applications, Imaging and Machine Vision Europe
9. Saidi, K., Cheok, G., Qiao, G., Horst, J. and Franaszek, M. (2020), Proceedings of the ASTM E57 Workshop on Standards for 3D Perception Systems for Robotic Assembly Applications December 2 & 3, 2019, Advanced Manufacturing Series (NIST AMS), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.AMS.100-33
10. Saidi, K. (2020), The Need for 3D Imaging Performance Standards for Robotic Assembly, Quality Magazine, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=929811