ARCHANGEL

Pison Technology Inc.

In April 2021, Pison Technology Inc. was awarded $1.2M for the Public Safety Innovation Accelerator Program: Augmented Reality (AR) cooperative agreement.  

Investments and technology advancement for the operational use of AR predominantly focus on hardware, software, and what end-users see through head-mounts or other displays. However, significantly less focus has been placed on the Human-Machine Interface (HMI) that enables users to interact with an AR system and its digital content, with even less insight into what makes AR HMIs effective for public safety organizations (PSOs) given their unique operating contexts.

That is why Pison is developing the Augmented Reality Control with a Handsfree Activated Neural Gesture Engagement Library (ARCHANGEL). ARCHANGEL provides a single wrist-worn solution with the capability to map intuitive finger, hand, and arm gestures to the control of different AR displays and AR controlled systems. It does this by using proprietary signal filtering and machine learning classifiers that characterize unique gestures of the fingers, wrist, and arm by monitoring micro-voltage changes in the neuromuscular system and motion of the hand, wrist, and arm. These gestures are then mapped to the intuitive and productive control of electronics using software development kits (SDKs) and developer resources. 

Under the ARCHANGEL program, this will involve mapping gesture controls to AR headsets, remote unmanned aerial and ground vehicle controls, and pass-through AR smartphone and tablet applications to understand what, when, where, and how gestures and AR capabilities can be effectively combined to empower first responders to perform their jobs safely and effectively.

Meet the team

• Principal Investigator Dr. Michael Jenkins has experience in the application of cognitive systems engineering, human factors, and user-centered techniques to design user interfaces, visualizations, hardware, and other technology-driven solutions that extend the capabilities of end-users in an intuitive and accessible manner.
• Experimental Lead Caroline Kingsley has experience in planning and execution of objective human subjects research, with a focus on characterizing the usability and utility of non-traditional human-machine interfaces (HMIs), including augmented and virtual reality (AR/VR) solutions.
• Technical Lead Arthur Wollocko has experience in VR/AR engineering using leading game development engines (e.g., Unity, Unreal) and leading engineering teams through the development of AR experiences.

Project overview

ARCHANGEL is powered by Pison’s wearable sensor, the Gambit, that is worn on the top of the wrist like a watch to allow individuals to use down to millimeter-scale hand and finger movements to control AR applications and other electronics.

The primary objective of this award is to conduct research to evaluate the usability and impact of, and establish standards for, different contextually calibrated AR HMIs for first responders focused capabilities. The team will work with a nationwide network of PSO first responders as well as the National Guard to design and validate gesture control standards for five distinct classes of first responder AR applications that can then be employed by any technology offeror.

Potential impacts of the project

ARCHANGEL has the potential to improve the general usability and near-term field readiness of the ecosystem of AR applications and technology built around the public safety use case. By providing a common HMI solution to interact with different AR-enabled technology capabilities, ARCHANGEL will address the gaps to fielding AR solutions and yield immediate and tangible benefits to first responders. 

For example, first responders will benefit from the ability of ARCHANGEL to support gesture control of AR interfaces while: 

• Holding an object
• Manipulating a vehicle/tool/weapon
• Wearing gloves or other personal protective equipment (PPE)
• In any body posture (e.g., standing, kneeling, sitting, prone)
• The hands are in a natural position (e.g., by your side vs. help up and extended in front of your face)

Additionally, Pison is actively engaged with a number of AR original equipment manufacturers (OEMs) and content providers (e.g., Microsoft, Google, Galvion, Overmatch, BadVR, Boston Dynamics) with potential to immediately adopt outcomes of the ARCHANGEL effort to help transition the outcomes and get solutions in the hands of our first responders.