Modernizing First Responder Training with VR and Speech Recognition

With over 60 million American adults owning a smart speaker, human-to-machine conversation have largely become a part of a daily ritual. Though these exchanges are seemingly simple to the user, the complex technology that backs the interactions is powered by artificial intelligence, machine learning, and speech recognition. These modern machines are only continuing to grow in popularity, and for many Americans, voice recognition technology is becoming as natural as owning a smartphone. The user interfaces that drive the functionality of smart speakers have brought new and improved convenience to modern-day Americans. By the same token, this capability has the potential to alleviate burdens from the operations of our nation's first responders.

The subject of PSCR award recipient Health Scholars' project is to bring similar voice-directed user interfaces to Public Safety to increase training efficacy. For public safety departments, training exercises can be time-consuming, costly, and labor-intensive. The Health Scholars project is intended to bring experience-based training to first responders in a scalable, accessible, and affordable way with innovative technologies. Their project is designed to investigate the use of virtual reality (VR) training for Public Safety by focusing on one of the most natural things first responders do: talk.

The Benefits of VR

Health Scholars is focused on reimagining Advanced Cardiac Life Support (ACLS) training for first responders through the use of VR and speech recognition technologies. This cardiac emergency training is a certification that healthcare professionals specializing in emergency services must renew every two years, accredited by the American Heart Association. "Emergency and fire agencies need to deliver ACLS training every year," explains Chris Ingwalson, "and they all do it pretty similarly. It requires an entire team to be offline for a full day, often on their day off. For a mid-size fire department, that can look like 20 paramedics and four instructors for a full day, which costs a lot of money and is very labor-intensive."

To combat the high time, labor, and financial costs associated with the traditional simulation training, Health Scholars designed a virtual, in-home environment intended to recreate a cardiopulmonary emergency call, providing a low-cost, easy-to-use alternative. VR technology offers a unique opportunity for Public Safety as it allows immersive, repeatable, and measurable training scenarios to first responders. "We've basically created a simulation of an ACLS skills exam," Ingwalson explains. Effectively replicating a public safety scenario for training purposes can be difficult to execute because many emergency response scenarios are high-stress and time-sensitive. However, VR can serve as a potential tool to simulate high-pressure situations in a realistic and immersive way while maintaining a low-cost.

Voice-Driven Training

In Health Scholars' virtual scenario, users wear a VR headset and speak into the built-in microphone to direct a team of virtual avatars through a cardiac arrest. Leveraging voice recognition and motion capture technologies, users can command their virtual team members to shock, perform cardiopulmonary resuscitation (CPR), and administer medications. "Our user interface is completely voice-driven, and the simulation does not require any hand controls or any type of click actions. That's a way that our application stands out and it's also a way that first responders can easily understand and use the technology without being afraid of the technology barrier of VR," Ingwalson explains.

With this VR-based training module, Health Scholars envisions a training set up in a public safety station, allowing first responders to schedule their training in their free time. Ingwalson states; "The benefit is that training can be completed on-site and whenever it is most convenient for the learner. Agencies can save money and still provide the instructors and those responsible for the training a standardized way to complete it and assess performance." In addition to the time and cost benefits, Ingwalson also speaks to the unique benefit to small, rural agencies. "Rural areas tend to have smaller call volume. So, paramedics can go a few months without seeing a cardiac emergency. For these agencies, VR training is an affordable to deliver refresher training and ensure competencies are maintained."

Public Safety Partnerships and Advocates

Health Scholars partnered with Clear Creek EMS, Arvada Fire Protection District, and Thornton Police Department to ensure their product would be useful and helpful to first responders. "Our partners have given us an enormous amount of direction as subject matter experts," Ingwalson explains, "They have also helped us with design and usability questions as well as data for training our voice translation engine." Health Scholars' president is an emergency room physician; so, the team originally began to tackle their application design from an in-hospital perspective. However, through the Public Safety Innovation Accelerator Program User Interface (PSIAP-UI) award, Health Scholars worked closely with first responders to tailor their product to the particular needs of a public safety use case. "Their attention to detail in creating this app was far beyond what we expected; if we hadn't won the award we may not have had the subject matter expertise we needed or ended up with these great partnerships and advocates within the first responder world - that's a huge benefit for us."

The PSIAP-UI award allowed Health Scholars the unique opportunity to test their idea to use voice technology for enhanced usability. "If we hadn't won the award to explore the voice technology, I think we could have gone down a traditional pathway to use hand controllers. The PSIAP-UI award allowed us to test what we thought would be a better usability decision and we proved that it was." Recently, Health Scholars secured Series B funding in the amount of $17 million that will allow them to continue to expand their technology applications to healthcare and public safety professionals. "PSCR made a huge difference for us, we thought voice technology was interesting, but it was difficult and costly. PSCR allowed us to get over the cost hurdle and prove to ourselves and the market that it can really work."