11/6/2024 Lilli Bresnahan
Written by Lilli Bresnahan
Illinois’ Grainger College of Engineering Jump ARCHES Research Program has approved $100,000 funding for phase 1 of the development of a mixed reality simulator for neonatal needle thoracentesis, led by Avinash Gupta, a teaching assistant professor at The University of Illinois Urbana-Champaign. Nicole Rau, an assistant professor of clinical pediatrics at the University of Illinois College of Medicine and Peoria and a clinical Neonatologist at OSF/ Children’s Hospital of Illinois, and Muhammad Jawad Javed, the Neonatology Fellowship Program Director and Medical Director Professor of Clinical Pediatrics.
Jump Applied Research in Community Health through Engineering and Simulation (ARCHES) Research Program was established in 2014. This program is an endowment partnership between Jump Simulation and Education Center at OSF HealthCare and The Grainger College of Engineering at U. of I. It aims to provide direct access and grants to engineers and physicians to fight against problems within HealthCare.
In 2023, Jump ARCHES funded the researchers’ Phase 0, which developed a guided Simulator using Microsoft HoloLens 2 for needle thoracentesis training. Researchers are improving upon discoveries from Phase 0, transitioning from a marker system to a marker-free system after realizing the former presented a serious problem for the use of small, one-time-use instruments that are common for neonatal procedures.
A marker system relies on specific visual markers to help the system discern the physical environment, for example QR codes. In contrast, marker-free systems use cameras, computer vision, and depth sensors to recognize and map the physical environment. Since the surgical parts are too small to be correctly identified by markers, a marker-free system is necessary. A marker-free system reduces the intrusiveness as the patient’s situation is highly sensitive. The advanced sensor technology and computer vision records in real-time the spatial tracking and interaction in mixed reality environments, which is critical in neonatal care where the smallest movements are significant. The new system is meant to augment accuracy and applicability in clinical settings, guaranteeing a smooth and realistic training for healthcare providers.
“To meet that goal, in Phase 1 we will focus on collecting data for Computer Vision and ML based object detection, tracking and guidance system,” according to the project’s executive summary. The collection of correct and incorrect procedures will be utilized as computer vision-based feedback for the simulator, which is one of the goals of phase 1.
“As far as Phase 1 of the project is concerned, one of the fundamental goals is to both create a database that helps us better understand the neonatal needle thoracentesis process and use this very database to support the development of the feedback-based simulator,” Gupta said.
Phase 1 will analyze the usability and engagement of the guided mixed reality simulator by recruiting experts in needle thoracentesis to evaluate the simulation. As more advanced technology comes to the market, such as Quest pro/Quest 3, the scientists will reevaluate and modify the established guided simulator, deciding in collaboration with these experts, the best device for this simulation.
Using the Microsoft HoloLens 2, they will develop a mixed reality platform to provide simulation training for various neonatal procedures. This technology will benefit trainees in pediatrics, family medicine, OBGYN, emergency medicine, and assist current practitioners with maintaining their high-leveled professional skills. In the future, the aim is for this platform to be used across any procedure among any specialty.
Due to the high mortality rate of newborns with a pneumothorax and because the physical simulator of the procedure exists, scientists focused on thoracentesis to advance the mixed reality simulator. Thoracentesis is a procedure which takes fluid out of your lungs, and it is especially common in newborns. The advancement of this project will ensure that community providers can maintain their skills in neonatal procedures without traveling to attend simulation workshops in academic centers.
Instead of the use of physical mannequins, which still require some type of teacher to guide the process, these scientists thought of a digital solution to train the physicians for their medical procedures. This has already been explored in virtual reality; however, now, these scientists are exploring mixed reality, allowing users to interact with virtual and physical environments simultaneously. "The use of this technology allows learners to have the haptic feedback of physical mannequins, while simultaneously receiving instructions and feedback from the MR headset overlaid on the physical environment," the summary said.
This innovative approach to training healthcare providers could address the gap in neonatal resuscitation proficiency, empowering healthcare providers in community hospitals with the skills and confidence they need to save lives.