Potential of AR & MR with a Futuristic Tech PoC for Better Healthcare

With new advancements in hardware and software technologies, Mixed Reality (MR), Virtual Reality (VR) and Augmented Reality (AR) are being harnessed for various retail and commercial applications across manufacturing, education, healthcare, remote support, etc. in the real world. At Xoriant, we are working on Proof-of-Concepts (PoCs) to deliver advanced immersive tech-based solutions to our clients.

This blog dives into the basics of immersive tech and includes insights from an exploratory PoC project. Read about a proposed innovative and futuristic solution for the medical fraternity using the latest technologies in AR and MR to transform healthcare services.

The World of Immersive Technologies

Long ago, creative and fiction writers wrote about immersive technologies in Sci-Fi novels and comics. Today, technology advancements in the 3D and 2D space have reached a whole new level. VR and AR became a trend earlier and now, we are talking more about Mixed Reality (MR) and Extended Reality (XR).

As per IDC, global spending on AR and VR will rise from just over 12 billion dollars in 2020 to 72.8 billion dollars in 2024. The commercial use cases for these spends will include training, industrial maintenance, and retail showcasing.

Today, the availability of new hardware is enabling creative and technology experts to employ new-age technologies for building powerful applications. One of the new technical frontiers is Mixed Reality that involves merging the virtual world with the real world. Companies are trying to achieve an immaculate blending of the digital and the real world to provide the end-user with a surreal experience. Mixed Reality also allows businesses to blend the real world with fictional and futuristic worlds while tapping numerous possibilities that can leave the stakeholders awe-struck.

Let’s take a closer look at the different areas of immersive tech.

Virtual Reality (VR):

Virtual Reality

It’s a world created through 3D/2D software. With the help of a headset (HMD - head-mounted display), you can experience the 3D/2D world and interact with objects present in it. This technology has already made heads turn in the space of gaming and simulation technologies. The VR experience becomes richer with controllers to interact with the virtual environment.

Augmented Reality (AR):

Augmented Reality

It’s a technology that enhances the real-world environment with 3D/2D/CG objects. Here, the major difference is computer-generated graphics (CG) elements do not respond or recognize real-world environments i.e., no interaction takes place between the real-world and CG objects.

Mixed Reality (MR):

Mixed Reality In MR, the distinction between virtual- and real-world fades away with innovations like Microsoft HoloLens and Magic Leap 1. Things look merged into each other and CG objects interact with real-world objects, which makes it difficult to segregate real and virtual objects. With the help of next-generation sensing and imaging technologies, one can interact with real and virtual world objects inside an MR environment. These technologies are useful in Gaming, Travel, Film and TV, Healthcare, and Sports domains.

Extended Reality (XR):

Now, the possibilities of immersive tech are being pushed with XR. Solutions merging real and virtual worlds by combining AR, MR, VR, and other future tech come under Extended Reality.

Using Adobe Aero for immersive AR

Different software such as Adobe Aero, Amazon Sumerian, Wikitude, Vuforia, etc. are available today to create engaging immersive content. Take, for instance, Aero, the Augmented Reality software by Adobe, which can be used to build an immersive AR experience for iPhones and iPads – it enables us to think beyond creating abstract arts and look at solving real-life problems.

At Xoriant, we considered a medical/healthcare use case and applied design thinking to solve a few problems with Adobe Aero. This was a Proof-of-Concept project to understand the technical feasibility of an innovative AR-based solution. The economic and functional viability of the experiment is yet to be gauged.

We chose to go with Adobe Aero due to its ease of use as well as the availability of supporting tools like Photoshop, Dimension, Mixamo, Substance Source, which were also from Adobe. Our team also tried using Blender 3D (an open-source 3D modeling application) for creating the models.

Xoriant’s AR and MR Proof-of-Concept for the Healthcare Industry

A major challenge in the medical field is the “time constraint” faced by professionals at healthcare centers to attend to patient needs. Doctors, nurses and other healthcare workers have to review the latest health records of a patient to prescribe medicines and suggest treatments accordingly. Real-time patient monitoring with connected medical devices and immersive technologies are being explored to minimize response time to patient diagnosis and support.

Xoriant’s PoC project explores a solution to aid surgeons with access to consolidated patient information on the AR screen. This way, an enormous amount of time can be saved which they otherwise invest in looking through physical files of medical records. Time is also lost when patients in each hospital room have to fetch test reports.
Instead, if readings on the AR screen can be updated with synchronized real-time patient monitoring and other MedTech devices, the healthcare worker can quickly access the records. With access to a patient's real-time health status on the AR app, doctors can start treatment and suggest remedies quickly.

The stepwise workflow and insights from this PoC project are shared below:

Step-1: Research the industry and understand their terminologies

To create an AR experience with real-time updates, we need to understand the industry and its precise needs. To propose an AR-based infrastructure for healthcare centers requires a good understanding of medical terminologies and approaches used by hospitals.

We did extensive research and created a comprehensive document that included descriptions and screenshots explaining different terminologies. This document includes details such as “What does a basic physical examination include” and “Medical History”. During this research, we learned in-depth about terminologies like History of Present Illness (HPI), Past Medical History (PMHx), and more.

Step-2: Storyboard for the PoC demo

Previously all medical records and charts came in paper forms, but many hospitals have now converted them into EMRs (Electronic Medical Records). Due to this digitization, our proposed solution to show all these charts in the AR program will be easier.

For this demo, we created a storyboard that was a guiding principle during the process. We used Adobe Aero to create the AR program. The plan was to run this program on an iPad which can replace the physical patient records folder and notepad in the patient’s room.

Step-3: How to run the AR app on an iPad?

As per the planned process, we would replace the paper notepad (health charts) on the patient's bed with an iPad and would run our AR program on this iPad as per Figure-1. We took a reference image of an ICU ward from hospitals where different types of advanced medical devices are usually present. For this experiment, reference images and random data were used. POC Figure 1 Figure-1

There are two ways to show health charts in the AR program,

Through menu buttons (Figure-2)
By directly generating buttons on the medical devices with QR codes pasted on the medical machines available inside the ICU (Figure-3)

Based on a discussion with our internal subject experts, we decided to proceed with the second option with a QR code. POC Figure 2

Figure-2

POC Figure 3 Figure-3

Step-4: Creating the AR solution using Adobe Aero

After receiving approval from all stakeholders, we started demo creation with Adobe Aero.

We created 3D buttons and defined the behavior of the buttons in the AR environment. Pictures of a hospital ward with health monitoring devices were used to create the AR environment.
In this experiment, when a user taps on these buttons, the relevant screenshots of various reports will be visible. If we consider applying this experiment in real life, live data can be fetched through IoT (Internet of Things) sensors with the right infrastructure.
On the same digital screen, we included an interactive menu so users can switch screens and check different medical details of the patient. We can also place a notes button that will enable doctors to add their comments/prescription on any of the given charts.

While putting the above PoC into practice in the real world can be challenging, the results of this exploratory project look promising. Some of the key considerations for turning this PoC into an actual healthcare solution include:

Reliable and easy-to-use connected devices are needed for fetching accurate data.
Safety protocols have to be followed while using the device in hospital rooms where patients with various health conditions are undergoing treatment.
Seamless connectivity is needed for the setup to work effectively.
As there are no set standards, and best practices are still evolving in the immersive tech space, learnings from PoCs can be used for creating a strategic plan along with estimated investment and outcomes.

Potential Applications of Immersive Tech

A PoC approach – from concept plan to demo – can be explored to solve many problems in the field of Data storage, Medical and other training, Online shopping, Construction and Logistics, Repair and Maintenance, Entertainment, Tourism, and many more.

At Xoriant, we are also working on other PoC projects around AR-based apps and integrations with IoT sensors for fetching real-time data from devices in industrial environments. Exploring the possibilities allows us to stay innovative and deliver cutting-edge solutions to clients.

Our innovators continue to explore future-focused solutions to enable businesses with the required technical expertise for taking creative ideas from PoCs to viable business outcomes across domains.

Whether you want to be an early adopter of futuristic immersive tech or need a solution around AR, VR, MR, and XR, you can talk to our experts at Xoriant. Write to info@xoriant.com