Augmented Reality, Mixed Reality, Virtual Reality, and the Future of Surgery: What type of headset will my surgeon wear in the operating room?

By Benjamin Service, MD

The future of surgery will be filled with new technology and innovations with the goal of helping patients achieve better and safer outcomes. Multiple companies are pioneering the use of specialized tools, including mixed reality headsets during surgery. This is not just the future of surgery—it is the cutting edge of surgical care today! Additionally, virtual reality goggles are being used in disruptive ways to help medical students, residents, and surgeons train for surgery with complex, immersive simulations. This type of technology is transformative for how we approach not only medical education but also the entire surgical field.

Before diving into applications of these disruptive innovations, let’s get on the same page about what we’re talking about. Big picture: there are three types of “reality” that have subtle but important differences.

1. Augmented reality is when digital information is overlaid onto real objects or places in the world. The idea is that the digital information enhances the user experience to make it more engaging and memorable. Augmented reality can use a heads-up display or more frequently use a device like a cell phone to see the digital content. This type of technology, also called AR, has been used for popular cell phone games such as Pokémon™ GO and can change the way we perceive normal or objects in the real world with digital overlays. Using AR apps can be fun and very useful!

2. Mixed reality (MR) applications are like augmented reality, however, there is an opportunity to connect the digital and real world with mixed reality. When using mixed reality, the user wears a special type of goggles with a heads-up display that places the digital objects into the real world. Unlike AR, mixed reality allows you to interact with those objects. In the case of surgery, you can interact with x-rays or surgical tools. The ability to interact with both the digital and real world will have many applications in surgery. Mixed reality is actively being used by surgeons in the operating room in 2023. Using a MR headset is something like stepping “inside” of a computer. Three-dimensional anatomy is even more compelling when I can literally reach out with my hand and (digitally) touch and move the shoulder replacement in the operating room.

3. Virtual reality, also called VR, is very different. The entire experience is digital without a focus on connection to the real world. For VR experiences, the user wears a pair of “goggles” that you can’t see through–all you see are 2 small screens that show a single virtual and completely digital experience. VR can be extremely immersive. Imagine being inside of a movie where you can interact with objects in that movie, drive a car, pick up tools, or even do surgery. VR is currently being used as a training and educational tool for high-risk situations. For example, VR is being used to help surgeons and trainees learn about surgical procedures without any risk of harm to patients and can be done anywhere, anytime. Future applications of virtual reality may be used in a variety of ways for surgeons, residents, and patients too.

Surgeons at the Orlando Health Jewett Orthopedic Institute are currently investigating the use of mixed reality in the operating room during shoulder replacement surgery. I use an augmented reality headset when performing anatomic shoulder replacement, reverse shoulder replacement, and revision shoulder replacement surgeries. The mixed reality goggles let me to see my patients’ anatomy on 3D planning software as a hologram right in the operating room! The software allows me to connect to my patients’ unique problems while performing the surgery and without physically touching any part of a computer. I work with the MR system virtually using my hands, voice, and by moving the headset itself. When using this software, the heads-up display shows the shoulder anatomy in space. I can walk completely around the 3D holographic digital shoulder image; it is like being inside of a computer! Instead of a mouse or controller, I use my hands to virtually type on a holographic keyboard, navigate menus, study the patient’s anatomy, and check the pre-planned size of the joint replacement.

The current mixed reality software allows me to connect with my patient’s problem in a unique way. In the future, this technology will also enhance accuracy and consistency by guiding surgeons to the ideal location for placing the components for a shoulder joint replacement. The goal of guided surgery is to help surgeons to achieve better outcomes by enhancing their accuracy and consistency.

Augmented reality will likely have many applications for physicians and all healthcare providers to help us better engage our patients. It could be used in many ways to help doctors have important information displayed but without the interference of a traditional computer screen or mobile device.

Virtual reality is growing in popularity not only as a gaming or social tool, but also has a valuable educational tool. Simulating surgery with VR experiences helps residents, fellows, and surgeons prepare for the operating room. Virtual reality education improves access for learners to high-quality simulation without the costs, infrastructure demands, and supply chain challenges of a cadaver lab. Another powerful feature of VR software is collaboration. Imagine virtually standing next to a surgeon (physically located in another part of the world) that can interact with you and the digital environment as if they were right there with you. These simulations can bring you virtually into the operating room to participate as lead surgeon in a virtual shoulder replacement surgery! Medical education has many opportunities to expand the use of VR to help empower and educate our next generation of physicians.

Technology is advancing rapidly. Augmented reality, mixed reality, and virtual reality will likely all have a place to help doctors get better. To answer the question in the title, a mixed reality headset is not only likely to be the headset of choice in the future, but they are in use now. The current applications for these technologies are the tip of the iceberg compared to where we will be in 10 years. It is an exciting time to be a surgeon.

Benjamin Service, MD, is a fellowship-trained, board certified, orthopedic surgeon specializing in shoulder surgery, elbow surgery, orthopedic sports medicine and orthopedic trauma.

Dr. Service received his undergraduate degree in biology from Florida Atlantic University, graduating magna cum laude. He received his medical degree from the University of Florida College of Medicine and was inducted into the prestigious Alpha Omega Alpha Honor Medical Society and Gold Humanism Honor Society. He served his residency in orthopedic surgery at Orlando Health and completed his fellowship in shoulder and elbow surgery at the University of Washington. He can be reached at (321) 843-8551 or – Benjamin.service@orlandohealth.com

Orthopedic Doctor Orlando - Elbow Surgeon - Shoulder Specialist (orlandohealth.com)