A partnership between Microsoft and Volkswagen will use the HoloLens 2 mixed reality headset as one of the key components of future mobility concepts. Together, the two companies want to enable HoloLens 2 in moving vehicles for the first time.
Microsoft has been working on a Moving Platform Mode (MPM) for the HoloLens 2's tracking to work on low-dynamic motion moving platforms for quite a while now. Since the HoloLens uses visible light cameras and an IMU for tracking, the headset needs the environment to remain stationary relative to Earth for it to work correctly. However, this does not stand true in many common and high-frequency-motion environments, such as planes, trains, cars, bikes, buses, small boats, elevators, etc.
Launched in October 2021, the MPM remedies this issue by telling the tracker that it can't rely on the headset's sensor inputs to agree with each other at all times completely. Instead, the HoloLens needs to rely more heavily on visual tracking and quickly identify incongruous inertial motion data and filter it out accordingly before it's able to use the IMU input.
As an early adopter of augmented reality technology, Volkswagen was already using an augmented reality head-up display in its ID. family of electric cars in 2020 to superimpose navigation arrows, lane markings, and other information onto the environment.
"We think mixed reality information is the most intuitive information we could provide to enhance our customers' user experience," said Dr. Andro Kleen, head of the data science team at Volkswagen Group Innovation. "Because what you see there, and what you need to process, is very close to what humans normally see and process. It's not so abstract."
However, the interest in Microsoft HoloLens came up during a research project that started in 2015 and used self-piloted vehicles and augmented reality to teach driving on a racing circuit. Volkswagen tried to use the HoloLens but soon found out the device wouldn't work on a moving vehicle and approached Microsoft to find a solution. In 2018, both companies started collaborating on MPM's development for the HoloLens 2.
During test phases, Volkswagen established a bidirectional data connection between the vehicle and the HoloLens to display and control real-time information from the car and implemented several demo use cases investigating how virtual interfaces could enhance the interior of future vehicles.
"We connected a positioning system that tracks the location of the vehicle. This way, we were able to also place 3D elements such as information on points of interest outside of the car. This opens up completely new possibilities to not only display holograms within the driver's forward-facing field of view, but also wherever the user wearing the glasses is looking," says Michael Wittkämper, augmented reality expert at Volkswagen.
Currently, the only supported environments for MPM are large marine vessels experiencing low-dynamic motion, and even then, holograms might wobble slightly, especially when on choppy water. Maritime companies and organizations are taking advantage of MPM and Microsoft Dynamics 365 Remote Assist to remotely connect maritime workers with mechanical experts. In the future, Microsoft and Volkswagen expect MPM to be shared across many other industries.
