First published in the Daily Maverick 168 weekly newspaper.

The augmented reality (AR) headset is a technological wonder in itself. In a lightweight frame that fits comfortably over your eyes, it has the entire computer built-in (mostly in a largeish, curved case on the back of your head).

The lenses depict AR images, or holograms, that are amazingly good, while sensors in the eye sockets track your eye movement. So, for instance, if you’re reading a page (of instructions or just email), it scrolls the text up as your eyes move down so you can seamlessly read it.

Other sensors track your hands as they move – and allow you to move objects in AR or play the piano, in one example. And the headset does this without additional controllers in your hands or clumsy tagged gloves.

It’s an amazing experience, as I discovered on a trip to Microsoft HQ in Seattle a few years ago, and at the launch of the HoloLens 2 at MWC Barcelona in 2019.

I’ve been a tourist on Mars and fixed a huge industrial metal machine (where the broken part was visible over the actual physical machine, and I could see where to adjust it and screw it back into place).

The use cases are quite fantastic. As the son of an architect – hi ma! – I know how much goes into conceptualising a building, or a factory. One current commercial use is for construction, where architects and engineers can plan buildings in a remarkable new way.

Microsoft calls it mixed reality, which is an apt description of being able to see the real world but with an overlay of useful information, such as visual instructions on how to fix a broken machine or how to put together the cab of a transport truck.

Last month, surgeons used it to demonstrate to their colleagues around the world their techniques and procedures: visualise a patient lying on an operating table, with a surgeon (wearing a HoloLens2) seeing bones, blood vessels and organs overlaid on their body.

South African surgeon Professor Stephen Roche – from Groote Schuur Hospital and the University of Cape Town – was one of 15 surgeons from 13 countries who performed 13 mixed-reality orthopaedic operations.

What makes the HoloLens2 so astounding is that it is infinitely more powerful when it’s part of a network – and that network can create a shared, group hologram, nogal.

Microsoft Mesh is a brilliant way to tie all of the HoloLens’s individual functionality into a remarkable system that “allows people in different physical locations to join collaborative and shared holographic experiences”.

Everyone is in the same hologram, and are able to interact with each other.

“This has been the dream for mixed reality, the idea from the very beginning,” said Microsoft Technical Fellow Alex Kipman. “You can actually feel like you’re in the same place with someone sharing content or you can teleport from different mixed-reality devices and be present with people even when you’re not physically together.”

Microsoft launched this concept by staging its annual Ignite conference, held virtually because of Covid, using Microsoft Mesh for the presenters. Kipman appeared as a “fully realised holoportation of himself”, as Microsoft later described his holographic version.

Kipman was joined by Hollywood filmmaker turned ocean explorer James Cameron and John Hanke, the CEO of AR company Niantic (which makes that runaway AAR successful game, Pokémon GO) to show off how they were using mixed reality to combine shared experiences. The prototype of the Pokémon GO game was very cool.

Mesh is possible because Microsoft has grown Azure into a cloud computing behemoth that hosts sexy applications like this.

“In these collaborative experiences, the content is not inside my device or inside my application.

“The holographic content is in the cloud, and I just need the special lenses that allow me to see it,” said Kipman.

“This is why we’ve been so passionate about mixed reality as the next big medium for collaborative computing. It’s magical when two people see the same hologram.”

I can’t wait to try it out. DM168

This story first appeared in our weekly Daily Maverick 168 newspaper which is available for free to Pick n Pay Smart Shoppers at these Pick n Pay stores.