https://www.epo.org/en/node/alex-kipman

Alex Kipman

HoloLens: mixed reality smart glasses

Finalist for the European Inventor Award 2018

Consumers may soon interact with the real world – and each other – through a new generation of data glasses. Developed by Brazilian software engineer and hardware inventor Alex Kipman and marketed by Microsoft, the HoloLens mixes reality with hologram-like overlays. Business people and hospital patients alike stand to benefit as applications like holographic phone conferencing and computer-assisted surgery adopt the technology.

Consumers today have access to a wealth of information through their tablets and smartphones. But, on the flip side, they also spend close to ten hours a day looking at computer screens. That's why computer expert Alex Kipman at Microsoft wants to direct the human gaze back to the real world, without losing access to computing power.

The inventor's mixed-reality data glasses, called the Microsoft HoloLens, blend views of the real world with digitally overlaid "holograms". Wearers can view and manipulate these lifelike images - for instance the three-dimensional image of another person - while still seeing the real world around them.

Loaded with computing power, the HoloLens reads the environment through the inventor's patented sensors. Users control the device without keyboards, game pads or touchscreens. Instead, the HoloLens can interpret hand gestures and voice commands thanks to built-in machine-learning algorithms. The resulting mixed-reality experience integrates computers seamlessly into daily activities - both at work and play.

Societal benefit

Some industry insiders believe that mixed-reality devices, like the HoloLens smart glasses, could replace smartphones. Experts at Gartner predict that, by 2020, 30% of web browsing sessions will be done without a screen. Major drivers for the technology include consumer demand for assisted navigation, gaming and live translations. In fact, by 2020, 100 million consumers will rely on augmented reality - as mixed reality is sometimes called - for personal shopping assistance (comScore).

Developers are currently creating a host of applications for Kipman's data glasses in the Windows Mixed Reality software environment of Windows 10. For example, the thyssenkrupp industrial group includes the glasses in their metering and quality control operations, while the famous American medical institution Cleveland Clinic offers an interactive digital human anatomy curriculum. In 2017, while operating on a patient to remove a malignant muscular tumour, Spanish surgeons used the HoloLens to simultaneously view live diagnostic pictures from magnetic resonance imaging and radiography data feeds.

Economic benefit

Perfected over the course of seven years, the HoloLens was first introduced by Microsoft as a prototype in 2015. It features Kipman's patented sensors for the company's Kinect video game controller. The HoloLens is currently available as a development edition aimed at software programmers for EUR 2 556. An enterprise version costs EUR 4 260.

Microsoft has sold around 50 000 HoloLens units, relatively few compared with sales of over one million virtual-reality headsets in the third quarter of 2017 alone. But the corporation, which earned EUR 73 billion during the fiscal year 2017, is strategically "seeding" the market for mixed-reality products through strategic development partnerships with companies such as Lowe's, Boeing, Saab and Volvo.

Consulting firm Digi-Capital estimated the world market for augmented- and virtual-reality devices at EUR 3.3 billion in 2016. The firm projects that, driven by next-generation mobile augmented-reality hardware, the market will reach EUR 85 billion by 2021. Once Microsoft has built an attractive mixed-reality library, products such as the three-dimensional teleconferencing app Holoportation could well join bestsellers like the Office 365 commercial suite with over 100 million users.

How it works

Current-generation mixed-reality glasses include "smart" ski goggles that project data such as altitude, temperature and pulse into the wearer's field of vision. But these devices display only two-dimensional text and graphics, while requiring a connection to a smartphone for computing power.

The HoloLens is a stand-alone device, independent from external PCs, tablets or smartphones. It contains all necessary computing power in its custom-built processors. These include a CPU (central processing unit), GPU (graphics processing unit) and custom HPU (holographic processing unit), an industry first.

These processors collect several terabytes of data every second as the HoloLens reads its environment. Data arrives through an inertial measurement unit (IMU), a depth-sensing camera, a 2.4-megapixel video camera, four microphones and an ambient light sensor.

As the name suggests, the HoloLens owes its innovative edge to the waveguide lenses that "trick" the human brain into interpreting photons as solid, three-dimensional things - hence holograms. They are computed on the GPU and displayed through the device's "light engines", two tiny liquid-crystal-on-silicon projectors which beam light onto the two lenses.

The image is adjusted between the two lenses for clarity and depth, then blended with the real world. As a result, holographic images appear to exist in front of the wearer - from a few centimetres to many metres away - with true depth of field.

The inventor

Born in Curitiba, Brazil, Alex Kipman fell in love with computer programming by playing video games at the age of six. "In software, you are only bound by the limits of your imagination," says Kipman. He joined a global company commensurate with his imagination when he joined Microsoft in 2001 after earning his Bachelor of Science degree in Computer Software Engineering from Rochester Institute of Technology.

The inventor landed his first smash hit - and became a star of the tech community - when his Kinect motion sensor broke all records as the fastest-selling consumer electronics device in 2008. So far, he is the primary inventor on more than 150 US and Asian patents as well as three granted European patents, including the technology behind the HoloLens.

Today, as Technical Fellow of the Windows and Devices Group at Microsoft, Kipman remains closely involved in developing new technology for the HoloLens and its related Windows Mixed Reality software platform. At just 40, his honours already include a Popular Mechanics Breakthrough Award (2009) and being named the IPO Education Foundation's Inventor of the Year (2012).

In his private life, the outspoken futurist likes to retreat to the simplicity of his family's off-the-grid farm in the Brazilian rainforest. Unplugged from digital devices, he seeks out real-life outdoor experiences with his wife and daughter.

Did you know?

The HoloLens takes a time-honoured technological principle into the age of wearable computers. As early as the 16th century, theatres employed a technique to project images onto angled sheets of transparent glass for visual effects. In the 1940s, projecting text onto angled glass became the principle behind heads-up displays used in fighter jets. Important data such as altitude and speed could be projected into a pilot's field of view, allowing full focus on the action without distraction. The same principle has been applied for decades in autocues to display text to a speaker, unseen by the audience.

Fast-forward to the 21st century and Kipman's HoloLens can make people's holographic likeness appear in thin air with new levels of sophistication - for instance during teleconferencing calls. And because the mixed-reality headset requires no computer screen or keyboard, it could be part of a revolution that would replace smartphones as the main communications devices.

Looking into a not-so-distant future, Kipman has predicted the "death" of the smartphone. Current industry trends appear to support his statement - such as the shift towards voice search with digital assistants like Cortana from Microsoft, Siri from Apple and Alexa from Amazon. Consumers may one day be talking and gesturing to holographic "ghosts", while the smartphone becomes a spectre of the past.