Bringing the physical and digital worlds together: Alex Kipman named European Inventor Award 2018 finalist
- Brazilian inventor Alex Kipman nominated for European Patent Office (EPO) prize for developing "mixed-reality" headset
- Kipman's HoloLens blends views of real world with digital holograms
- Mixed-reality devices market could reach EUR 71 billion by 2021, potentially replacing smartphones as main internet gateways
- Applications include computer-assisted surgery, assisted shopping, live translation and interpersonal communications
- EPO President Battistelli: "Kipman's invention is a vision for the future of computing with the potential to change how we connect and collaborate."
Munich, 24 April 2018 - In the near future, mobile computing may no longer take place on internet-connected smartphones, but on wearable data glasses that blend images of the real world with digital information. Brazilian software engineer and hardware inventor Alex Kipman has developed a computer-powered headset featuring holographic lenses for an immersive, "mixed-reality" experience. In contrast to virtual reality technology, which replaces the physical world with a wholly simulated environment, his mixed-reality headset combines views of the real world with layers of digital information, including holographic images perceived by the human eye as actual objects. By transforming how users can interact and work with digital content, and doing away with the need to stare at a screen, Kipman's "HoloLens" allows for applications such as computer-assisted surgeries, holographic teleconferencing and guided navigation.
For this achievement, Alex Kipman has been nominated as a finalist for the European Inventor Award 2018 in the category "Non-EPO countries". The winners of the European Patent Office's annual innovation prize will be announced at a ceremony in Paris, Saint-Germain-en-Laye, on 7 June 2018.
"Alex Kipman's invention is a vision for the future of computing with the potential to change how we connect and collaborate," said EPO President Benoît Battistelli announcing the European Inventor Award 2018 finalists. "His ideas are enhancing the real world with digital information that helps people achieve higher levels of efficiency, and creating new opportunities in fields such as engineering, communications, and healthcare."
Combining the real and digital worlds
The Brazilian inventor has a strong track record in innovation, having developed the "Kinect" motion controller for the Xbox video game system which set the Guinness Book world record for fastest-selling consumer electronics device when it came out in 2010. Kipman has built on this success with the HoloLens: The headset, marketed by Microsoft, gives users a full field of vision in which computer-generated objects - such as a ball or 3D-image of a person - blend seamlessly with reality. Compared to other mixed-reality (also known as "augmented reality") devices currently available, Kipman's invention differs in three key aspects: First, the HoloLens is self-contained with built-in processors, requiring no external connection to smartphones or PCs. Second, it replaces two-dimensional displays with 3D-holographic images for realistic "mixed-reality" experiences. And third, it eliminates physical controls such as keyboards, game pads, or touch screens. Instead, users can interact with digital objects in their field of vision via gestures and voice commands.
Alex Kipman has invented a device that makes holographic computing possible: The headset superimposes graphical elements over the natural backdrop, blending computer-generated objects with reality and enabling headset wearers to manipulate these objects as well. The headset lenses are transparent, and the holographs are viewed through a frame in the middle of the user's view. This preserves peripheral vision, which means people wearing the headset can continue to interact freely with the environment around them. "The magic of HoloLens is that it can pin holograms right into your world with more precision and reality than ever before," says the inventor.
Thanks to built-in machine learning and AI-algorithms, the headset is contextually aware of its surroundings and able to interpret user commands such as "move that chair". It observes its surroundings via an inertial measurement unit (a device that measures linear and angular motion), a depth-sensing camera, a 2.4-megapixel video camera, four microphones and an ambient light sensor - so it "knows" which chair the wearer is currently seeing. It collects several terabytes of data every second, piped in via sensors and then computed.
The HoloLens owes its innovative edge to the waveguide lenses, powered by a built-in holographic processing unit, which "trick" the human brain into interpreting light signals as solid, 3D holograms. This hologram is computed on the processor and then displayed via the device's "light engines" (projectors), featuring two tiny liquid crystal on silicon (LQoD) displays. "It's an experience where content is no longer tied to a screen but exists in both time and space laid over the real world," says Kipman.
Holograms and real-world applications: more than entertainment
The HoloLens incorporates several patent-protected Microsoft inventions, including Kipman's motion sensors initially brought to market for the Xbox Kinect video game controller. Speaking about the role of IP and patent protection, Kipman says: "Developing such a product [as the HoloLens] you can't do in the garage. Because it requires a level of investment and patience, and long term basic research."
The software for the HoloLens is developed in the Windows 10 operating system in the Windows Mixed Reality software environment. Current applications of the device include telecommunications (e.g. for enhanced video calls), a 3D virtual tourism application, and a suite of mixed-reality games, as well as industrial uses for metering and quality control, and interactive digital human anatomy. A host of promising new applications are being developed. In 2017, a Spanish team of surgeons used the mixed-reality headset successfully while operating on a patient with a malignant muscular tumour. During the procedure, the surgeons were able to view live diagnostic imaging from MRI and radiography data feeds, without taking their eyes off the patient.
Once ready for mass adoption, the HoloLens could lead a fundamental shift, as mixed-reality devices replace today's smartphones as the main communications devices. According to experts at market researcher Gartner, 30% of web browsing sessions could be done without a screen by 2020. At that point, 50% of all search queries could originate, not from keyboards or touchscreens, but voice commands as computing moves into an era beyond screens.
Creating a booming world market
Analysts predict massive growth in the mixed-reality industry. The world market for mixed- and virtual-reality devices was worth some EUR 3 billion in 2016, but driven by next-generation mobile mixed-reality hardware such as the HoloLens, it is projected to reach EUR 92 billion by 2021 (mixed reality dominating with EUR 71 billion compared to EUR 21 billion for virtual reality), according to consultancy Digi-Capital. Major drivers for the rise of mixed-reality technology include, for example, consumer demand for personal advice during shopping trips, with more than 100 million people expected to be conducting their shopping with the assistance of the technology by 2020.
Microsoft has sold some 50 000 HoloLens units to date to software developers and commercial clients, and its strategic development partnerships with companies such as Lowe's, Case Western Reserve University, Boeing, Saab, and Volvo aim at "seeding" the market for mixed-reality products. Once the company has built an attractive library of programmes and apps, and as mixed-reality technology trickles down from the premium to the consumer segment, these titles could join its own 3D-teleconferencing app, in which participants see realistic holographic likenesses of the people with whom they are talking. Another promising application of HoloLens technology can help bring down language barriers by providing live translation of written text and spoken words as holographic subtitles.
A passion for computer software
Born in Curitiba, Brazil, Alex Kipman fell in love with computer software programming at a young age, having started playing Atari video games at the age of 6. "Software is the only art form on earth not bound by the laws of physics," Kipman says. "In software, you are only bound by the limits of your imagination."
The inventor started his professional career at Microsoft in Redmond right after graduating from the Rochester Institute of Technology in 2001. In a career spanning more than 15 years as a developer for the company, he has been named as the primary inventor on more than 150 US and Asian patents, as well as two granted European patents and more than a dozen European patent applications. His current position as Technical Fellow of the Windows and Devices Group enables him to maintain a hands-on approach to building the HoloLens and related Windows Mixed Reality software platform.
Kipman has been awarded the Popular
Mechanics Breakthrough Award (2009), Microsoft Outstanding Technical
Achievement Award (2012), and IPO Education Foundation's Inventor of the Year
Award (2012). He also received the honour of being named one of TIME magazine's
Top 25 "Nerds of the Year" (2010), an epithet he wears proudly.
In his private time, the passionate futurist likes to retreat to the simplicity of his family's off-the-grid farm in the Brazilian rainforest. Unplugged from digital devices, he seeks out real-life outdoor experiences with his wife and daughter, which is also where the concept for his smart glasses originated: "My ideas about computing in the real world come from the fact that I don't live in a lab, but rather like to spend my time in nature."
Additional resources
View the patent:
EP3028121
Changing the face of media:
Alex Kipman predicts a world in which smartphones are replaced by mixed-reality devices such as the HoloLens. This industry-changing vision puts the inventor in company with other past European Inventor Award winners and finalists whose innovations fundamentally altered the ways we interact with sound, video and text. Joseph M. Jacobson and Barrett Comiskey helped develop the electronic ink displays that ushered in the e-reader revolution. The paper transistors from Elvira Fortunato and Rodrigo Martins have shown that animated billboards and papers might be right around the corner. CD, DVD and Blu-ray coding from Kornelis A. Schouhamer Immink moved video and music from analogue into the digital age, and Karlheinz Brandenburg's MP3 codex enabled music to be stored on our smartphones and handheld players. It's hard to say where our changing relationship with media will take us, but mixed-reality will likely be part of the journey.
Contacts at the European Patent Office in Munich, Germany:
Jana Mittermaier
Director External Communication
Rainer Osterwalder
Press Spokesperson
Tel. +49 (0)89
2399 1820
Mobile: +49
(0)163 8399527
press@epo.org