6 Key Points from Mark Zuckerberg’s Interview on AR/VR

Published March 15, 2021

Facebook’s CEO Mark Zuckerberg was recently interviewed by The Information to discuss the company’s work in augmented and virtual reality. Facebook acquired Oculus seven years ago and has since become the leader in VR. According to internal organizational data reviewed by The Information, nearly 10,000 people, or nearly one-fifth of Facebook’s total global workforce, are working in the group developing AR and VR devices.

During a 46 min conversation, Zuckerberg elaborated on his vision and the challenges ahead.

It’s not just about Gaming.

Mark Zuckerberg revealed that he wanted to work on AR/VR even before he started Facebook. He said his motivation to develop the technology is to enable people to “teleport around”. For example, sitting on someone’s else couch and have a conversation with the avatar. He believes it can have a very profound impact on social interaction and work, potentially creating more opportunities outside cities.

AR and VR should be lumped together.

He said that AR and VR are two sides of the same coin. They will be based on the same ecosystem. VR is easier to implement first and can build on the gaming technology stack. AR is still a few years off but will benefit from the progress in VR. The ideal device is something that can do both, for instance a pair of see-through AR glasses that can shade to turn into a VR headset.

VR requires huge improvements in display technologies.

Zuckerberg’s long-term vision for VR (10-year timeframe) is a virtual environment that is as vivid and with the same resolution as the real world. He said that even High Dynamic Range (HDR) displays cannot currently replicate the full range of nits of what the eyes can see. In terms of resolution, he thinks the high level of details will require displays that can provide foveated rendering. The optics also represent a challenge. Currently the lenses have only one focal length. For a more realistic experience, they should transition to varifocal.

Sensors and audio are also critical.

Zuckerberg said that spatial audio is a key technology. Even with basic avatars, the sensor of space provided by audio makes a big difference on the realistic feel. For more advanced avatars with facial expressions, he said that face tracking and eye tracking will be essential.

Zuckerberg also commented on the neural interface developed by Ctrl-Labs, a startup acquired by Facebook in 2019. He said that Facebook had already started working on neural interface even before the acquisition. The idea is to have something like a strap that can read signals sent by the brain’s motor neurons. Neuroplasticity of the brain means we will be able to learn to control inputs differently, for instance acting as if we had a third hand. He does not believe that reading brain waves will work. Elon Musk’s implanted chips can be good for prosthetics but will not be widely accepted for AR/VR.

Thermal management is one of the biggest challenges.

Facebook is not only developing the native OS but also custom silicon chips for VR. Zuckerberg said that, unlike phones that have a large surface area, AR/VR devices are much more difficult to cool. It is not possible to dissipate heat on the user’s face so it is still a challenge to power a real-time scene without overheating. A custom build silicon chip can reduce overheating but also enable additional functionalities. For example, driving an AR display will require rendering different objects in different planes. Retrofitting existing technologies from the phone industry does not work, so low-level work is needed.

Different business model than Apple

Zuckerberg said that he wants to price the products as low as possible. One possible way to do this is to use the app store and ecosystem to generate revenues. Unlike other companies, he does not want AR/VR to cost a premium. Although we will still have phones in five years, Zuckerberg believes that AR/VR will become the primary platform for most people over time. He said that he thinks about the platform the same way that Google thought about Android.

Image Description
Written by

Dr. Guillaume Chansin