Reconstructing a video from just a few pixels with machine learning

What is it about?

The “fovea” is the most sensitive region in the human eye. It is located at the center of a thin layer called the retina. Our vision is the most detailed within this layer. The sharpness of the vision decreases as we move away from it. Understanding these inner workings of the eye can help advance virtual reality. To this end, a new study tried to assess these. The authors developed a model called DeepFovea. They trained it to automatically reconstruct foveated videos. The idea was to take an input video and keep only one tenth of its original pixels. Most of these pixels are focused on the most important area of the images. In contrast, the “peripheral zone” retains only a few random pixels. DeepFovea can reconstruct a video to match the original based on this limited information. This preserves details where they matter the most—in the fovea.

Featured Image

Photo by v2osk on Unsplash

Photo by v2osk on Unsplash

Why is it important?

Virtual reality headsets are becoming increasingly popular. Users expect high resolutions out of these devices. But even modern computers struggle to render frames for virtual reality games. DeepFovea can let the graphics card produce only a tiny fraction of pixels each frame. The model can quickly reconstruct the rest of the video without worsening quality. Besides virtual reality, DeepFovea can be used to reduce the size of videos. This makes them easier to both store in memory and stream. The study also shows that videos reconstructed using DeepFovea mostly match videos compressed using standard methods. KEY TAKEAWAY: DeepFovea can initiate future research into video compression and reconstruction using machine learning.