Semi-Dense Depth Interpolation using Deep Convolutional Neural Networks

P. 1407–1415.

Makarov I., Vladimir Aliev, Olga Gerasimova

With advances of recent technologies, augmented reality systems and autonomous vehicles gained a lot of interest from academics and industry. Both these areas rely on scene geometry understanding, which usually requires depth map estimation. However, in case of systems with limited computational resources, such as smartphones or autonomous robots, high resolution dense depth map estimation may be challenging.
In this paper, we study the problem of semi-dense depth map interpolation along with low resolution depth map upsampling. We present an end-to-end learnable residual convolutional neural network architecture that achieves fast interpolation of semi-dense depth maps with different sparse depth distributions: uniform, sparse grid and along intensity image gradient. We also propose a loss function combining classical mean squared error with perceptual loss widely used in intensity image super-resolution and style transfer tasks. We show that with some modifications, this architecture can be used for depth map super-resolution. Finally, we evaluate our results on both synthetic and real data, and consider applications for autonomous vehicles and creating AR/MR video games.

Keywords: computer vision, Augmented Reality, convolutional neural networks, Autonomous vehicles, Depth map, Mixed Reality

In book

Proceedings of the 25th ACM international conference on Multimedia (ACM MM'17), Mountain View, CA USA, 23-27 October 2017.

NY: Association for Computing Machinery (ACM), 2017.