Video Frame Interpolation via Adaptive Separable Convolution
Simon Niklaus, Long Mai, and Feng Liu
IEEE International Conference on Computer Vision

Standard video frame interpolation methods first estimate optical flow between input frames and then synthesize an intermediate frame guided by motion. Recent approaches merge these two steps into a single convolution process by convolving input frames with spatially adaptive kernels that account for motion and re-sampling simultaneously. These methods require large kernels to handle large motion, which limits the number of pixels whose kernels can be estimated at once due to the large memory demand.

To address this problem, this paper formulates frame interpolation as local separable convolution over input frames using pairs of 1D kernels. Compared to regular 2D kernels, the 1D kernels require significantly fewer parameters to be estimated. Our method develops a deep fully convolutional neural network that takes two input frames and estimates pairs of 1D kernels for all pixels simultaneously. Since our method is able to estimate kernels and synthesizes the whole video frame at once, it allows for the incorporation of perceptual loss to train the neural network to produce visually pleasing frames.

This deep neural network is trained end-to-end using widely available video data without any human annotation. Both qualitative and quantitative experiments show that our method provides a practical solution to high-quality video frame interpolation.

No paper is perfect and it is important to be upfront about issues once they become apparent. As such, I like to take the opportunity to mention some for this paper below.

• We were not aware of the Filter Flow paper from Seitz and Baker at the time, but it would have been important to discuss it in the related work section.
• We pad the input images when inserting them into the network to make sure that the ouput size is not affected by the adaptive convolutions. However, it is perfectly fine to only pad the input images when applying the adaptive convolutions which is more efficient and leads to slightly better results.

Please do not hesitate to reach out to me with questions and suggestions.