Simon Niklaus - Benchmarking Video Frame Interpolation

Benchmarking Video Frame Interpolation

Benchmarking Video Frame Interpolation
Simon Kiefhaber, Simon Niklaus, Feng Liu, and Simone Schaub-Meyer

Video frame interpolation, the task of synthesizing new frames in between two or more given ones, is becoming an increasingly popular research target. However, the current evaluation of frame interpolation techniques is not ideal. Due to the plethora of test datasets available and inconsistent computation of error metrics, a coherent and fair comparison across papers is very challenging. Furthermore, new test sets have been proposed as part of method papers so they are unable to provide the in-depth evaluation of a dedicated benchmarking paper. Another severe downside is that these test sets violate the assumption of linearity when given two input frames, making it impossible to solve without an oracle.

We hence strongly believe that the community would greatly benefit from a benchmarking paper, which is what we propose. Specifically, we present a benchmark which establishes consistent error metrics by utilizing a submission website that computes them, provides insights by analyzing the interpolation quality with respect to various per-pixel attributes such as the motion magnitude, contains a carefully designed test set adhering to the assumption of linearity by utilizing synthetic data, and evaluates the computational efficiency in a coherent manner.

Hover over the table below to see more details here.

	1K					2K					4K
	rank	all	0-occ.	1-occ.	2-occ.	rank	all	0-occ.	1-occ.	2-occ.	rank	all	0-occ.	1-occ.	2-occ.
SplatSyn	05 / 21	28.45	30.89	21.71	17.20	01 / 21	28.40	30.91	21.43	16.71	01 / 21	27.40	29.71	20.48	15.97
BiFormer	-	-	-	-	-	-	-	-	-	-	02 / 21	26.78	29.55	19.35	15.55
UPR-Net-L	06 / 21	28.31	30.28	22.14	17.46	02 / 21	27.09	28.76	21.20	16.47	03 / 21	25.69	27.05	20.28	14.65
DAIN	08 / 21	27.83	31.47	19.97	16.94	03 / 21	26.81	29.77	19.32	16.40	-	-	-	-	-
M2M	03 / 21	28.61	31.73	21.21	17.16	04 / 21	25.81	28.12	18.95	15.99	06 / 21	20.61	21.44	16.29	14.49
CtxSyn	01 / 21	29.36	32.72	21.81	17.34	07 / 21	23.33	24.65	17.91	15.81	09 / 21	18.80	19.47	14.98	13.75
ABME	02 / 21	28.63	31.62	21.36	16.95	05 / 21	23.95	25.35	18.41	15.54	10 / 21	18.35	18.80	15.41	13.97
SoftSplat	04 / 21	28.55	31.28	21.46	17.65	09 / 21	22.59	23.58	17.91	16.05	08 / 21	18.85	19.45	15.28	14.04
AMT-G	07 / 21	28.18	30.62	21.35	17.80	11 / 21	20.97	21.71	17.00	15.72	-	-	-	-	-
EMA	09 / 21	25.95	27.36	20.53	17.68	10 / 21	21.29	21.94	17.61	16.17	-	-	-	-	-
FLDR	10 / 21	24.09	25.80	18.14	15.56	06 / 21	23.80	25.38	17.96	15.12	04 / 21	23.53	25.05	17.68	14.47
EDSC	13 / 21	22.90	23.91	18.29	16.35	11 / 21	20.97	21.60	17.38	15.91	07 / 21	19.14	19.57	16.32	15.02
XVFI	12 / 21	23.73	25.36	17.90	15.33	08 / 21	23.25	24.74	17.54	14.95	05 / 21	22.48	23.71	17.15	14.33
VFIformer	11 / 21	23.83	24.79	19.35	17.26	13 / 21	19.91	20.48	16.56	15.68	-	-	-	-	-
SepConv++	17 / 21	21.85	22.64	17.84	16.28	16 / 21	19.23	19.74	16.08	15.25	11 / 21	17.84	18.24	15.11	14.07
SepConv	16 / 21	22.00	23.06	17.25	15.86	15 / 21	19.35	19.90	16.05	15.17	12 / 21	17.79	18.16	15.24	14.28
RRIN	15 / 21	22.17	23.08	17.81	16.09	14 / 21	19.39	19.94	16.10	15.17	14 / 21	17.61	18.04	14.73	13.75
CAIN	18 / 21	21.83	22.69	17.64	15.92	17 / 21	18.95	19.47	15.80	14.79	13 / 21	17.67	18.04	15.09	14.03
VOS	19 / 21	20.95	21.70	17.06	15.74	18 / 21	18.74	19.29	15.44	14.73	15 / 21	17.38	17.90	14.11	13.38
AdaCoF	20 / 21	20.44	21.10	16.83	15.40	20 / 21	18.27	18.67	15.62	14.42	16 / 21	17.09	17.35	15.13	13.87
BMBC	14 / 21	22.18	23.14	17.74	15.16	19 / 21	18.53	19.04	15.43	14.26	-	-	-	-	-

Multiframe

	PSNR^*								PSNR^*_σ
	rank	t: 0.125	t: 0.25	t: 0.375	t: 0.5	t: 0.625	t: 0.75	t: 0.875	rank	t: 0.125	t: 0.25	t: 0.375	t: 0.5	t: 0.625	t: 0.75	t: 0.875
M2M	01 / 20	31.83	29.86	28.93	28.61	28.57	29.19	30.89	04 / 20	21.94	20.53	19.87	19.52	19.48	19.81	20.99
SplatSyn	02 / 20	31.78	29.72	28.70	28.45	28.53	29.30	31.02	02 / 20	22.32	20.87	20.23	19.97	19.96	20.29	21.50
CtxSyn	05 / 20	28.72	29.07	29.30	29.36	29.26	29.14	29.11	01 / 20	21.87	21.19	20.77	20.55	20.56	20.85	21.61
ABME	03 / 20	31.00	29.43	28.99	28.63	28.67	28.91	30.57	03 / 20	21.38	20.26	20.05	19.72	19.86	19.95	21.02
SoftSplat	04 / 20	30.78	29.30	28.79	28.55	28.52	28.78	30.04	05 / 20	21.19	20.19	19.82	19.62	19.59	19.73	20.53
UPR-Net-L	06 / 20	27.90	27.85	28.34	28.31	28.03	27.69	27.90	06 / 20	19.06	19.38	19.91	19.77	19.69	19.46	19.70
DAIN	07 / 20	27.14	26.81	27.15	27.83	27.05	26.88	27.55	08 / 20	19.51	18.68	18.59	18.71	18.55	18.73	19.80
EMA	08 / 20	29.17	27.26	26.45	25.95	26.29	26.99	28.75	07 / 20	20.32	19.11	18.66	18.38	18.51	18.86	20.00
FLDR	09 / 20	27.30	25.24	24.38	24.09	24.26	24.97	26.82	09 / 20	19.17	17.84	17.37	17.20	17.27	17.68	18.92
VFIformer	10 / 20	26.88	25.07	24.31	23.83	24.10	24.75	26.60	11 / 20	18.45	17.33	16.92	16.61	16.73	17.08	18.28
XVFI	11 / 20	26.43	24.71	23.99	23.73	23.83	24.39	25.85	10 / 20	18.54	17.39	16.95	16.79	16.85	17.20	18.16
EDSC	12 / 20	25.71	23.52	23.09	22.90	23.05	23.60	25.33	12 / 20	18.19	16.85	16.70	16.65	16.68	16.87	18.00
SepConv++	14 / 20	25.08	23.43	22.54	21.85	22.47	23.27	24.74	13 / 20	18.07	16.94	16.36	15.85	16.30	16.82	17.84
SepConv	13 / 20	25.67	23.62	22.68	22.00	22.54	23.34	25.12	15 / 20	17.84	16.66	16.14	15.72	16.01	16.41	17.38
CAIN	15 / 20	25.27	23.30	22.44	21.83	22.37	23.19	25.04	14 / 20	17.88	16.66	16.17	15.84	16.10	16.55	17.68
BMBC	16 / 20	24.31	23.13	22.64	22.18	22.43	22.89	24.09	17 / 20	17.19	16.13	15.69	15.37	15.52	15.90	16.98
VOS	17 / 20	24.38	22.58	21.68	20.95	21.60	22.47	24.36	16 / 20	17.38	16.19	15.63	15.13	15.57	16.10	17.34
AdaCoF	18 / 20	23.65	21.83	21.10	20.44	20.98	21.58	23.11	18 / 20	17.00	15.79	15.33	14.83	15.23	15.59	16.58
RRIN	19 / 20	21.18	21.45	21.92	22.17	21.92	21.49	21.15	19 / 20	15.03	15.25	15.56	15.65	15.51	15.32	15.11
AMT-G	20 / 20	16.69	17.68	19.44	28.18	19.47	17.37	16.29	20 / 20	12.14	12.74	13.86	19.28	13.88	12.50	11.86

Runtime

	1 frame						7 frames
	rank	1K	rank	2K	rank	4K	rank	1K	rank	2K	rank	4K
SplatSyn	07 / 21	0.03	01 / 21	0.04	01 / 21	0.09	01 / 21	0.04	01 / 21	0.06	01 / 21	0.20
M2M	01 / 21	0.02	02 / 21	0.05	02 / 21	0.17	06 / 21	0.17	02 / 21	0.34	02 / 21	1.18
CAIN	01 / 21	0.02	03 / 21	0.06	03 / 21	0.21	04 / 21	0.14	03 / 21	0.39	03 / 21	1.48
VOS	01 / 21	0.02	03 / 21	0.06	04 / 21	0.23	02 / 21	0.11	04 / 21	0.42	04 / 21	1.61
AdaCoF	01 / 21	0.02	03 / 21	0.06	04 / 21	0.23	03 / 21	0.12	04 / 21	0.42	05 / 21	1.62
EDSC	01 / 21	0.02	06 / 21	0.09	06 / 21	0.33	06 / 21	0.17	06 / 21	0.61	06 / 21	2.35
SepConv++	01 / 21	0.02	11 / 21	0.15	09 / 21	0.58	05 / 21	0.16	12 / 21	1.03	11 / 21	4.03
SepConv	08 / 21	0.04	08 / 21	0.13	07 / 21	0.46	10 / 21	0.28	10 / 21	0.96	07 / 21	3.17
RRIN	08 / 21	0.04	08 / 21	0.13	08 / 21	0.57	09 / 21	0.25	09 / 21	0.91	09 / 21	3.99
CtxSyn	12 / 21	0.06	13 / 21	0.19	13 / 21	0.76	08 / 21	0.23	07 / 21	0.86	08 / 21	3.49
XVFI	10 / 21	0.05	07 / 21	0.12	11 / 21	0.61	12 / 21	0.34	08 / 21	0.90	14 / 21	4.37
FLDR	10 / 21	0.05	10 / 21	0.14	10 / 21	0.59	12 / 21	0.34	11 / 21	0.97	10 / 21	4.00
SoftSplat	12 / 21	0.06	14 / 21	0.21	14 / 21	0.81	11 / 21	0.30	13 / 21	1.08	12 / 21	4.23
UPR-Net-L	12 / 21	0.06	12 / 21	0.16	11 / 21	0.61	14 / 21	0.41	14 / 21	1.17	13 / 21	4.30
BiFormer	-	-	-	-	15 / 21	1.92	-	-	-	-	15 / 21	13.48
EMA	15 / 21	0.07	15 / 21	0.28	-	-	16 / 21	0.52	15 / 21	1.98	-	-
AMT-G	15 / 21	0.07	16 / 21	0.31	-	-	15 / 21	0.48	16 / 21	2.20	-	-
DAIN	17 / 21	0.21	17 / 21	0.81	-	-	17 / 21	0.68	17 / 21	2.87	-	-
ABME	18 / 21	0.29	18 / 21	1.08	16 / 21	4.34	18 / 21	2.00	18 / 21	7.88	16 / 21	30.51
VFIformer	19 / 21	1.26	19 / 21	4.85	-	-	19 / 21	8.88	19 / 21	33.77	-	-
BMBC	20 / 21	1.97	20 / 21	7.66	-	-	20 / 21	13.83	20 / 21	54.24	-	-