MPIB: An MPI-Based Bokeh Rendering Framework
for Realistic Partial Occlusion Effects



1Huazhong University of Science and Technology
2Adobe Research

ECCV 2022
Teaser Image



GitHub
Paper
Supplementary
Material
Poster

Video

Abstract

Partial occlusion effects are a phenomenon that blurry objects near a camera are semi-transparent, resulting in partial appearance of occluded background. However, it is challenging for existing bokeh rendering methods to simulate realistic partial occlusion effects due to the missing information of the occluded area in an all-in-focus image. Inspired by the learnable 3D scene representation, Multiplane Image (MPI), we attempt to address the partial occlusion by introducing a novel MPI-based high-resolution bokeh rendering framework, termed MPIB. To this end, we first present an analysis on how to apply the MPI representation to bokeh rendering. Based on this analysis, we propose an MPI representation module combined with a background inpainting module to implement high-resolution scene representation. This representation can then be reused to render various bokeh effects according to the controlling parameters. To train and test our model, we also design a ray-tracing-based bokeh generator for data generation. Extensive experiments on synthesized and real-world images validate the effectiveness and flexibility of this framework.

Method

  • Framework
  • Bokeh Generator
  • MPIB takes an all-in-focus image and a potentially imperfect disparity map as input to obtain a 3D scene representation of multiple RGBA planes. This representation is generated by an MPI representation module and a background inpainting module. Then, the scene representation can be reused to produce multiple bokeh images according to different controlling parameters.


    Framework
  • The ray-tracing-based bokeh generator is capable of creating a realistic bokeh image from a background RGB image and some foreground RGBA images with customized disparities and different controlling parameters. We design this bokeh generator for producing training and test data.


    Error Analysis

Visualization

  • Comparison
  • Animation
  • Comparison with SOTA on real-world images. Disparity inputs are predicted by DPT.

    comparison image 1 comparison image 2 comparison image 3 comparison image 4 comparison image 5 comparison image 6
  • Animations on real-world images with different controlling parameters.

    Blur Amount
    Image K_1 Image K_2 Image K_3
    Animation K_1 Animation K_2 Animation K_3

    Refocused Disparity
    Image df_1 Image df_2 Image df_3
    Animation df_1 Animation df_2 Animation df_3

    Gamma Value
    Image gamma_1 Image gamma_2 Image gamma_3
    Animation gamma_1 Animation gamma_2 Animation gamma_3

Citation