BokehMe: When Neural Rendering Meets
Classical Rendering



1Huazhong University of Science and Technology
2Adobe Research

CVPR 2022 (Oral Presentation)
Teaser Image



GitHub
Paper
Supplementary
Material
Dataset

Video

Abstract

We propose BokehMe, a hybrid bokeh rendering framework that marries a neural renderer with a classical physically motivated renderer. Given a single image and a potentially imperfect disparity map, BokehMe generates high-resolution photo-realistic bokeh effects with adjustable blur size, focal plane, and aperture shape. To this end, we analyze the errors from the classical scattering-based method and derive a formulation to calculate an error map. Based on this formulation, we implement the classical renderer by a scattering-based method and propose a two-stage neural renderer to fix the erroneous areas from the classical renderer. The neural renderer employs a dynamic multi-scale scheme to efficiently handle arbitrary blur sizes, and it is trained to handle imperfect disparity input. Experiments show that our method compares favorably against previous methods on both synthetic image data and real image data with predicted disparity. A user study is further conducted to validate the advantage of our method.

Method

  • Framework
  • Classical Renderer
  • Neural Renderer
  • Taking an all-in-focus image, a potential imperfect dispairty map, and some controlling parameters as input, BokehMe can produce artifact-free and highly controllable bokeh effects by combining a classical renderer and a neural renderer.


    Framework
  • The classical renderer is implemented by a scattering-based method. To determine the areas rendered incorrectly by this method, we conduct an analysis on the error between scattering-based rendering and real rendering on a virtual lens system and derive an error map which identifies these areas.


    Error Analysis
  • The neural renderer aims to handle the areas rendered incorrectly by the classical renderer and overcome the limitations of the blur range. To this end, we first use ARNet to adaptively resize the input and produce a low-resolution error map and a low-resolution bokeh image. Then, the bokeh image is iteratively upsampled by IUNet without losing quality, while the error map is bilinearly upsampled, which enables a smooth fusion of the rendered results from the two renderers.


    Neural Renderer

Visualization

  • BLB
  • EBB400
  • IPB
  • Animation
  • Results on the BLB dataset which is rendered by Blender 2.93.

    BLB image 1 BLB image 2 BLB image 3 BLB image 4 BLB image 5
        
        
  • Results on the EBB400 dataset where the images come from EBB! and the disparity inputs are predicted by MiDaS.

    EBB image 1 EBB image 2 EBB image 3 EBB image 4 EBB image 5 image 6
  • Results on the IPB dataset which is captured by iPhone 12 Portrait mode.

    IPB image 1
    IPB image 2
    IPB image 3
    IPB image 4
    IPB image 5
    IPB image 6
    IPB image 7
    IPB image 8
  • Animations on real-world images with different controlling parameters.

    Blur Amount
    Image K_1 Image K_2 Image K_3
    Animation K_1 Animation K_2 Animation K_3

    Refocused Disparity
    Image df_1 Image df_2 Image df_3
    Animation df_1 Animation df_2 Animation df_3

    Gamma Value
    Image gamma_1 Image gamma_2 Image gamma_3
    Animation gamma_1 Animation gamma_2 Animation gamma_3

    Aperture Shape
    Image aperture_1 Image aperture_2 Image aperture_3
    Animation aperture_1 Animation aperture_2 Animation aperture_3

Dataset

We release the BLB dataset rendered by Blender 2.93. The dataset contains 10 scenes, each consisting of an all-in-focus image, a disparity map, a stack of bokeh images with 5 blur amounts and 10 refocused disparities, and a parameter file. We additionally provide 15 corrupted disparity maps (through gaussian blur, dilation, erosion) for each scene. All 3D Blender models come from https://cloud.blender.org/p/gallery. Our BLB dataset can be downloaded from Google Drive or Baidu Netdisk.


BLB image 1 BLB image 2 BLB image 3 BLB image 4 BLB image 5
BLB image 6 BLB image 7 BLB image 8 BLB image 9 BLB image 10

Citation