Appearance manipulation of a single photograph (top images) when using off-the-shelf software like Photoshop directly (left arrow) and when using the same in combination with our new layering (right arrow). For the car example, the image was decomposed into layers (albedo, irradiance, specular, and ambient occlusion), which were then manipulated individually: specular highlights were strengthened and blurred; irradiance and ambient occlusion were darkened and have added contrast; the albedo color was changed. While the image generated without our decomposition took much more effort (selections, adjustments with curves, and feathered image areas), the result is still inferior. For the statue example, a different decomposition splitting the original image into light directions was used. The light coming from the left was changed to become more blue, while light coming from the right was changed to become more red. A similar effect is hard to achieve in Photoshop even after one order of magnitude more effort.

Abstract

Photographers routinely compose multiple manipulated photos of the same scene into a single image, producing a fidelity difficult to achieve using any individual photo. Alternately, 3D artists set up rendering systems to produce layered images to isolate individual aspects of the light transport, which are composed into the final result in post-production. Regrettably, these approaches either take considerable time and effort to capture, or remain limited to synthetic scenes. In this paper, we suggest a method to decompose a single image into multiple layers that approximates effects such as shadow, diffuse illumination, albedo, and specular shading. To this end, we extend the idea of intrinsic images along two axes: first, by complementing shading and reflectance with specularity and occlusion, and second, by introducing directional dependence. We do so by training a convolutional neural network (CNN) with synthetic data. Such decompositions can then be manipulated in any off-the-shelf image manipulation software and composited back. We demonstrate the effectiveness of our decomposition on synthetic (i. e., rendered) and real data (i. e., photographs), and use them for photo manipulations, which are otherwise impossible to perform based on single images. We provide comparisons with state-of-the-art methods and also evaluate the quality of our decompositions via a user study measuring the effectiveness of the resultant photo retouching setup.