The creation of realistic content and materials plays a central role in both the videogame and movie industries. Even though in the last few years hardware components and software greatly improved the design of realistic assets, this aspect is still a bottleneck in terms of control capabilities and ease of use.
Designing a realistic material means identifying a set of physical parameters that define the behaviour of light at each point location, which is described by functions called Bidirectional Scattering Distribution Function (BSDFs).
Many methodologies are used in material design, such as 3D material scanning or material synthesis with Generative Adversarial Networks (GANs). However, each technique did not allow artists to edit these materials in an easy way. Materials could also be generated using procedural programs. These are programs that map a data-flow graph of consecutive operations controlled by a set of procedural parameters. This means that physical parameters are not directly manipulated by artists, but are generally computed by a program given some high-level parameters. So, designing a specific and realistic material is challenging even for experienced artists.
We aim to build an Inverse Procedural Appearance Design framework to facilitate the usage of procedural programs. Given an image that represents the desired appearance, we aim to identify the procedural parameters that minimize the distance between the output of the procedural program and the image itself.
Once a first estimation of the procedural parameters is computed, artists can modify them to obtain slightly different results that better match the desired appearance, thus reducing the time consumed and allowing an easier interaction and control.
This project aims to build a framework for Inverse Procedural Appearance Design that exploits both classical optimization techniques and Neural Networks.
Previously described works performed well on regular textures but reached quite poor results on more complex appearances. This project aims to generalize over all BRDFs parameters, including normal maps, displacement maps and 3D volumetric textures. On the contrary to 2D textures, volumetric ones are characterized by a higher dimensionality in the physical parameters, increasing the complexity of the problem
We also aim to include the design of lighting into the same generalized framework.
Oppositely to all the systems that are based on image synthesis models, we aim to solve the problem in the context of images generated with surface and volumetric rendering equations. To do so, we aim to exploit the idea of differentiable rendering [7], changing the modules that compose a procedural program in favour of differentiable functions. We also aim to mix this neural strategy, which can provide the first attempt of parameters estimation, with a precise optimization using strategies like Markov Chain Monte Carlo (MCMC).
The second aspect this work plans to address is the issue of user control. Previous attempts to solve the same problem provided good-looking results, but they did not allow artists to manipulate the results in an easy way if the desired outcome was not obtained. In practice, it ended in manipulating the entire image in a very time-consuming step. We aim to generate a framework that allows users to edit the obtained result using a sketch-based input, providing an ad-hoc optimization as already explored in [8] and [9].
[7] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. ACM Transactions on Graphics, 38(6), 2019.
[8] Xiaobo An and Fabio Pellacini. Appprop: Allpairs appearance-space edit propagation. ACM Transactions on Graphics, 2008.
[9] Fabio Pellacini. Envylight: An interface for editing natural illumination. ACM Transactions on Graphics, 2010.