FitMe: Deep Photorealistic 3D Morphable Model Avatars

Published in the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023) (CVPR Paper Link)

See the Project Page for more details and HQ videos.

Alexandros Lattas, Stylianos Moschoglou, Stylianos Ploumpis,
Baris Gecer, Jiankang Deng, Stefanos Zafeiriou
Imperial College London

Overview

In this paper, we introduce FitMe, a facial reflectance model and a differentiable rendering optimization pipeline, that can be used to acquire high-fidelity renderable human avatars from single or multiple images.

The model consists of a multi-modal style-based generator, that captures facial appearance in terms of diffuse and specular reflectance, and a PCA-based shape model. We employ a fast differentiable rendering process that can be used in an optimization pipeline, while also achieving photorealistic facial shading. Our optimization process accurately captures both the facial reflectance and shape in high-detail, by exploiting the expressivity of the style-based latent representation and of our shape model.

FitMe achieves state-of-the-art reflectance acquisition and identity preservation on single "in-the-wild" facial images, while it produces impressive scan-like results, when given multiple unconstrained facial images pertaining to the same identity. In contrast with recent implicit avatar reconstructions, FitMe requires only one minute and produces relightable mesh and texture-based avatars, that can be used by end-user applications.

Method

In two steps, we optimize the latent vector of the generator, the shape and rendering parameters, by combining 3DMM fitting, GAN inversion and GAN tuning. The optimization is guided by accurate diffuse and specular differentiable rendering.

Results

FitMe can be optimized for arbitrary facial images, from a single internet sourced picture, to multiple selfie photographs.

Even with three unconstrained quick selfies of a subject, in an arbitrary environment and illumination, FitMe produces high-fidelity relfectance and shape that can be photorealistically relighted.

selfie1.mp4

CVPR Video on Youtube

Code

This model is not currently publicly available, but we have released the rendering code used in AvatarMe++ and FitMe as a Pytorch3D extension Pytorch3D-Me. It includes the Blinn-Phong shader, subsurface scattering, and occlusion approximations.

Reference

@InProceedings{Lattas_2023_CVPR,
      author    = {Lattas, Alexandros and Moschoglou, Stylianos and Ploumpis, Stylianos and Gecer, Baris and Deng, Jiankang and Zafeiriou, Stefanos},
      title     = {FitMe: Deep Photorealistic 3D Morphable Model Avatars},
      booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
      month     = {June},
      year      = {2023},
      pages     = {8629-8640}
  }