We introduce MyStyle, a personalized deep generative prior trained with a few shots of an individual, which allows to reconstruct, enhance and edit images that are faithful to their key facial characteristics. Given a small reference set of portrait images of a person (~100), we tune the weights of a pretrained StyleGAN face generator to form a local, low-dimensional, personalized manifold in the latent space. We show that this manifold constitutes a personalized region that spans latent codes associated with diverse portrait images of the individual. Moreover, we demonstrate that we obtain a personalized generative prior, and propose a unified approach to apply it to various ill-posed image enhancement problems, such as inpainting and super-resolution, as well as semantic editing. Using the personalized generative prior we obtain outputs that exhibit high-fidelity to the input images and are also faithful to the key facial characteristics of the individual in the reference set. We demonstrate our method on numerous widely recognizable individuals (in fair-use images) for whom we have the prior knowledge for a qualitative evaluation of the expected outcome. We evaluate our approach against few-shots baselines and show that our personalized prior, quantitatively and qualitatively, outperforms state-of-the-art alternatives.
This project aims to provide users with an effective tool to correct and enhance photographs of themselves and of close friends and family. Previous deep-learning based approaches formed facial priors using large, generic and commonly biased datasets. Therefore, their results unfortunately underperformed for underrepresented groups. On the contrary, our approach forms a personalized prior for each individual using their on own photo album. This ensures our results to be faithful and accurate to the individual's identity, specifically for underrepresented groups in large available datasets. Hopefully, faithfully enhancing their images will improve users' satisfaction from their captured photographs.
At the same time, in the wrong hands, our tool might be used for malicious purposes. Our approach allows synthesizing novel images of individuals for which a photo album is available. Unfortunately, malicious parties might try to use such images to mislead viewers. This is a common issue, existing in even the simplest tools for enhancing, generating or manipulating images. Importantly, since our method requires ~100 images of an individual to operate, it sets the entry bar high for a malicious party while still being easily accessible for our intended users. Second, current research shows that deep generated images are still easily detectable as such, and thus reduces the risk of misleading audiences at scale. Future research in generative modeling, and specifically of personalized generative priors, must continue investigating and revalidating these concerns.
@article{nitzan2022mystyle,
title={MyStyle: A Personalized Generative Prior},
author={Nitzan, Yotam and Aberman, Kfir and He, Qiurui and Liba, Orly and Yarom, Michal and Gandelsman, Yossi and Mosseri, Inbar and Pritch, Yael and Cohen-Or, Daniel},
journal={arXiv preprint arXiv:2203.17272},
year={2022}
}