Enhancing the Realism of Sketch and Painted Portraits with Adaptable Patches

Abstract

Realizing unrealistic faces is a complicated task that requires a rich imagination and comprehension of facial structures. When face matching, warping, or stitching techniques are applied, existing methods are generally incapable of capturing detailed personal characteristics, are disturbed by block boundary artefacts, or require painting-photo pairs for training. This paper presents a data-driven framework to enhance the realism of sketch and portrait paintings based only on photo samples. It retrieves the optimal patches of adaptable shapes and numbers according to the content of the input portrait and collected photos. These patches are then seamlessly stitched by chromatic gain and offset compensation and multilevel blending. Experiments and user evaluations show that the proposed method is able to generate realistic and novel results for a moderately sized photo collection.

Demo Video

Experiments

This section demonstrates and compares the results of the proposed method with those generated by related methods: regular-patch-based method (extension of the synthesis component of the Visio-lization method [Mohammed et al. 2009]), and best-matched face method.

Several results of the proposed and comparative methods are shown below. Please refer to our supplementary document for additional experiments regarding other characteristic labels.

Figure. Realizing faces from cartoons, sketches and stylized paintings. The first to the fourth columns are respectively the input portraits, results of the proposed method, results of the regular-patch-based method, and results of the best-matched face method.

Figure. Influence of the proposed edge feature on the compared regular-patch-based method. (a) The input sketch. (b) Result of the regular-patch-based method with edge features. (c) Result of the regular-patch-based method without edge features.

Publication

Yin-Hsuan Lee, Yu-Kai Chang, Yu-Lun Chang, I-Chen Lin*, Yu-Shuen Wang, Wen-Chieh Lin, "Enhancing the Realism of Sketch and Painted Portraits with Adaptable Patches," Computer Graphics Forum (also presented in Pacific Graphics'17), 37(1): 214-225, Feb., 2018. (SCI, EI)

Paper: preprint_version (about 7MB), published version (link to the Wiley Online Library)
Supplementary file: Lee_FaceRealism_CGF18_SupDoc (pdf, about 1.96MB)

BibTex

@article{LeeCGF18,
author = {Yin-Hsuan Lee and YYu-Kai Chang and Yu-Lun Chang and I-Chen Lin and Yu-Shuen Wang and Wen-Chieh Lin},
title = {Enhancing the Realism of Sketch and Painted Portraits with Adaptable Patches},
journal = {Computer Graphics Forum (also presented in Pacific Graphics'17)},
volume = {37},
number = {1},
pages = {214-225},
month = {Feb},
year = {2018},
doi = {10.1111/cgf.13261}
}

Acknowledgements

The authors would like to acknowledge the providers of all the images and photographs published on the Internet. These images are for academic and noncommercial usage only. The authors also thank the volunteers who participated in the user evaluations. This paper was partially supported by the Ministry of Science and Technology, Taiwan, under grant no. MOST 104-2221-E-009-129-MY2.