4th International Conference

Digital Culture & AudioVisual Challenges

Interdisciplinary Creativity in Arts and Technology

Hybrid - Corfu/Online, May 13-14, 2022

ShareThis
Effects of Blind Face Restoration on Face Retrieval in Historical Image Analysis
Date and Time: 13/05/2022 (15:45-17:40)
Mert Seker, Anssi Männistö, Alexandros Iosifidis, Jenni Raitoharju

Blind face restoration [8, 2, 3] refers to the process of achieving a high quality, sharp and noise free facial image from an input that is low quality, blurry and noisy. The main aim of blind face restoration methods is to generate a visually pleasing version of the original image while maintaining the facial structure of the person. In contrast, face retrieval systems [6, 7] use as input a query image and faces in the search dataset that are matching with the query face are given as results. This is performed by comparing the query face with every existing face in the dataset using a face recognition algorithm [9, 1, 4] that can generate a similarity score between two faces. The similarity score is then compared to a predefined threshold in order to predict if the faces belong to the same person or not and the faces that are predicted to belong to the same person as the query image are given as matching results. There are many factors that can negatively impact the performance of a face recognition method such as facial pose, age, lighting conditions, occlusion, and quality of the images in terms of sharpness, noise and color.

In this work, we investigate if the negative impacts of low quality of historical images on face retrieval performance can be alleviated by applying blind face restoration prior to running facial retrieval. The experiments were conducted on historical images dating back to the World War II era [5]. This digital archive contains around 160 000 photos, of which only little over 800 are colored photos. Faces in many of the photographs are blurry, noisy, and mostly black and white. The ability to remove these deteriorations from historical photos, would potentially enhance the face retrieval performance on them. We selected for our experiments a subset of 442 faces, including 53 faces of five famous historical figures (Carl Gustaf Emil Mannerheim, Johan Wilhelm Rangell, Adolf Hitler, Prince Gustaf Adolf, and Risto Ryti), which were used as query images. High quality images of these five historical figures that were not included in the dataset were used for querying.

Two different versions of a state-of-the-art blind face restoration method were applied on the face dataset. The first version aims to restore the faces in terms of sharpness and noise, while the other version also colorizes the images in addition to performing the same restoration as the first version. This resulted in three different versions of the same dataset: original version, where the quality is low, an upscaled version without colorization, and an upscaled version with colorization. The query images of the historical figures were then used separately in the three versions of the face dataset to find matching images by using face recognition. This way, three different sets of results were obtained, one for each version of the dataset. The results showed that upscaling and colorization of the faces resulted in a very minor boost in finding more matching images. However, it was observed that applying blind face restoration resulted in a significantly fewer wrong matches (faces belonging to different people that were wrongly predicted as belonging to the same person). The upscaled and colorized version of the dataset had the least amount of wrong matches, followed by the upscaled and non-colorized version, and finally the original version. In other words, applying blind face restoration did not help in finding more images of the query person that were missed before due to low quality, but it helped greatly in removing the false results.

In conclusion, we explored and analyzed the positive impacts of blind face restoration on face retrieval. We observed better face retrieval performance in a series of experiments where we applied different face restoration methods on historical low-quality images. The significant reduction in the number of false face matches by using face restoration can greatly improve the usability of the results of a face retrieval applied on a large dataset.

References
[1] Anil K Jain and Stan Z Li. Handbook of face recognition. Vol. 1. Springer,2011.
[2] Xiaoming Li et al. “Blind face restoration via deep multi-scale component dictionaries”. In: European Conference on Computer Vision. Springer.2020, pp. 399–415.
[3] Yehong Liao and Xueyin Lin. “Blind image restoration with eigen-face sub-space”. In: IEEE Transactions on Image Processing 14.11 (2005), pp. 1766-1772.
[4] Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. “Deep Face Recognition”. In: Proceedings of the British Machine Vision Conference
(BMVC). Ed. by Mark W. Jones Xianghua Xie and Gary K. L. Tam. BMVA Press, Sept. 2015, pp. 41.1–41.12.
[5] SA-kuva: Finnish Wartime Photograph Archive. Finland: The Finnish Defence Forces. url: http://sa-kuva.fi/.
[6] Jinhui Tang, Zechao Li, and Xiang Zhu. “Supervised deep hashing for scalable face image retrieval”. In: Pattern Recognition 75 (2018). Distance Metric Learning for Pattern Recognition, pp. 25–32.
[7] Ahmad S Tarawneh et al. “Deep face image retrieval: A comparative study with dictionary learning”. In: 2019 10th International Conference on Information and Communication Systems (ICICS). IEEE. 2019, pp. 185–192.
[8] Xintao Wang et al. “Towards real-world blind face restoration with generative facial prior”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, pp. 9168–9178.
[9] Wenyi Zhao et al. “Face recognition: A literature survey”. In: ACM com- puting surveys (CSUR) 35.4 (2003), pp. 399–458.

Mert Seker

Mert Seker received his MSc. degree from Tampere University, Finland in 2021 majoring in Data Engineering & Machine Learning. He is currently work- ing as a researcher at Tampere University, working on interdisciplinary projects involving the application of state-of-the-art computer vision and machine learn- ing methods on photographic studies. His main fields of studies are computer vision and machine learning.

Anssi Männistö
Anssi Männistö received his Ph.D. degree in the Department of Social Sci- ences at the University of Tampere, Finland in 2000. Since 2003 he has worked as a senior lecturer of visual journalism, currently in the Faculty of Information Technology and Communication Sciences, Tampere University, Finland. His research interests include new modes of journalism and photojournalism, use of modern data visualizations in journalism and education and developing meth- ods in quantitative and content analyze of photographs.
Alexandros Iosifidis
Alexandros Iosifidis is a Professor at Aarhus University, Denmark. He has contributed in more than thirty R&D projects financed by EU, Finnish, and Danish funding agencies and companies. He has co-authored 90+ articles in international journals and 120+ papers in international conferences/workshops proposing novel Machine Learning techniques and their application in a vari- ety of coming from Computer Vision, Signal Processing and Financial Modeling.
Jenni Raitoharju
Jenni Raitoharju received her Ph.D. degree at Tampere University of Tech- nology, Finland in 2017. She works as a Senior Research Scientist at the Finnish Environment Institute, Jyv ̈askyl ̈a. She has co-authored 26 international journal papers and 42 papers in international conferences. Her research interests include machine learning and pattern recognition methods along with applications in biomonitoring, humanities, and autonomous systems. She is the chair of Young Academy Finland 2019-2022.

Back
   
Text To SpeechText To Speech Text ReadabilityText Readability Color ContrastColor Contrast
Accessibility Options