Facial emotion recognition under face mask occlusion using vision transformers

Abstract

Facial emotion recognition (FER) systems face significant challenges when individuals wear face masks, as critical facial regions are occluded. This paper addresses this limitation by employing vision transformers (ViT), which offer a promising alternative with reduced computational complexity compared to traditional deep learning methods. We propose a ViT-based FER framework that fine-tunes a pre-trained ViT architecture to enhance emotion recognition under mask-induced occlusion. The model is fine-tuned and evaluated on the AffectNet dataset, which originally represents eight emotion categories. These categories are restructured into five broader classes to mitigate the impact of occluded features. The model’s performance is assessed using standard metrics, including accuracy, precision, recall, and F1 score. Experimental results demonstrate that the proposed framework achieves an accuracy of 81%, outperforming several state-of-the-art approaches. These findings highlight the potential of vision transformers in recognizing emotions under masked conditions and support the development of more robust FER systems for real-world applications in healthcare, surveillance, and human–computer interaction. This work introduces a scalable and effective approach that integrates self-attention, synthetic mask augmentation, and emotion class restructuring to improve emotion recognition under facial occlusion.