Ensemble-based face expression recognition approach for image sentiment analysis

Abstract

Sentiment analysis based on images is an evolving area of study. Developing a reliable facial expression recognition (FER) device remains a difficult challenge as recognizing emotional feelings reflected in an image is dependent on a diverse set of factors. This paper presented an ensemble-based model for FER that incorporates multiple classification models: i) customized convolutional neural network (CNN), ii) ResNet50, and iii) InceptionV3. The model averaging ensemble classifier method is used to ensemble the predictions from the three models. Subsequently, the proposed FER model is trained and tested on a dataset with an uncontrolled environment (FER-2013 dataset). The experiment demonstrated that ensembling multiple classifiers outperformed all single classifiers in classifying positive and neutral expressions (91.7%, 81.7% and 76.5% accuracy rate for happy, surprise, and neutral, respectively). However, when classifying disgust, anger, and sadness, the ResNet50 model alone is the better choice. Although the Custom CNN performs the best in classifying fear expression (55.7% accuracy), the proposed FER model can still classify fear expression with comparable performance (52.8% accuracy). This paper demonstrated the potential of using the ensemble-based method to enhance the performance of FER. As a result, the proposed FER model has shown a 72.3% accuracy rate.