Deep learning-based Facial Micro-Expression Analysis: A Survey of Datasets and Algorithms
Mingjian Xie
Macau University of Science and Technology, Macao
Submission:November 25, 2023; Published:January 30, 2024
*Corresponding author:Mingjian xie, Macau University of Science and Technology, Macao
How to cite this article:Mingjian x. Deep learning-based Facial Micro-Expression Analysis: A Survey of Datasets and Algorithms. Curr Trends Biomedical Eng & Biosci. 2024; 22(3): 556086. DOI:10.19080/CTBEB.2024.22.556086
Abstract
In recent years, a variety of facial expression feature extraction methods and machine learning methods have been applied to the automatic recognition of facial expressions, and the automatic detection and recognition technology of micro-expressions has also been developed. This article mainly introduces the seven micro-expression datasets and five algorithms for facial micro-expression recognition based on deep learning as the basis for micro-expression recognition. With the gradual deepening of the development in micro-expression, the existing datasets are difficult to meet the development requirements of deep learning, and the construction of more diverse large datasets is an important direction of deep learning research. At the same time, how to design a more effective deep learning algorithm on the existing dataset is worth continuing to explore. Index Terms-micro-expression, datasets, algorithms, survey.
Keywords:Algorithms; Datasets; Facial Micro-Expression; Clinical Medicine; criminal Investigation; Computer facial expression
Introduction
The human face is the most important and direct carrier for human emotion expression and communication. Human facial expressions play an important role in people’s daily communication and emotional expression. Normally, a complete facial expression lasts 0.5 to 4 seconds [1], which is easily recognized by human. In recent years, with the development of pattern recognition technology and face analysis technology, the automatic recognition technology of facial expressions has been made significant progress, and a variety of facial expression feature extraction methods and machine learning methods have been applied to the automatic recognition of facial expressions. However, psychological research pointed out that human emotion recognition based on facial expressions can be misleading [2]. As a special facial expression, micro-expression is defined as a fast facial movement that is not dominated by people’s consciousness and can reveal true emotions [3]. Micro-expression was first discovered by Haggard and Isaacs [4]. They found that micro-expression which can reveal depression is related to the self-defense mechanism. Because micro-expression is a natural and uncontrollable true emotional expression that can reflect people’s real thoughts, it is used in criminal Investigation, Justice, Clinical Medicine and other fields, and has a wide range of application prospects. Micro-expression is a special facial expression.
Compared with ordinary expressions, micro-expression mainly has the following characteristics
a) The duration of micro-expressions is shorter, usually only 0.065s~0.5s [5].
b) The intensity of micro-expression movements is lower and difficult to detect.
c) Micro-expressions is produced in an unconscious state which is usually difficult to conceal or disguise.
d) The analysis of micro-expression usually needs to be in the video, while ordinary expressions can be analyzed in the image.
Computer facial expression research began in the 1970s. Suwa and Sugie [6] made an initial attempt at facial expression recognition in 1978. They tracked a facial video animation and obtained the movement patterns of 20 key points on each frame of the picture. And they compared this motion rule with the pre-established key point motion models of different expressions, and performed expression analysis, made expression analysis. Coding and recognition as an important part of multi-mode human-computer interaction attract attention. The real development of facial expression recognition was in the 1990s when major well-known research institutes around the world began to conduct research in this area. With the gradual deepening of the understanding of micro-expression, the automatic detection and recognition technology of micro-expression has also been developed. This paper mainly introduces the research progress of micro-expression datasets and facial micro-expression recognition based on Deep learning.
Micro-expression Datasets
The development of micro-expression analysis techniques largely has been dependent on complete Datasets. In the early research of automatic micro-expression recognition, due to the difficulty of constructing spontaneous micro-expression Datasets, researchers used performance micro-expression data for experiments. Shreve et al. established a micro-expression Dataset named USF-HD [7], which includes 100 clips of performing microexpression. They played some videos with micro-expressions to the participants and asked them to imitate the micro-expressions in the video. Polikovsky et al. [8] also constructed a performance micro-expression Datasets. They asked participants to perform seven expressions at low intensity and asked to return to a neutral expression as soon as possible. However, this approach makes the collection process of micro-expression samples missing a lot of details, and the Datasets lacks an accurate definition of microexpression categories and explanations for the labeling process.
The biggest problem with the performance micro-expression dataset is that it is obviously different from the real spontaneous micro-expression. Therefore, it is necessary to construct a spontaneous micro-expression Datasets with as much data as possible and representative. To this end, Zhao Guoying et al. established the first spontaneous micro-expression Datasets SMIC [9]. They let the experimenters watch video clips that can strongly induce emotions, and asked the experimenters to hide their true feelings, maintain a neutral expression when watching the film, and then report true feelings. In addition, CASME [10], CAS(ME)² [11], CASMEⅡ [12], SAMM [13], and MMEW [14] microexpression datasets use similar expression induction methods. MEVIEW [15] used another quite different elicitation paradigm: namely, constructing a high-stakes situation by making use of poker games or TV interviews with difficult questions.
The conditions of various micro-expression Datasets are shown in (Table 1). The table summarizes the recording parameters of each Datasets, the number of experiments, the number of samples, whether to mark AU (Action Unit), and the number of micro-expression categories. Sufficient micro-expression training samples are a prerequisite for micro-expression analysis and recognition. Comprehensive comparison of various Datasets, the MMEW [14] Datasets has the largest number of samples, high video frame rate, high image resolution, and well-labeled. But in the existing micro-expression research, since this data set was just established in 2021, there are relatively few related studies using this data set.
Recognition Algorithm based on Deep Learning
Micro-expression recognition refers to classifying a given micro-expression segment into a specific emotion category. Compared with the micro-expression detection technology, there are more researches on the micro-expression recognition technology. In recent years, due to the unique temporal and spatial characteristics of micro-expression, deep learning technology has gradually been introduced into the field of microexpression recognition, and feature extraction no longer depends on human design. This essay mainly summarizes and compares the five classic methods of using deep learning (excluding transfer learning) to solve facial micro-expression recognition.
Patel et al. [16] proposed SDF (Selective Deep Features) as the first essay to explore the application of deep learning in the task of micro-expression recognition. Due to the lack of micro-expression data, SDF is proposed to use the transfer learning of objects and facial expressions based on the CNN model, using feature selection to remove deep features that are irrelevant to the task. It has good generalization ability. Hao et al. [17] proposed an efficient microexpression recognition deep network (Deep Belief Network Based on Double Weber Local Descriptor) which is A feature selection framework of deep learning for micro-expression recognition task. The network adopts a two-stage strategy: the first stage uses dual Weber local descriptors to extract initial local texture features. In the second stage, the deep belief network (DWLD+DBN) is used to extract global texture features.
Min Peng et al. [18] proposed a dual-stream network called DTSCNN (Dual Temporal Scale Convolutional Neural Network) to construct a micro-expression recognition system. Each stream contains an independent shallow neural network to prevent overfitting. In order to allow the network to acquire higher-level features, the author also input the optical flow sequence into the network. Wang et al. [19] proposed a TLCNN (Transfer Longterm Convolutional Neural Network), which uses Deep CNN to extract features from each frame of micro-expression video and combines LSTM to learn sequence information. DH Kim et al. [20] proposed the name ELRCN (Enriched Long-term Recurrent Convolutional Network). It extracts deep spatial features through the CNN module, encodes each micro-expression frame into a feature vector, and uses time domain learning. However, the amount of data in the micro-expression Datasets is small, and the method used in the essay has a certain risk of overfitting. Deep learning is currently the most popular method of microexpression recognition. For ease of reading, (Table 2) summarizes the advantages and disadvantages of these five algorithms. Xianye Ben et al. [14] evaluated the recognition rate of the most advanced methods in the field of deep learning on MMEW and SAMM (Table 3).
It can be seen that with the development of deep learning, the accuracy of recognition has also improved. At present, the existing micro-expression recognition mainly focuses on the extraction of micro-expression features. Since micro-expression is a change in time series, the main works currently focuses on the extraction of time series features from video sequences. For the classification steps of micro-expression, most of them directly use existing machine learning methods. At present, there are two main difficulties in the work of micro-expression recognition: firstly, the duration of micro-expression is short, the action intensity is low, and features are difficult to extract. Therefore, we need to perform appropriate data preprocessing and feature extraction; Secondly, due to the difficulties in data collection and identification of microexpression, there are few existing micro-expression Datasets, which makes the application of deep learning in the recognition of micro-expression difficult.
![Click here to view Large Table 1](images/CTBEB.MS.ID.556086.T001.png)
![Click here to view Large Table 2](images/CTBEB.MS.ID.556086.T002.png)
![Click here to view Large Table 3](images/CTBEB.MS.ID.556086.T003.png)
At present, there are two main trends in the research of facial micro-expression recognition based on deep learning. The first is Transfor-Learning from other domains, which is the migration from ordinary expression recognition models to micro-expression recognition; the second is Cross-Dataset learning, which is training the model on one Dataset and verifying on another Dataset, or jointly training and verifying on multiple Datasets. With the development of deep learning, micro-expression recognition has shown an end-to-end trend. However, the existing Datasets are difficult to meet the development requirements of deep learning, and the construction of more diverse large Datasets is an important direction of deep learning research. At the same time, how to design a more effective deep learning algorithm on the existing Datasets is worth continuing to explore.
References
- Ekman P (2003) Emotions revealed: recognizing faces and feelings to improve communication and emotional life. Holt Paperback 128(8): 140-140.
- Porter S, Brinke LT (2008) Reading between the lies. Psychological Science 19(5): 508-514.
- Ekman P (2010) Darwin, deception, and facial expression. Annals of the New York Academy of Sciences 1000(1): 205-221.
- Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Springer US.
- Yan WJ, Wu Q, Liang J, Chen YH, Fu X (2013) How fast are the leaked facial expressions: the duration of micro-expressions. Journal of Nonverbal Behavior 37(4): 217-230.
- Suwa M, Sugie N, Fujimora K (1978) A preliminary note on pattern recognition of human emotional expression.
- Shreve M, Godavarthy S, Goldgof D, Sarkar S (2011) Macro- and micro-expression spotting in long videos using spatio-temporal strain. IEEE Computer Society. pp. 51-56.
- Polikovsky S, Kameda Y, Ohta Y (2010) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. International Conference on Crime Detection & Prevention. IET.
- Li X, Pfister T, Huang X, Zhao G, Pietikainen M (2013) A Spontaneous Micro-expression Database: Inducement, collection and baseline. Automatic Face and Gesture Recognition (FG), 10th IEEE International Conference and Workshops on. IEEE.
- Yan WJ, Wu Q, Liu YJ, Wang SJ, Fu X (2013) CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. IEEE International Conference & Workshops on Automatic Face & Gesture Recognition. IEEE.
- Qu F, Wang, SJ, Yan WJ, Li H, Wu S, et al. (1949) Cas(me)^2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing. Pp. 1-1.
- Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, et al. (2014) Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. Plos One 9(1): e86041.
- Davison AK, Lansley C, Costen N, Tan K, Yap MH (2018) SAMM: a spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing 9(1): 116-129.
- Ben X, Ren Y, Zhang J, Wang SJ, Kpalma K et al., (2021) Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(9): 5826 – 5846.
- JM Petr Husak, Jan Cech (2017) Spotting facial micro-expressions “in the wild”. In: 22nd Computer Vision Winter Workshop.
- D Patel, Hong X, Zhao G (2017) Selective deep features for micro-expression recognition. International Conference on Pattern Recognition. IEEE.
- Hao XL, Tian M (2017) Deep Belief Network Based on Double Weber Local Descriptor in Micro-expression Recognition. International Conference on Multimedia and Ubiquitous Engineering International Conference on Future Information Technology.
- Min P, Wang C, Chen T, Liu G, Fu X (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Frontiers in Psychology 8: 1745.
- Wang, Jing S, Bing J, Liu, Yong J, et al. (2018) Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing, pp. 251-262.
- Khor HQ, See J, Phan R, Lin W (2018) Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition. IEEE International Conference on Automatic Face & Gesture Recognition. arXiv, pp. 667-674.