eCite Digital Repository
Context-aware emotion recognition in the wild using spatio-temporal and temporal-pyramid models
Citation
Do, N-T and Kim, S-H and Yang, H-J and Lee, G-S and Yeom, S, Context-aware emotion recognition in the wild using spatio-temporal and temporal-pyramid models, Sensors, 21, (7) Article 2344. ISSN 1424-8220 (2021) [Refereed Article]
 | PDF (Published version) 2Mb |
Copyright Statement
Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Abstract
Emotion recognition plays an important role in human–computer interactions. Recent studies have focused on video emotion recognition in the wild and have run into difficulties related to occlusion, illumination, complex behavior over time, and auditory cues. State-of-the-art methods use multiple modalities, such as frame-level, spatiotemporal, and audio approaches. However, such methods have difficulties in exploiting long-term dependencies in temporal information, capturing contextual information, and integrating multi-modal information. In this paper, we introduce a multi-modal flexible system for video-based emotion recognition in the wild. Our system tracks and votes on significant faces corresponding to persons of interest in a video to classify seven basic emotions. The key contribution of this study is that it proposes the use of face feature extraction with context-aware and statistical information for emotion recognition. We also build two model architectures to effectively exploit long-term dependencies in temporal information with a temporal-pyramid model and a spatiotemporal model with "Conv2D+LSTM+3DCNN+Classify" architecture. Finally, we propose the best selection ensemble to improve the accuracy of multi-modal fusion. The best selection ensemble selects the best combination from spatiotemporal and temporal-pyramid models to achieve the best accuracy for classifying the seven basic emotions. In our experiment, we take benchmark measurement on the AFEW dataset with high accuracy.
Item Details
| Item Type: | Refereed Article |
|---|---|
| Keywords: | video emotion recognition, spatiotemporal, temporal-pyramid, best selection ensemble, facial emotion recognition |
| Research Division: | Information and Computing Sciences |
| Research Group: | Artificial intelligence |
| Research Field: | Artificial intelligence not elsewhere classified |
| Objective Division: | Information and Communication Services |
| Objective Group: | Information systems, technologies and services |
| Objective Field: | Artificial intelligence |
| UTAS Author: | Yeom, S (Dr Soonja Yeom) |
| ID Code: | 143632 |
| Year Published: | 2021 |
| Web of Science® Times Cited: | 1 |
| Deposited By: | Information and Communication Technology |
| Deposited On: | 2021-03-28 |
| Last Modified: | 2021-05-26 |
| Downloads: | 7 View Download Statistics |
Repository Staff Only: item control page