Cover Image

Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos

Facial Micro-Expressions (MEs) reveal a person's hidden emotions in high stake situations within a fraction of a second and at a low intensity. The broad range of potential real-world applications that can be applied has drawn considerable attention from researchers in recent years. However, bo...

Full description

Saved in:
Bibliographic Details
Main Authors: Liong, Gen-Bing, See, John, Chan, Chee Seng
Format: Article
Published: Elsevier 2023
Subjects:
QA75 Electronic computers. Computer science
Online Access:http://eprints.um.edu.my/39363/
Tags: Add Tag
No Tags, Be the first to tag this record!
id my.um.eprints.39363
record_format eprints
spelling my.um.eprints.393632023-11-28T08:17:43Z http://eprints.um.edu.my/39363/ Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos Liong, Gen-Bing See, John Chan, Chee Seng QA75 Electronic computers. Computer science Facial Micro-Expressions (MEs) reveal a person's hidden emotions in high stake situations within a fraction of a second and at a low intensity. The broad range of potential real-world applications that can be applied has drawn considerable attention from researchers in recent years. However, both spotting and recognition tasks are often treated separately. In this paper, we present Micro-Expression Analysis Network (MEAN), a shallow multi-stream multi-output network architecture comprising of task-specific (spotting and recognition) networks that is designed to effectively learn a meaningful representation from both ME class labels and location-wise pseudo-labels. Notably, this is the first known work that addresses ME analysis on long videos using a deep learning approach, whereby ME spotting and recognition are performed sequentially in a two-step procedure: first spotting the ME intervals using the spotting network, and proceeding to predict their emotion classes using the recognition network. We report extensive benchmark results on the ME analysis task on both short video datasets (CASME II, SMIC-E-HS, SMIC-E-VIS, and SMIC-E-NIR), and long video datasets (CAS(ME)2 and SAMMLV); the latter in particular demonstrates the capability of the proposed approach under unconstrained settings. Besides the standard measures, we promote the usage of fairer metrics in evaluating the performance of a complete ME analysis system. We also provide visual explanations of where the network is ``looking''and showcasing the effectiveness of inductive transfer applied during network training. An analysis is performed on the in-the-wild dataset (MEVIEW) to open up further research into real-world scenarios. Elsevier 2023-01 Article PeerReviewed Liong, Gen-Bing and See, John and Chan, Chee Seng (2023) Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos. Signal Processing-Image Communication, 110. ISSN 0923-5965, DOI https://doi.org/10.1016/j.image.2022.116875 <https://doi.org/10.1016/j.image.2022.116875>. 10.1016/j.image.2022.116875
institution Universiti Malaya
building UM Library
collection Institutional Repository
continent Asia
country Malaysia
content_provider Universiti Malaya
content_source UM Research Repository
url_provider http://eprints.um.edu.my/
topic QA75 Electronic computers. Computer science
spellingShingle QA75 Electronic computers. Computer science
Liong, Gen-Bing
See, John
Chan, Chee Seng
Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos
description Facial Micro-Expressions (MEs) reveal a person's hidden emotions in high stake situations within a fraction of a second and at a low intensity. The broad range of potential real-world applications that can be applied has drawn considerable attention from researchers in recent years. However, both spotting and recognition tasks are often treated separately. In this paper, we present Micro-Expression Analysis Network (MEAN), a shallow multi-stream multi-output network architecture comprising of task-specific (spotting and recognition) networks that is designed to effectively learn a meaningful representation from both ME class labels and location-wise pseudo-labels. Notably, this is the first known work that addresses ME analysis on long videos using a deep learning approach, whereby ME spotting and recognition are performed sequentially in a two-step procedure: first spotting the ME intervals using the spotting network, and proceeding to predict their emotion classes using the recognition network. We report extensive benchmark results on the ME analysis task on both short video datasets (CASME II, SMIC-E-HS, SMIC-E-VIS, and SMIC-E-NIR), and long video datasets (CAS(ME)2 and SAMMLV); the latter in particular demonstrates the capability of the proposed approach under unconstrained settings. Besides the standard measures, we promote the usage of fairer metrics in evaluating the performance of a complete ME analysis system. We also provide visual explanations of where the network is ``looking''and showcasing the effectiveness of inductive transfer applied during network training. An analysis is performed on the in-the-wild dataset (MEVIEW) to open up further research into real-world scenarios.
format Article
author Liong, Gen-Bing
See, John
Chan, Chee Seng
author_facet Liong, Gen-Bing
See, John
Chan, Chee Seng
author_sort Liong, Gen-Bing
title Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos
title_short Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos
title_full Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos
title_fullStr Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos
title_full_unstemmed Spot-then-recognize: A micro-expression analysis network for seamless evaluation of long videos
title_sort spot-then-recognize: a micro-expression analysis network for seamless evaluation of long videos
publisher Elsevier
publishDate 2023
url http://eprints.um.edu.my/39363/
_version_ 1783876687002337280
score 13.211869

Similar Items