Multimodal user's affective state analysis in naturalistic interaction

[journal]

Full reference

G. Caridakis, K. Karpouzis, M. Wallace, L. Kessous, N. Amir, Multimodal user's affective state analysis in naturalistic interaction, Journal of Multimodal User Interfaces 3(1-2), pp. 49-66, 2010

Abstract

Affective and human-centered computing have attracted an abundance of attention during the past years, mainly due to the abundance of environments and applications able to exploit and adapt to multimodal input from the users. The combination of facial expressions with prosody information allows us to capture the users’ emotional state in an unintrusive manner, relying on the best performing modality in cases where one modality suffers from noise or bad sensing conditions. In this paper, we describe a multi-cue, dynamic approach to detect emotion in naturalistic video sequences, where input is taken from nearly real world situations, contrary to controlled recording conditions of audiovisual material. Recognition is performed via a recurrent neural network, whose short term memory and approximation capabilities cater for modeling dynamic events in facial and prosodic expressivity. This approach also differs from existing work in that it models user expressivity using a dimensional representation, instead of detecting discrete ‘universal emotions’, which are scarce in everyday human-machine interaction. The algorithm is deployed on an audiovisual database which was recorded simulating human-human discourse and, therefore, contains less extreme expressivity and subtle variations of a number of emotion labels. Results show that in turns lasting more than a few frames, recognition rates rise to 98%.

Download

Click here to access the paper.

15 known citations

1. M. Tkalcic, U. Burnik and A. Kosir, Using affective parameters in a content-based recommender system for images, User Modeling and User-Adapted Interaction, vol 20, pp. 279-311, 2010
2. E. Alepis and M. Virvou, Multimodal Object Oriented User Interfaces in Mobile Affective Interaction, Multimedia Tools and Applications, vol 59(1), pp. 41-63, 2010
3. G. McKeown, M. Valstar, R. Cowie, M. Pantic and M. Schroder, The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent, IEEE Transactions on Affective Computing, vol 3(1), pp. 5-17, 2012
4. S. Scherer, M. Glodek, G. Layher, M. Schels, M. Schmidt, F. Schwenker, H. Neumann and G. Palm, A Generic Framework for the Inference of User States in Human Computer Interaction: How patterns of low level behavioral cues support complex User States in HCI, Journal on Multimodal User Interfaces, vol 6, pp. 117-141, 2012
5. S. Scherer, M. Glodek, F. Schwenker, N. Campbell and G. Palm, Spotting laughter in natural multiparty conversations: A comparison of automatic online and offline approaches using audiovisual data, ACM Transactions on Interactive Intelligent Systems, vol 2(1), 2012
6. L. Zhang, D. Tjondronegoro and V. Chandran, Evaluation of Texture and Geometry for Dimensional Facial Expression Recognition, International Conference on Digital Image Computing Techniques and Applications, 2011
7. D. Rocchesso, Explorations in Sonic Interaction Design, Logos, 2011
8. R. Cowie, G. McKeown and E. Douglas-Cowie, Tracing emotion: an overview, International Journal of Synthetic Emotions, vol 3(1), pp. 1-17, 2012
9. A. Panning, I. Siegert, A. Al-Hamadi, A. Wendemuth, D. Rösner, J. Frommer, G. Krell and B. Michaelis, Multimodal Affect Recognition in Spontaneous HCI Environment, IEEE International Conference on Signal Processing, Communications and Computing, 2012
10. B. De Carolis, I. Mazzotta, N. Novielli and S. Pizzutilo, User Modeling in Social Interaction with a Caring Agent in E. Martin, P.A. Haya and R.M. Carro (eds) User Modeling and Adaptation for Daily Routines, pp. 89-119, Springer Human Computer Interaction Series, 2013
11. B.D. Carolis, S. Ferilli and N. Novielli, Recognizing the User Social Attitude in Multimodal Interaction in Smart Environments, Proceedings of Ambient Intelligence, 2012
12. B.De Carolis and N. Novielli, Recognizing signals of social attitude in interacting with Ambient Conversational Systems, Journal on Multimodal User Interfaces, vol 8, pp. 143-60, 2014
13. L. Zhang, Towards spontaneous facial expression recognition in real-world video, PhD thesis, Queensland University of Technology, 2012
14. R. Misra and M.J. LaVaglio, The Role of Affective Interface in E-learning, The International Journal of Learning, vol 18(12), pp. 215-224, 2012
15. D. Prylipko, D. Rosner, I. Siegert, S. Günther, R. Friesen, M. Haase, B. Vlasenko and A. Wendemuth, Analysis of significant dialog events in realistic human–computer interaction, Journal on Multimodal User Interfaces, vol 8, pp. 75-86, 2014