Εργαστήριο Γνώσης και Αβεβαιότητας

Knowledge and Uncertainty Research Laboratory

Multimodal user's affective state analysis in naturalistic interaction

[journal]


Full reference

G. Caridakis, K. Karpouzis, M. Wallace, L. Kessous, N. Amir, Multimodal user's affective state analysis in naturalistic interaction, Journal of Multimodal User Interfaces 3(1-2), pp. 49-66, 2010


Abstract

Affective and human-centered computing have attracted an abundance of attention during the past years, mainly due to the abundance of environments and applications able to exploit and adapt to multimodal input from the users. The combination of facial expressions with prosody information allows us to capture the users’ emotional state in an unintrusive manner, relying on the best performing modality in cases where one modality suffers from noise or bad sensing conditions. In this paper, we describe a multi-cue, dynamic approach to detect emotion in naturalistic video sequences, where input is taken from nearly real world situations, contrary to controlled recording conditions of audiovisual material. Recognition is performed via a recurrent neural network, whose short term memory and approximation capabilities cater for modeling dynamic events in facial and prosodic expressivity. This approach also differs from existing work in that it models user expressivity using a dimensional representation, instead of detecting discrete ‘universal emotions’, which are scarce in everyday human-machine interaction. The algorithm is deployed on an audiovisual database which was recorded simulating human-human discourse and, therefore, contains less extreme expressivity and subtle variations of a number of emotion labels. Results show that in turns lasting more than a few frames, recognition rates rise to 98%.

replica Omega watches Breitling replica watches Cartier replica watches Replica IWC Regulateur watches cheap bell ross replica best bell ross replica bell ross replica for sale bell ross replica uk rolex replica watches replica Omega watches replica rolex watches rolex replica watches uk


Download

Click here to access the paper.


17 known citations

  1. M. Tkalcic, U. Burnik and A. Kosir, Using affective parameters in a content-based recommender system for images, User Modeling and User-Adapted Interaction, vol 20, pp. 279-311, 2010
  2. Yannakakis, G.N., Togelius, J., Khaled, R., Jhala, A, Karpouzis,K., Paiva, A. and Vasalou., A. (2010): Siren: Towards Adaptive Serious Games for Teaching Conflict Resolution. In proceedings European Conference on Games-Based Learning (ECGBL), 412-417
  3. E. Alepis and M. Virvou, Multimodal Object Oriented User Interfaces in Mobile Affective Interaction, Multimedia Tools and Applications, vol 59(1), pp. 41-63, 2010
  4. G. McKeown, M. Valstar, R. Cowie, M. Pantic and M. Schroder, The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent, IEEE Transactions on Affective Computing, vol 3(1), pp. 5-17, 2012
  5. S. Scherer, M. Glodek, G. Layher, M. Schels, M. Schmidt, F. Schwenker, H. Neumann and G. Palm, A Generic Framework for the Inference of User States in Human Computer Interaction: How patterns of low level behavioral cues support complex User States in HCI, Journal on Multimodal User Interfaces, vol 6, pp. 117-141, 2012
  6. S. Scherer, M. Glodek, F. Schwenker, N. Campbell and G. Palm, Spotting laughter in natural multiparty conversations: A comparison of automatic online and offline approaches using audiovisual data, ACM Transactions on Interactive Intelligent Systems, vol 2(1), 2012
  7. L. Zhang, D. Tjondronegoro and V. Chandran, Evaluation of Texture and Geometry for Dimensional Facial Expression Recognition, International Conference on Digital Image Computing Techniques and Applications, 2011
  8. "G. Caridakis, S. Asteriadis, K. Karpouzis, ""Non-manual cues in automatic sign language recognition"", ABRA: Affect and Behaviour Related Assistance Workshop, 4th ACM International PETRA Conference, Crete, Greece, May 25-27, 2011"
  9. D. Rocchesso, Explorations in Sonic Interaction Design, Logos, 2011
  10. R. Cowie, G. McKeown and E. Douglas-Cowie, Tracing emotion: an overview, International Journal of Synthetic Emotions, vol 3(1), pp. 1-17, 2012
  11. A. Panning, I. Siegert, A. Al-Hamadi, A. Wendemuth, D. Rösner, J. Frommer, G. Krell and B. Michaelis, Multimodal Affect Recognition in Spontaneous HCI Environment, IEEE International Conference on Signal Processing, Communications and Computing, 2012
  12. B. De Carolis, I. Mazzotta, N. Novielli and S. Pizzutilo, User Modeling in Social Interaction with a Caring Agent in E. Martin, P.A. Haya and R.M. Carro (eds) User Modeling and Adaptation for Daily Routines, pp. 89-119, Springer Human Computer Interaction Series, 2013
  13. B.D. Carolis, S. Ferilli and N. Novielli, Recognizing the User Social Attitude in Multimodal Interaction in Smart Environments, Proceedings of Ambient Intelligence, 2012
  14. B.De Carolis and N. Novielli, Recognizing signals of social attitude in interacting with Ambient Conversational Systems, Journal on Multimodal User Interfaces, vol 8, pp. 143-60, 2014
  15. L. Zhang, Towards spontaneous facial expression recognition in real-world video, PhD thesis, Queensland University of Technology, 2012
  16. R. Misra and M.J. LaVaglio, The Role of Affective Interface in E-learning, The International Journal of Learning, vol 18(12), pp. 215-224, 2012
  17. D. Prylipko, D. Rosner, I. Siegert, S. Günther, R. Friesen, M. Haase, B. Vlasenko and A. Wendemuth, Analysis of significant dialog events in realistic human–computer interaction, Journal on Multimodal User Interfaces, vol 8, pp. 75-86, 2014