Contrôle du mouvement de la tête et du clignement des yeux d’un avatar à partir de la parole par des réseaux profonds
- Laboratoire : Inria Nancy Grand Est – LORIA
- Ville : Nancy, France.
- Équipe : Multispeech
- Thématique : Intelligence artificielle / Interaction / Traitement de la parole multimodale
- Contact : Slim Ouni (Slim.Ouni@loria.fr)
(For English Version, see below)
Au cours des interactions sociales, les orateurs produisent souvent des gestes spontanés pour accompagner leur discours. Ces mouvements coordonnés des mains, de la tête, du regard, de la posture et de certaines expressions faciales transmettent des intentions communicatives et modulent la façon dont les auditeurs perçoivent le message d’une manière subtile, mais importante. Bien que la corrélation entre le clignement des yeux et la parole ne soit pas claire, il est important d’ajouter ce geste pour avoir un comportement communicatif naturel pendant la parole. L’étude des de contrôles des mouvements de la tête à partir de la parole est motivée par le fort couplage qui existe entre les mouvements de la tête et la parole, fournissant une solution attrayante pour créer des comportements qui sont synchronisés temporellement avec la parole.
Le but de ce travail est de développer un modèle de contrôle du mouvement de la tête pendant la parole. Le but est de rendre le mouvement de la tête cohérent et intelligible avec la parole. De même, un modèle de contrôle du clignement des yeux sera également développé. Nous avons déjà enregistré un grand corpus de capture de mouvement d’un locuteur prononçant plusieurs phrases de manière normale et expressive. Ce corpus a été enregistré en même temps que la parole acoustique. L’idée principale est d’utiliser des techniques d’apprentissage profond pour entrainer un réseau neuronal capable d’estimer les mouvements de la tête (et le clignement des yeux) à partir de la parole pour les articulations normales et expressives.
N’hésitez pas à contacter le responsable du stage pour tout complément d’information.
Compétences espérées
De bonnes connaissances informatique et en machine learning sont nécessaires. Une première expérience avec l’utilisation d’une librairie de réseaux de neurones (comme PyTorch ou TensorFlow, …) est appréciée.
Bourse d’excellence
Le laboratoire propose un nombre limité de bourses d’excellence pour des candidats excellents (un bon parcours académique) qui sont des étudiants français (hors de la région Grand-Est) ou étrangers, qui souhaitent faire une thèse par la suite. Cette bourse couvre la mobilité à une hauteur de 1000€ et une indemnité de 1000€ par mois. Pour candidater à ce financement, il faut répondre à cette offre de stage en m’envoyant votre CV avant le 26/11/2020
Bibliographie
- Kobald, S. O., Wascher, E., Heppner, H., & Getzmann, S. (2019). Eye blinks are related to auditory information processing: evidence from a complex speech perception task. Psychological research, 83(6), 1281-1291.
- Ishi, C. T., Funayama, T., Minato, T., & Ishiguro, H. (2016, October). Motion generation in android robots during laughing speech. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3327-3332). IEEE.
- Murali, S., & Haendel, B. (2020). The Latency of Spontaneous Eye Blinks Marks Relevant Visual and Auditory Information Processing. bioRxiv.
- Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological science, 15(2), 133-137.
- Hadar, U., Steiner, T. J., Grant, E. C., & Rose, F. C. (1983). Kinematics of head movements accompanying speech during conversation. Human Movement Science, 2(1-2), 35-46.
- Hömke, P., Holler, J., & Levinson, S. C. (2017). Eye blinking as addressee feedback in face-to-face conversation. Research on Language and Social Interaction, 50(1), 54-70.
- Esteve-Gibert, N., Borràs-Comes, J., Asor, E., Swerts, M., & Prieto, P. (2017). The timing of head movements: The role of prosodic heads and edges. The Journal of the Acoustical Society of America, 141(6), 4727-4739.
Head movement and eye blinking driven by speech using deep learning techniques
During social interactions, speakers often produce spontaneous gestures to accompany their speech. These coordinated movements of hands, head, eye gaze, posture and some facial expressions, convey communicative intentions, and modulate how listeners perceive the message in a subtle but important way. Although, it is not clear the correlation between eye blinking with speech, it is important to add this gesture to have a natural communicative behavior during speech. Studying speech-driven head movement methods are motivated by the strong coupling that exists between head movements and speech, providing an appealing solution to create behaviors that are timely synchronized with speech.
The purpose of this work is to develop a control model of the head movement during speech. The purpose is to make head movement coherent and intelligible with speech. Similarly, a control model for eye blinking will also be developed. We have already recorded a large motion capture corpus of a speaker uttering several sentences in normal and expressive manner. This corpus was recorded simultaneously with acoustic speech. The main idea is to use deep learning techniques to train a neural network that can estimate head movements (and eye blinking) from speech for normal articulation and expressive ones.
Feel free to contact the internship supervisor for any further information.
Skills and profile
Appropriate candidates would have strong background in computer science and machine learning. A first experience with the use of a neural network library (such as PyTorch or TensorFlow,…) is appreciated.
Context
The work will be done within a dynamic research team Multispeech research team), at the research center Inria Nancy Grand Est (LORIA) and you will integrate a team composed of both experienced and young researchers (PhD, postdocs and engineers) and closely supervised by a senior researcher. We have a motion capture facilities and an articulograph in the laboratory that can be used to acquire data in this project. Several speech processing tools are available in the team.
This internship can be a great opportunity to discover research in the field of spoken communication and 3D avatar animation using automatic learning techniques.
Excellence Internship (Bourse d’excellence)
It is possible to apply for a highly competitive internship funding for outstanding candidates (with good academic background) who are French students (outside the Grand-Est region) or foreign students, who are interested to pursue doctoral thesis in the lab. This funding will cover mobility expenses (limited to 1000€ ) and 1000€ per month. To apply for this funding, you need contact me and send me your CV before 26/11/2020.
Bibliography
- Kobald, S. O., Wascher, E., Heppner, H., & Getzmann, S. (2019). Eye blinks are related to auditory information processing: evidence from a complex speech perception task. Psychological research, 83(6), 1281-1291.
- Ishi, C. T., Funayama, T., Minato, T., & Ishiguro, H. (2016, October). Motion generation in android robots during laughing speech. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 3327-3332). IEEE.
- Murali, S., & Haendel, B. (2020). The Latency of Spontaneous Eye Blinks Marks Relevant Visual and Auditory Information Processing. bioRxiv.
- Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological science, 15(2), 133-137.
- Hadar, U., Steiner, T. J., Grant, E. C., & Rose, F. C. (1983). Kinematics of head movements accompanying speech during conversation. Human Movement Science, 2(1-2), 35-46.
- Hömke, P., Holler, J., & Levinson, S. C. (2017). Eye blinking as addressee feedback in face-to-face conversation. Research on Language and Social Interaction, 50(1), 54-70.
- Esteve-Gibert, N., Borràs-Comes, J., Asor, E., Swerts, M., & Prieto, P. (2017). The timing of head movements: The role of prosodic heads and edges. The Journal of the Acoustical Society of America, 141(6), 4727-4739.
