Gesture Learning in Music and Language 01:07:52
- Set Séminaires Recherche & Technologie
- Saison 2018-2019 - None - None > Gesture Learning in Music and Language, par Xiao XIAO
- Sept. 25, 2018
- Xiao Xiao (conférencier)
How do we learn to perform skilled gestures, and how can interactive technologies help us in our learning? This talk examines these questions in three domains: piano, Mandarin Chinese, and theremin.
I first present two projects to support learning the expression dimension of piano playing— MirrorFugue and Andante. MirrorFugue simulates the presence of a virtual pianist whose reflection appears to play the physically moving keys. It encourages learning expressive gestures through imitation. Andante presents music as miniature figures that appear to walk and dance on the piano keyboard. It aims to help children understand the expressivity of rhythms and phrasing in terms of movements of familiar bodily movements.
Next, I discuss my current project on learning Mandarin tones using a vocal synthesizer controlled by hand gestures on a graphical tablet. Learning to identify and pronounce tones is one of the biggest difficulties for non-native learners of Mandarin. In collaboration with LAM (Lutheries - Acoustique - Musique) at UPMC, I developed a method that enables learners to practice pronunciation by tracing visual guides derived from the frequency curve of native speakers. I present results from preliminary evaluations of this method.
Finally, I share personal finding from my experiences teaching myself how to play the theremin in the past 16 months, and the process of developing gesture vocabularies for intervals, musical motifs, and expressive phrasing. This talk concludes by reflecting on common themes across all three domains, including learning by imitation, the role of multimodality, repurposing movement, and gesture design.