Conducting Actions Elicit Specific Acoustic Features in How People Vocalize: Cross-modal Correspondence between Gestures and Sounds as a Function of Available Information

Abstract

Music conductors use hand-arm gestures to shape the sound that their musicians produce. Six series of experiments are conducted in order to understand the nature of such cross-modal links between observed hand gestures and their accompanying vocal responses using several experimental manipulations to a simple cross-modal task. The task involved adults watching video clips of four different types of hand gestures (referred to as flicks, punches, floats and glides, following Laban and Lawrence, 1947), and producing the syllable /da/ repeatedly along with the observed gestures. Experiment 1 explored the specific cross-modal links between the kinematic features of the movement and acoustic features of accompanying vocal sounds. Experiment 2 explored the role of instruction and how automatic or deliberate the gesture-sound correspondence is. Experiment 3 explored the role of music background, and whether such associations stem from music experience or from everyday life experience. Experiment 4 explored whether perceiving the velocity patterns of the gestures provides sufficient basis for such coupling. Experiment 5 explored the role of auditory feedback and whether gesture-sound coupling is driven auditorily or through the vocal-motor system. And finally, experiment 6, a motor practice study, explored the role of gestural motor practice as a potential motor-based mechanism mediating/enhancing this visual-to-auditory mapping. Results showed a strong and intrinsic coupling between visual and auditory processes, especially for time and intensity related features, that does not entirely depend on a deliberate/learned cognitive strategy (experiment 2), musical background (experiment 3), featural body representations (experiment 4), or auditory feedback (experiment 5); and one that is strengthened by the absence of auditory feedback with increased attention to vocal-motor representations for sounds (experiment 5), and by motor practice of observed movements, which activates the brain regions involved in executing the movements (experiment 6).