The Dynamic Multimodality of Talking in Context

 

Rick Dale

 

University of California, Merced

 

Traditional linguistics and psycholinguistics study various aspects of language by isolating levels of analysis: sounds, words, sentences. In the past couple of decades, it has become an important theoretical agenda to find "interfaces" between these levels of analysis. But any bout of language involves all of these behavioral levels simultaneously, dynamically unfolding while we control very high levels of analysis such as topics or goals of conversation. I will describe an approach we are taking to measure and characterize the manner in which humans dynamically couple these behavioral characteristics simultaneously during spontaneous language usage. By using a multimodal "observer's library," we can track the face, eyes, body, and speech of spontaneous language in various situations. Quantification of some of these time series together leads to synthesis of language in complex, natural circumstances. Results emphasize the self-organization of language use both within and between individuals. I discuss theoretical next steps for understanding this dynamic multimodality.