GESPIN – GESTURE & SPEECH IN INTERACTION – Poznań, 24-26 September 2009
Multimodal attribution: How gestures are syntactically integrated into spoken language Ellen Fricke European University Viadrina, Frankfurt/Oder science@ellenfricke.de
Abstract Can gestures take over grammatical functions in spoken language? Are there points of structural integration into vocal syntax? The first goal of my presentation is to give proof that in German spoken language co-speech gestures can be structurally integrated as constituents of nominal phrases. The second goal is to show that those syntactically integrated gestures can function as attributes to the verbal nucleus of nominal phrases. The assumption that co-speech gestures are integrated on the level of language use is widely accepted (e.g. Kendon 2004, McNeill 2002 and 2005, Müller 2008). For example, co-speech gestures are co-ordinated with the intonation as as well as semantically and pragmatically coexpressive with the verbal utterance But what about multimodal integration on the level of the language system? The idea that a certain syntactic function can be instantiated by entities of different modalities, e.g. visual or auditory, is not new and can be traced back to linguists and semioticians such as Bühler (1934), Hjelmslev (1943), and Pike (1967). Nevertheless, most descriptive studies in linguistics so far are based on vocal speech and its auditory channel alone. What is the syntactic relationship between speech and co-speech gestures in multimodal grammar? This question seems to be a blind spot in the field of gesture studies too. My claim is that co-speech gestures are able to instantiate the syntactic function of an attribute within verbal nominal phrases. On the one hand, they can be syntactically integrated by verbal deictics, for instance, son or so ein in German [engl. paraphrase: such a or like this] as evidenced in the following example, which obligatorily requires a qualitative description on the level of the language system (Fricke 2007, 2008). This description of a quality can be instantiated verbally or gesturally: sone gelb-goldenen Tafeln [‘such yellow golden plates’] (+ rectangular gesture). In this example we can observe a division of labour: the verbal adjective provides a description of a colour, whereas the co-verbal gesture gives a description of a shape. On the other hand, gestures are semantically integrated: Co-speech gestures can semantically modify the nucleus of the nominal phrase and are for that reason not only covered by syntactic but also by semantic definitions of attribution (Fricke 2008). Another finding is that gestural attributes can either be object-related to the reference object intended by the speaker or interpretant-related to the speaker's mental 'meaning-like' prototype which is a typical image of an entity in stable association with a verbal word form (Fricke 2008).
Bibliography Bühler, K. (1934/1982). Sprachtheorie. Die Darstellungsfunktion der Sprache. Stuttgart, New York: Fischer. Eisenberg, P. (1999). Grundriß der deutschen Grammatik. Bd. 2: Der Satz. Stuttgart, Weimar: Metzler. Fricke, E. (2007). Origo, Geste und Raum: Lokaldeixis im Deutschen. Berlin, New York: de Gruyter. Fricke, E. (2008). Grundlagen einer multimodalen Grammatik des Deutschen: Syntaktische Strukturen und Funktionen. Habilitation, European University Viadrina Frankfurt/Oder. (publication by de Gruyter in 2009). Hjelmslev, L. (1943/1974). Prolegomena zu einer Sprachtheorie. München: Hueber. Kendon, A. (2004). Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. McNeill, D. (1992). Hand and Mind: What Gestures Reveal about Thought. Chicago: Chicago University Press.
GESPIN – GESTURE & SPEECH IN INTERACTION – Poznań, 24-26 September 2009 McNeill, D. (2005). Gesture and Thought. Chicago: The University of Chicago Press. Müller, C. (2008). Metaphors. Dead and Alive, Sleeping and Waking. A Cognitive Approach to Metaphors in Language Use. Chicago, University of Chicago Press. Pike, K. L. (1967). Language in Relation to a Unified Theory of the Structure of Human Behavior. Den Haag, Paris: Mouton.