LIS

GESPIN – GESTURE & SPEECH IN INTERACTION – Poznań, 24-26 September 2009

Motion events in Polish: speech and gesture1 Magdalena Lis VU University Amsterdam De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands magdalena.lis1@gmail.com

Abstract The present paper discusses some preliminary findings regarding the representation of motion events in speech and co-speech gesture of Polish speakers. Fifteen native speakers of Polish were videotaped while narrating a cartoon, and their speech and gesture production in their descriptions of two scenes was analyzed. It was found that Polish speakers verbally expressed both path and manner component of motion event, which are both easily codable in Polish, but that in the co-occuring gestures more than half of them omitted manner. On the other hand, the speakers were likely to gesturally represent the arc-shaped trajectory of a single, directed swing motion, while dropping this trajectory from their speech, since it is difficult to encode in Polish. At the same time, in both cases the speakers tended to gesturally reproduce motion direction, which they did not express verbally, although it is easily codable in Polish. The results corroborate the claim that gesture can encode information that is absent in concurrent speech. Furthermore, as a possible account of the data a hypothesis is suggested that gestural representation may not only encode the spatial characteristics of motion that are not expressed in speech, but may be also influenced by the degree of linguistic codability of motion event components. Such an interpretation of the data is consistent with the claim that linguistic, gestural and spatial representations of events interact during speech production.

1

Introduction

Motion is an everyday phenomenon and all languages have means to describe it. However, languages differ in how they map semantic components of motion events onto linguistic forms (e.g., Talmy 1985, 1991; Slobin 1987). In this respect, a distinction can be made between ‘satelliteframed’ languages and ‘verb-framed’ languages. Satellite-framed languages, among which Polish and English, prefer to lexicalize the motion path (the course followed by a moving object) in a ‘satellite’ to the verb (verb particle or affix) or a preposition, leaving the main verb slot open for the encoding of information about manner (the particular way in which the motion occurs). In contrast, verb-framed languages, e.g., Spanish, Turkish or Japanese, tend to encode the motion path in the main verb, adding information about manner in a subordinated verb or adverbial gerund (Talmy 1991). Speakers of verb-framed languages typically need a second clause to express manner and thus tend to omit the manner component altogether in discourse. Speakers of satelliteframed languages, on the contrary, typically package both manner and path in one clause; manner

1

This paper is based on a part of the author's Master thesis „Verplaatsingsgebeurens in het Nederlands en het Pools.

Spraak, gebaren en thinking for speaking” supervised by Dr. Saskia Daalder, Department of Language and Communication, VU University Amsterdam.

GESPIN proceedings, vol. I

is thus highly codable (i.e., easily expressed) in these languages, most of their verbs of motion prespecify manner, and information about manner occurs frequently in motion descriptions (Slobin 2003). But in the meaning-form mapping in the domain of motion events, satellite-framed languages also vary among themselves. For example, Polish does not have a determinate verb to express a single, arc-shaped, Tarzan-like motion from one location to another, nor a readily accessible paraphrase for it either. While such motion is not easily codable in Polish, it is readily expressed in English with the verb swing.

Recent works have suggested that linguistic patterns may influence gestural representation ‘online’, i.e. at the moment of speaking (e.g., Brown 2007; Kita & Özyürek 2003; McNeill & Duncan 2000), although agreement on the precise nature of this influence is lacking. In a number of studies, Kita and Özyürek (e.g., Kita & Özyürek 2003; Özyürek & Kita 1999; Özyürek et al. 2005) have found that the semantic content of gesture may not only be influenced by the spatio-motoric properties of the event referred to, which do not have to be verbalized, but also by the languagespecific packaging of information into a processing unit for speech production – approximately a clause. For example, in one of their studies (Kita & Özyürek 2003) the majority of subjects gesturally reproduced the direction in which they saw the motion happening in the stimulus cartoon, although none of them mentioned the direction in speech. At the same time, the speakers of the verb-framed languages Turkish and Japanese in the study were more likely to distribute path and manner over two clauses in speech as well as over separate gestures than the speakers of the satellite-framed language English, who preferred to conflate both elements within one clause and one gesture. However, McNeill and Duncan (McNeill & Duncan 2000; McNeill 2009) have found other types of semantic links between linguistic and gestural representation of motion events – they have suggested that gesture may compensate for the lack of information about manner in a verbframed language and modulate (downplay or highlight) manner in a satellite-framed one. In their study, speakers of the verb-framed language Spanish left out manner from their speech but encoded it in their gestures, whereas English speakers expressed both manner and path in their speech, but more than half of them omitted manner in gesture (downplayed it), while the remainder gesturally highlighted manner already expressed verbally.

The aim of the present work is to give some insight into the representation of motion events in speech and co-speech gesture of Polish speakers, and, in doing so, contribute to the existing crosslinguistic literature and discussion on the semantic relationship between linguistic and gestural encoding of events. An experiment will be described and its results will be discussed.

2

Method2

Fifteen adult native speakers of Polish were videotaped while narrating a Tweety Bird and Sylvester the Cat cartoon to a listener. An analysis was made of their speech and gesture production in descriptions of two ‘target events’, namely the Rolling Event and the Swing Event.

The first event depicts the cat rolling through a street. The second shows the cat’s Tarzan-like swing on a rope from one building to another. In the stimulus cartoon the first motion is seen as happening from left to right (on the screen) and the second as going in the opposite direction.

Speakers’ verbal accounts of the target events were segmented into clauses. Only iconic hand and arm gestures co-occuring with these clauses and depicting the target events (the ‘target gestures’), were taken into account. The first analysis focused on the representation of manner and path, and the second on the arched trajectory of the motion. In both cases the encoding of motion direction (as viewed in the stimulus cartoon by the speakers) was analyzed as well.

In the analysis of the Rolling Event, the clauses were coded depending on whether they included information about the manner of the rolling motion, about the motion path or about both.

Gestures were classified according to the same principle as either Manner Only (e.g., hands rotate), Path Only (e.g., hands move horizontally without rotating) or Conflated (e.g., hands rotate as they move horizontally). In the second analysis it was for each clause describing the Swing Event

2

The procedure follows Kita & Özyürek’s (2003) methodology.

2

Magdalena Lis: Motion events in Polish: speech and gesture

specified whether or not the clause expressed the arc-shaped trajectory of the motion. The same was done for the gestures, which were classified as either Arc (drawing a downward concave trajectory) or Straight (without such a trajectory).

For all target gestures except the Manner Only gestures, which by definition do not depict the route of motion, the direction was also defined. It was either to the left, to the right or away from the speaker’s body. Since the analysis looked at the encoding of motion direction as viewed in the stimulus cartoon, only lateral gestures were of interest, for they represent the viewer’s perspective, while gestures directed away from the body give the perspective of the protagonist (Kita & Özyürek 2003).

2.1

Results

Fourteen (out of 15) speakers described the Rolling Event in their speech. All of them mentioned both manner and path in their descriptions and packaged them together within one clause (a manner verb mostly accompanied by a path-prefix and/or a preposition). Nine of those speakers also produced a target gesture. More than half (55%) of them made exclusively Path Only gestures, totally dropping the manner of motion from their gestural representations. The remaining speakers depicted manner gesturally, but only two speakers (2/9 = 22%) encoded manner and path together by means of Conflated gestures and these were in both cases accompanied by Manner Only and/or Path Only gesture, which separated the two components from each other.

Twelve (out of 15) speakers encoded the Swing Event in their speech. Most of them (8/12 = 67%) did not lexicalize the arched trajectory of the motion. Three speakers (25%) did lexicalize this feature, despite the lack of easily available relevant lexical means; all three expressed the arc by means of adverbials (e.g., ‘like a Tarzan’, ‘on a swinging rope’). Additionally, one speaker produced both type clauses, one with the arc encoded, another without it. Out of these twelve speakers four were excluded from further analysis as they failed to produce a target gesture.

Among the remaining eight speakers, the proportion of arc and non-arc clauses was comparable to the proportion found in the above-mentioned speech analysis (five speakers did not encode the arc, two did, one speaker produced one clause with information about the arc and a second one without it, each accompanied by a separate gesture). The gesture repertoire of the vast majority of these speakers (7/8 = 87%) consisted exclusively of Arc gestures.

Not taking Manner Only gestures into account, speakers produced 23 gesture tokens in total.

One gesture was directed away from the body. The remainder were lateral gesture tokens; all of them (22/22 = 100%) reflected the direction of the motion as viewed in the stimulus by the speakers (to the right for the Rolling Event, to the left for the Swing Event).

3

Discussion

In line with Talmy’s (1985, 1991) typology Polish speakers followed the satellite-framed pattern.

They expressed path in adverbial adjuncts and manner in the main verb root and packaged the two components of motion events within one clause. Their gestural representation tended to consist of only one of those elements; usually the manner was left out. Furthermore, Polish speakers were likely to omit information about the arc-shaped trajectory of the swing motion in their speech.

Their gestures, however, often represented this trajectory. In the lateral gestures speakers regularly reproduced the direction in which they had seen the motion happening in the stimulus, but no information about this direction appeared in their speech.

The results are in line with the view that gesture can encode spatial properties of a referent that are not expressed in concurrent speech, as held by Kita and Özyürek (2003). However, their claim that the packaging of information in gesture parallels the packaging of information into clauses seems not to account for the Polish data presented. While Polish speakers conflated manner and path within one clause, their gestures were likely to reflect only one of these elements. Moreover, whereas speakers’ gestural representations mostly included the arched trajectory of the swing motion, the accompanying clauses tended not to do so.

Our results rather suggest that the encoding of information in gesture may complement and modulate the representation of information in speech and that a hypothesized criterion may be the

3

GESPIN proceedings, vol. I

degree of linguistic codability of that information. It may be the case that gesture gives information about a significant component of motion events that is not easily codable in speech; as illustrated by the Arc gestures of Polish speakers. On the other hand gesture may modulate a ‘too highly codable’ feature; as demonstrated by the gestures accompanying Manner-Path Conflated clauses in the present study. At the same time, gesture may by its own nature give information about a motion event feature that, although easily codable in the language, is discursively too insignificant to be mentioned in speech; as the discussed gestural reproduction of motion direction shows.

Kita and Özyürek (2003: 19) explain that the direction of motion as viewed in the stimulus is not a feature of a motion event itself and is of no importance to the plot development, but gestures depicting translocational motion by definition have to specify direction; they regularly reproduce the direction of the referential motion. As concerns gestural modulation, McNeill and Duncan (2000) point out that the vast majority of verbs in satellite-framed languages specify manner linguistically, even when that feature is not part of the speaker’s core idea. They further suggest that in such situations gestures downplay manner – 57% of English speakers in their study omitted manner in gesture despite its presence in the concurrent speech, whereas when manner is part of speaker’s focus gestures tend to highlight it – the remaining English speakers gesturally encoded manner already present in their speech (McNeill 2009). McNeill and Duncan have also observed gestural complementation of information which is absent in speech. In their analysis, 80% of speakers of Spanish – a language where manner is not easily codable and tends to be omitted in discourse – gestured about manner when there was no manner component in the accompanying verbal representation. McNeill (2009: 528) concludes that “[g]estural manner tends to expand the encoding resources of Spanish and to modulate them in English, following the packaging of manner information in these languages – unavoidable manner in English, all-too-avoidable in Spanish.”

The suggested explanation of the Polish data is largely based on the above-mentioned claim by McNeill and Duncan; it also incorporates insights from Kita and Özyürek’s (2003) study.

Nevertheless, the comparison with their findings and conclusions should be taken with caution.

Firstly, general coding conventions are lacking (e.g., regarding which linguistic elements and which gestures to count as depicting manner and which as path), hampering comparison across studies and across languages. Moreover, the complementing in McNeill and Duncan’s (2000) research concerns information about manner, while in the presented distinction between arc and non-arc clauses, manner could still be found in both types clauses. Furthermore, in their study different directions of modulation correlate with different discursive significance of manner component and with different speech-gesture synchronization patterns, the investigation of which extends the scope of this paper.

The small scope of the present study does not allow for generalizations and firm conclusions; nor can it be ruled out on its basis that other factors than the degree of linguistic codability are responsible for the gestural patterns found in the Polish data discussed. To shed more light on the issue, further investigation is needed. It may be hoped that this study can at least be a useful basis for further research.

4

Magdalena Lis: Motion events in Polish: speech and gesture

Bibliography Brown, A. 2007. Crosslinguistic influence in first and second languages: Convergence in speech and gesture. MPI Series in Psycholinguistics 47. Nijmegen: Max Planck Institute for Psycholinguistics.

Kita, S., Özyürek, A. 2003. What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 1632.

McNeill, D., Duncan, S. 2000. Growth points in thinking-for-speaking. In D. McNeill (ed.). Language and gesture (pp.

141-161). Cambridge: Cambridge University Press.

McNeill, D. 2009. Imagery for speaking. In J. Guo, E. Lieven, S. Ervin-Tripp, N. Budwig, S. Özçaliskan & K. Nakamura (eds.). Crosslinguistic approaches to the psychology of language: Research in the tradition of Dan Isaac Slobin (pp.

513-530). Mahwah: Erlbaum. Özyürek, A., Kita, S. 1999. Expressing manner and path in English and Turkish: Differences in speech, gesture, and conceptualization. In M. Hahn & S. Stonness (eds.). Proceedings of the 21st annual conference of the Cognitive Science Society, 507-512. Özyürek, A., Kita, S. Allen, S., Furman, R., Brown, A. 2005. How does linguistic framing of events influence co-speech gestures: Insights from crosslinguistic variations and similarities. Gesture, 5, 216-241.

Slobin, D. I. 1987. Thinking for speaking. Proceedings of the 13th Annual Meeting of the Berkeley Linguistics Society, 435-445.

Slobin, D. I. 2003. Language and thought online: Cognitive consequences of linguistic relativity. In D. Gentner & S. Goldin-Meadow (eds.). Language in mind: Advances in the study of language and thought (pp. 157-191).

Cambridge: MIT Press.

Talmy, L. 1985. Lexicalization patterns: Semantic structure in lexical forms. In T. Shopen (ed.). Language typology and syntactic description, vol. III (pp. 57-149). Cambridge: Cambridge University Press.

Talmy, L. 1991. Path to realization: A typology of event conflation. Proceedings of the 17th Annual Meeting of the Berkeley Linguistics Society, 480-519.

5