CIENKI

GESPIN – GESTURE & SPEECH IN INTERACTION – Poznań, 24-26 September 2009

Mental space builders in speech and in co-speech gesture Alan Cienki Vrije Universiteit, Dept. of Language and Communication Faculteit der Letteren, De Boelelaan 1105 NL-1081HV Amsterdam, Netherlands a.cienki@let.vu.nl

Abstract In cognitive linguistics, the notion of ‘mental spaces’ is often employed to explain how language users set up and refer to aspects of mental models of the ongoing discourse. Previous research has focussed on verbal devices speakers use to set up mental spaces, but some research has indicated ways in which manual gesture with speech may also reflect spacebuilding. In the present study, excerpts from seven conversations between native speakers of English were coded for 100 space building verbal expressions in order to explore the extent to which co-verbal behaviors accompany them and to see what forms they take. Nearly half of the space-building verbal expressions were accompanied by gestures, and these had various possible forms (manual, body, head, gaze shift), even for a single category of space builders.

The results suggest that a complex of factors needs to be taken into account in order to more adequately research multimodal processes which may be used to set up mental spaces while speaking.

1

Introduction

The theory of mental spaces has played a foundational role in cognitive linguistics since its introduction, most notably in Fauconnier (1984). Mental spaces have been proposed as “small conceptual packets constructed as we think and talk, for puposes of local understanding and action” (Fauconnier & Turner 2002: 102), furthermore “allowing a fine-grained partitioning of our discourse and knowledge structures” (Fauconnier 1997: 11). The theory thus makes claims on the cognitive level about how language use reflects mental models of discourse (Fauconnier 1994: xxxix). The theory also provides a diagrammatic means of analyzing how speakers and hearers1 might be reasoning with mental models of discourse: the proposed mental spaces are represented in analyses via the physical spaces of drawn circles (see Figure 1). We know that when speakers gesture manually they may also use physical spaces to refer to subjects mentioned (McNeill 1992; Sweetser 2007). To what degree are the mental spaces that speakers verbally set up in discourse actually reflected by speakers via the use of gestures of some kind? This paper presents an exploratory study to indicate possible methods and directions for future research in this area.

1.1

Space builders

Mental spaces are proposed as “constructs distinct from linguistic structures but built up in any discourse according to guidelines provided by linguistic expressions” (Fauconnier 1994: 16). Any

1 The theory extends to written language use as well, but in the context of this paper we will constrain the discussion to speech and gesture.

GESPIN proceedings, vol. I

such linguistic expression which either establishes a new space or refers to one already introduced in the discourse is called a space builder. Fauconnier (1994) describes a number of important types of mental spaces and the kinds of words and phrases used to build them in English. Table 1 provides an overview and categorization of them.

Table 1: Categories of mental spaces and the verbal expressions which set them up

Mental State space2

Space Building words (examples) last week, in 1929, at that time at the factory, in the other apartment, where in his new theory, under these conditions possibly, maybe, theoretically if...then, it is possible that, otherwise Max believes ___, Mary hopes ___, Gertrude claims ___

Counterfactual space

not, never

Type of mental space Time space Space space Domain (of activity) space Possibility space Hypothetical space

Grammatical categories involved Prepositional phrases, time/space adverbials Adverbs Connectives, adverbs, certain presentational clauses Subject‐verb combinations Negatives

According to the theory, use of a space building expression sets up a mental space for reasoning that is distinct from the “here and now” of the speaker’s mental model of the discourse context. So if the speaker says, “I hope this economic crisis ends soon,” the use of the verb hope is claimed to set up a mental representation different than the one which the speaker has of the current reality, as illustrated in Figure 1.

speaker’s model (M) of

speaker’s model (M') of reality in the discourse

the mental state hoped to context

become real

e

e'

e: economic crisis exists, end uncertain

e': economic crisis imagined with foreseeable end

Figure 1: For the example sentence “I hope this economic crisis ends soon.”

1.2

Research questions

Research on sign languages (e.g., Liddell 2003) indicates extensive use of physical spaces to set up and refer to mental spaces in discourse. Perhaps this is not surprising given the spatio-visual basis of sign languages. But the study of gesture with speech has revealed that mental spaces building

2

This was not labelled as such in Fauconnier (1994) but is presented as a category of space builders there.

2

Alan Cienki: Mental space builders in speech and in co-speech gesture

can also be seen in co-verbal manual gesture (e.g., Cienki 2005, 2008a, 2008b; Parrill & Sweetser 2004; Sweetser 2007; Williams 2008).

This paper presents exploratory research to begin addressing the following questions:: To what degree does the setting up (building) of mental spaces in discourse receive expression co-verbally?

What form(s) of co-verbal behavior are involved (only manual gesture or other types of bodily gesture?)? What types of verbal mental space builders co-occur with bodily expression?

The findings can provide some indication of the degree to which the deployment of mental spaces in discourse is purely a verbal phenomenon or rather a multimodal one.

2

Data and methods

The conversations analyzed were elicited from pairs of university students who were native speakers of American English, and they were asked to discuss several questions about how they take exams at their university. Data from the same two minutes of seven conversations were used, providing 14 minutes of talk from 14 speakers in total.

Transcripts of the speech were coded for verbal space builders, using the categories described above. This analysis revealed a few new candidates for space-building (SB) words, not previously discussed in the relevant literature, such as the verb phrase let’s say or simply say to introduce a Possibility space. However the coding was done conservatively so as to only count clear instances of verbal space builders since the goal was to obtain a sample of the different types of SB words used in the talk rather than to do a complete quantified study of them.

This coding yielded 100 SB verbal expressions in total, with a range of 9 to 17 SB verbal expressions per clip (average = 14. If more than 17 SB verbal expressions were found for a clip, only the first 17 were used. The breakdown of the different types of SBs used is shown in Table 2.

Table 2: Types and quantities of SB words used Space Builder type # of instances Counterfactual 26 Hypothetical 26 Possibility 16 Time 12 Domain 8 Mental state 7 Space 5

Total 100 The gesture analysis was performed by viewing what each speaker was doing, if anything, at the time of uttering a coded SB verbal expression. Not only manual gestures were coded, but also body shifts and head movements. Gaze shift and change in facial expression were not always visible, but in the context of this exploratory study, they were also coded if observed.

3

GESPIN proceedings, vol. I

3

Results

Nearly half of the SB verbal expressions (43%) were accompanied by some type of gesture, while 57% were not. Most of the SB types occurred both with and without gestures, as shown in Table 3; the exceptions were Mental State SBs, which did not occur with gestures, and Space SBs, which were all accompanied by gestures, which raises questions for larger-scale research in the future. In the given data, all the Mental State verbal SBs were instances of the verb think (mostly the introductory phrase I think).

Table 3: Occurrence of SB types with and without gesture Space Builder type # of instances With gesture No gesture Counterfactual 26 10 16 Hypothetical 26 10 16 Possibility 16 8 8 Time 12 5 7 Domain 8 5 3 Mental State 7 0 7 Space 5 5 0

Totals 100 43 57 In terms of the types of gestures used, head gestures were used most frequently (22 times) with SB words, followed in frequency by hand gestures (19 times). Body shift (7 times) and gaze shift (7 times) thus hold a kind of second place to head and hand gestures. Only one change of facial expression was observed (conspicuous eyebrow raise), but as noted above, the video material did not allow for as complete an analysis of this facial gestures.

Note that the totals for the types of gestures used amount to more than the 43 total from Table 3. This is because with 12 of the 43 SB verbal expressions accompanied by gesture, more than one category of gesture was used. The co-occurrences are broken down in Table 4.

Table 4: Use of more than one gesture type with an SB verbal expression

with Hand with Head with Body with Gaze

Hand

5 0 0

Head 5

5 2

Body 0 5

0

Gaze 0 2 0

Total co‐ occurrences 5 12 5 2

Total instances of this gesture type 19 22 7 7

The frequent use of head gestures appears to correlate with their status as frequently accompanying other gesture types (12 out of the 22 times they were used they co-occurred with other types of gestures). This is not the case with hand gestures in this context, which were only accompanied 5 times by other gesture types (all head movements).

It is worth mention that the many head and hand gestures were of different forms. The breakdown of the 22 head gestures is as follows – 9: turn to right; 4: shake left-right; 3: move back; 2: move down; 2: move up; 1: move forward; and 1: turn to left. Manual gestures were made with the right hand (10 times), left hand (4 times), and both hands (5 times); palm-up open-hand

4

Alan Cienki: Mental space builders in speech and in co-speech gesture

gestures (5 times) as well as palm-down open-hand (4 times); with other forms covering a range of types, including finger-bunch, pointing, and flexing the stretched fingers outward.

On the whole, the type of gesture use appears to be variable across the SB types, shown in Table 5, although it is interesting to note the more frequent use of hand gestures with Space SBs and yet not with Time SBs, and the use of head gestures with Possibility, Hypothetical and Counterfactual SB words.

Table 5: Types of SBs invoked verbally and gestures used with them

Counterfactual Domain Hypothetical Possibility Space Time

Hand 2 3 5 3 5 1

4

Body 2 0 3 0 0 1

Head 4 3 5 5 1 3

Gaze 2 0 1 2 0 2

Face 0 0 1 0 0 0

Discussion and conclusions

Although previous research has indicated that gesture can play a role as a kind of SB expression, from the present study it is clear that co-speech gesture only takes on that role some of the time.

This contrasts with the extensive use of physical space for this purpose in sign languages, but is in line with the different status of manual and other kinds of embodied behavior with speech, which are dependent on the verbal code for meaning making.

The fact that most of the categories of SBs in the data were expressed sometimes with gesture and sometimes without, and that the gestures could be of various types, suggests that there are other factors which help determine the specifics of gesture use with verbal SBs. One important factor observed in the gesture coding was the placement of prosodic stress in relation to the SB word(s) in a given intonation unit, with co-occurrence with stress increasing the likelihood that the SB word would be accompanied by a gesture. Other contextual factors likely playing a role include aspects of the speakers’ positioning in the physical setting, the genre and the purpose of the talk, and the role of the SB in light of the previous discourse.

Words alone often serve as space building expressions, but nearly half of the time the expression of space building in these data is accomplished multimodally. One question remaining is whether – and if so, how – gestures alone can reflect the speaker’s setting up of mental spaces.

Anecdotal evidence suggests that this is possible, given observations of speakers’ use of just the palm-up open-hand (Müller 2004), or pointing to ‘empty’ spaces (McNeill et al. 1993), to present or indicate discourse topics. The study here provides insight into the range of gestural behaviors which may be used as speakers are setting up mental spaces while speaking, but a more complete picture of the relative roles of verbal and co-verbal behaviors as space builders awaits further study.

5

GESPIN proceedings, vol. I

Bibliography Cienki, A. 2005. Gesture and the question of literal versus non-literal reference. In: S. Coulson and B. LewandowskaTomaszczyk, eds., The literal and nonliteral in language and thought. Frankfurt am Main: Peter Lang. pp. 281-298.

Cienki, A. 2008a. Looking at analyses of mental spaces and blending/Looking at and experiencing discourse in interaction. In: T. Oakley and A. Hougaard, eds., Mental spaces in discourse and interaction. Amsterdam: John Benjamins. pp. 235-245.

Cienki, A. 2008b. Why study metaphor and gesture? In: A. Cienki and C. Müller, eds. Metaphor and gesture.

Amsterdam: John Benjamins. pp. 5-25.

Fauconnier, G. 1984. Espaces mentaux. Paris: Minuit.

Fauconnier, G. 1994. Mental spaces. Cambridge: Cambridge Univ. Press. First published in 1985 by MIT Press.

Fauconnier, G. 1997. Mappings in thought and language. Cambridge: Cambridge Univ. Press.

Fauconnier, G., Turner, M. 2002. The way we think. New York: Basic Books.

Liddell, S.K. 2003. Grammar, gesture, and meaning in American Sign Language. Cambridge: Cambridge Univ. Press.

McNeill, D. 1992. Hand and mind: What gestures reveal about thought. Chicago: Univ. of Chicago Press.

McNeill, D. Cassell, J., Levy, T. 1993. Abstract deixis. Semiotica 95: 5-19.

Müller, C. 2004. Forms and uses of the Palm Up Open Hand: A case of a gesture family? In: C. Müller and R. Posner, eds., The semantics and pragmatics of everyday gesture. Berlin: Weidler. pp. 233-256.

Parrill, F., Sweetser, E. 2004. What we mean by meaning: Conceptual integration in gesture analysis and transcription.

Gesture 4: 197-219.

Sweetser, E. 2007. Looking at space to study mental spaces: Co-speech gesture as a crucial data source in cognitive linguistics. In: M. Gonzalez-Marquez, I. Mittelberg, S. Coulson, and M. J. Spivey, eds., Methods in cognitive linguistics. Amsterdam: John Benjamins. pp. 201-224.

Williams, R. F. 2008. Gesture as a conceptual mapping tool. In: A. Cienki and C. Müller, eds., Metaphor and gesture.

Amsterdam: John Benjamins. pp. 55-92.

6