CH25 - The Encyclopedia of Human-Computer Interaction

1757

Chapter

25

25. Semiotics and Human-Computer Interaction

by Clarisse Sieckenius de Souza.

T

his chapter covers why and how Semiotics can help advance some of the major goals of Human-Computer Interaction (HCI) and be useful when de-

signing interactive products. It begins with brief definitions and explanations of a few central concepts in Semiotics. This is followed by a discussion of harder challenges involved in bringing Semiotics into the domain of HCI research and the consequences of viewing computers as media. Following Semiotic Engineering concepts, which we have been developing and using for two decades now, we then revisit computer-mediated communication in view of 21st century literacy issues. First, we show that basic computing skills exhibited by contemporary users are in fact semiotic engineering abilities of the same sort as required from professional designers. Then we show how these skills can leverage an individual’s participation in a variety of social processes. In conclusion, the chapter

1758

Encyclopedia of Human-Computer Interaction

presents our personal answer to the question that most readers certainly have in mind: ‘So, what’s in it for me?’

25.1 Introduction Semiotics and HCI have more in common than is usually acknowledged in either side of the cultural divide that has been keeping them apart. Researchers and professional practitioners working with these disciplines have different interests and perspectives when selecting, a0pplying and building knowledge. As a result, mutual understanding has not only been rare, but usually perceived as “more cost than benefit”.

This chapter is an essay on why and how Semiotics can help advance some of the major goals in HCI. It begins with a definition of Semiotics and a brief explanation of a few central concepts that will be used throughout the chapter.

Next, it discusses some deeper challenges of using Semiotics in the context of HCI research. Then it explores the notion of computers as media, the hallmark of all semiotic approaches to HCI proposed to date. It highlights the interdisciplinary work of pioneers like Mihai Nadin and Peter Bøgh Andersen and ends with a description of Semiotic Engineering, a comprehensive semiotic theory of HCI which we have been developing and using at SERG, the Semiotic Engineering Research Group in Rio de Janeiro, since 1990. In subsequent sections, we take a closer look at computer-mediated communication in view of contemporary opinions that having basic programming skills is as important for citizens of the 21st century as reading, writing and counting have been in the 20th century and before.

Some examples of basic computing literacy skills exhibited by contemporary users are framed as semiotic engineering abilities of the same sort as required from professional HCI designers. This helps us to show why computers are indeed the most pervasive media used by contemporary societies, and it points at computing literacy issues with which HCI is not only necessarily involved, but also in which

25. Semiotics

1759

it has clear vested interests. The chapter then provides our answer to the provocative question implied by its title, proposed by the editors of the Encyclopedia of HCI: “So, what’s in it for me?” The answer, I hope, will attract more attention to this fascinating discipline in times when computers so obviously coalesce and transform society’s means of communication and participation.

25.2 Some preliminary definitions Semiotics is the study of signs. Although strictly correct, this definition is not helpful for those who do not know what signs are and how they can be studied.

So, let us begin with additional definitions. The selected material will also give the reader a flavor of the cultural divide between Semiotics and HCI as expressed by terminology and conceptual framing.

In the Encyclopedia of Semiotics (Bouissac 1998), the entry for “sign” explores numerous variations in the way philosophers and semioticians have defined the central object of interest in this discipline over the centuries. Here are two of them: a.

In the tradition inaugurated by Ferdinand de Saussure: “[...] the sign can be understood as a correlation of differences.” (Bouissac 1998: p. 573)

b.

In the tradition inaugurated by Charles Sanders Peirce: “[...] the defining characteristic of signs is their capacity to determine additional signs [in the mind].” (Bouissac 1998: p. 574)

Saussure and Peirce are the founding fathers of Semiotics. Although they were contemporaries (Saussure died in 1913 and Peirce in 1914), they lived in different continents (Europe and North America, respectively) and followed completely independent paths. Saussure was a linguist, interested in a formal characterization of natural languages. Peirce was a logician, interested in meaning and knowledge discovery processes. The definitions above are both very true to their respective

1760

Encyclopedia of Human-Computer Interaction

theories, but they are also mysterious for non-semioticians. I picked them up because they also illustrate rhetorical differences between disciplines, which we have to understand in order to reap the benefits of interdisciplinary research.

25.2.1 Saussure

Figure 25.1: Oppositions in form that do not correspond to oppositions in meaning.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

Keeping in mind that Saussure (Saussure 1972) was focused almost exclusively on linguistic signs, the correlates with which he defined the sign were: an acoustic image (the ‘signifiant’, or signifier); and a concept (the ‘signifié’, or signified).

The former is a physical entity, whereas the latter is an abstract one. Differences determine what constitutes a sign at various levels and dimensions. Just as a brief illustration, take the word ‘privacy’ in English. Its acoustic image in spoken language is not unique: some say |’prɪvəsi|, while others say |’praɪvɪsi|. Although in English the long vowel |aɪ| is different from the short vowel |ɪ| (e. g. |baɪt| and |bɪt| are not the same word), this difference is neutralized in words like ‘privacy’: The concept, no matter the pronunciation, is the same. So, what is a sign?

In phonology, |aɪ| and |ɪ| are signs because they have significant differences. One is associated with the concept of ‘long vowel’, and the other with ‘short vowel’.

However, in syntax or semantics |’prɪvɪsi| and |’praɪvɪsi| are not distinct signs.

Difference, as the reader can infer, is a very powerful principle in meaning making and sense making. In natural languages, differences and oppositions are systemic. That is, certain opposing combinations are recurrent and meaningful in the language, whereas others aren’t. Therefore, a language can be described as an inventory of signs and sign combinations (or structures), for which the correlation between the signifiant and the signifié can be formally established by sys-

25. Semiotics

1761

temic differences. Here is a simple example of how this principle can be applied in the context of HCI.

Figure 25.2: Well-formed sign structures in HCI. Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

Figure 25.3: Ill-formed sign structures in HCI. Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

1762

Encyclopedia of Human-Computer Interaction

In Figure 25.1, we see different renditions of a familiar window control element.

The signifiants constitute different signs in the equivalent of HCI phonology.

Nevertheless, in HCI syntax and semantics, the whole collection corresponds to a single lexical item (a single sign for ‘close window’). In Figure 25.2 and Figure 25.3, we see the parallel with Saussure’s patterns of structures. In one case, the ‘close window’ sign is combined with other signs that are systemically used in conjunction with it, whereas in the other case it is combined with signs that should never co-occur with it. Therefore, in this interface language, only one of the two structures is well-formed.

Note that signifiant/signifié correlations in interface languages are much weaker than in natural languages. Semiotic correlations are programmed into software. Compared to natural language, this can be a blessing and a curse. The blessing is that in interface languages, unlike in natural languages, the conceptual correlate of signs can (and must) be fully specified and instantiated by a mechanical process. Thus, we always have access to the origin of meaning. The curse is that mechanical instantiation of the signifiés as correlates of interface signifiants is arbitrarily programmed by human minds. That is, nothing can prevent a mischievous programmer from specifying that ‘|x|’ causes a window to freeze, rather than close. This is not the case in natural languages: neither you nor I can decide and establish that from now on ‘privacy’ means something else in English. The correlation between parts of signs in formal grammatical descriptions of natural languages is not established by individuals.

This example gives us a chance to mention another fundamental dichotomy in Saussurean theory, that between langue (language) and parole (speech). This significant opposition has been posed to account for the fact that individual variations in language use do not affect the system. It is easy to collect numerous examples of individual linguistic behavior contradicting the general rules and conventions of a given language system (e. g. we make occasional grammatical mistakes

25. Semiotics

1763

as we speak, which may be due to heavy cognitive loads). However, individual and sporadic variations do not change or affect the language system. It is only if and when such variations make their way into a formal abstract and general sign system, extending over large spans of space and time, that they actually cause a change in langue. Other variations, within smaller spatial and temporal coverage, correspond to changes in parole.

Saussure’s interest was in a theory of langue, his formal theory of signs.

This refers us to a silent Cartesian tradition running under the surface of nonCartesian concepts, such as the determining forces of social structures and conventions. The Cartesian gene in the theory is the assumption that linguists can describe langue and that individual minds can have access to supra-individual concepts and formulate them in abstract, free of psychological, social, spatial and temporal contingencies. In other words, a linguist’s description of langue objects should be warranted by this individual’s ability to capture abstract ‘signifiés’ that exist independently of his or her mind and to preserve them from contamination of individual interpretation when building formal theories of language.

Part of Descartes’ much more complex philosophy, not discussed here, was that individual minds could have access to ultimate meanings by exercising methodical doubt and analysis (Descartes 2004). Eventually, Descartes believed, the mind would reach an unquestionable state, some primary cognition that constitutes the true meaning of the object(s) under consideration. In the remainder of this chapter, whenever we use the word Cartesian we are referring to this specific aspect of Descartes’ theory, an important one for discussing alternative semiotic theories and their application to HCI.

25.2.2 Peirce Peirce (Peirce 1992), as already mentioned, was interested in logic and the origin of meaning. His theory can actually be framed as a long and deeply elaborate re-

1764

Encyclopedia of Human-Computer Interaction

buttal to the Cartesian canons we just mentioned (Santaella 2004). He defined signs as a three-part structure consisting of a representation (the representamen), an object and a mediating element (the interpretant) that binds the object to its representation in somebody’s mind. In other words, unless there is a mental mediation, nothing is a representation of anything. This will probably remind the reader of the proverbial saying that “meaning is in the mind of the beholder”. But, as the next paragraphs will show, the theory goes far beyond intuitive interpretations of the old adage.

The definitional feature of signs mentioned in (b) on page 2 — that they can generate other signs — refers to the fact that, according to Peirce, the mediating interpretant, just like the representamen, stands for the object that it binds to a particular representation. In other words, the interpretant is a second-order representation of the object itself. The sign has, in itself, a generative recursive seed that produces other signs, ad infinitum. For example, let us go back to the word ‘privacy’. This is a sign where |’prɪvɪsi| is the representamen and something outthere-in-the-world is its object because in some minds (like yours and mine) there is an interpretant that binds them together. Is this interpretant unique? Is it the same for everyone? Is it the same for you or me throughout our entire lives? The answer is ‘no’. From the very moment that we inspect this interpretant (i. e. that we take it to mean the object out-there-in-the-world which we call |’praɪvɪsi|), it becomes a second-order representation of its very object. A newly generated interpretant (e. g. an explanation or further elaboration referring to the object) is instantiated, which generates yet another sign instantiation as soon as our mind engages in further interpretation.

In order to illustrate how these concepts can be used in a practical HCI context description, let us go back to the ‘|x|’ window control (see Figure 25.4). In a Peircean account of this sign, ‘|x|’ is the representamen of the whole sign, whereas the specified computational procedure that causes a well-determined subse-

25. Semiotics

1765

quent state of the system (in which the ‘Street video’ window is no longer visible) is its object. All of the interesting things have to do with the interpretant. What mediating sign (which will generate other signs) is binding representation and object together? To answer the question, we would have to inspect minds, like the programmers’ and the users’, for example.

An exhaustive account for all interpretants — all possible reasons and related facts that can cause ‘|x|’ to stand for whatever it is that makes the window disappear when we click on it — in programmers’ and users’ minds, over large spans of time and space, is beyond the ability of individual minds, caught in the contingencies of momentary interpretation.

We can already see how Peirce’s theory differs in focus and essence from Saussure’s, and also why, in the context of this chapter, a semiotic theory’s positioning relative to certain aspects of Cartesian tradition can extensively determine its fate in HCI. For instance, the tension between a Peircean perspective on meaning and the algorithmic nature of interface sign interpretation and generation processes in computer systems cannot be missed. Neither can the apparently smooth compatibility between Saussure’s theory and the basic tenets of formal language and automata theory in Computer Science (Hopcroft and Ullman 1979) be ignored.

Nevertheless, there is more promise in Peircean theory for HCI than meets the eye. Let us briefly introduce a couple of additional concepts and definitions.

1766

Encyclopedia of Human-Computer Interaction

Figure 25.4: How simple interface signs can generate unlimited semiosis.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

The first is Peirce’s most widely known and used definition of sign: “A sign [...] is something which stands to somebody for something in some respect or capacity. It [...] creates in the mind of that person an equivalent sign, or perhaps a more developed sign.” (Buchler 1955: p. 99) Note that the mediation of the interpretant is done ‘in some respect or capacity’ (i. e. for some potentially partial and arbitrary reason). Thus, although most users will tend to generate an interpretation for ‘|x|’ that is at least superficially appropriate for the immediate context

25. Semiotics

1767

of interaction sketched in Figure 25.4, subsequently generated signs are likely to be very different. For example, thanks to some unfortunate previous experience, one user may accept that ‘|x|’ means ‘close the window’, but develop further interpretations according to which the presence of ‘|x|’ in that particular window is actually the result of negligent programming. She may believe that if she clicks on it and closes that window, the browser will freeze. In her mind, to prevent the unfortunate situation, the ‘|x|’ control should not be active, especially because the ‘Close & Return’ button is there to control the closing operation safely. Some other user, however, thanks to luckier interactions, may develop the interpretation that the ‘|x|’ control and the ‘Close & Return’ button mean exactly the same thing.

The difference, he may think, is a practical one. The generic ‘close a window’ can be communicated with different signs: to click on ‘|x|’ or press some specific key combination. The latter can be faster than the point-and-click alternative. Thus, whereas advanced users may interpret the presence of ‘|x|’ as meaning that the use of keyboard input to control the window on Figure 25.4 is allowed, novice users may miss this signification of ‘|x|’ altogether and interpret that to close the window they must click on the ‘Close & Return’ button. The next steps in each user’s theories of what ‘|x|’ means in different situations are virtually impossible to predict since they depend substantially on the users’ experience with software, their level of technical awareness, and virtually all other meanings that populate their minds.

The second concept of Peircean theory that this example helps to illustrate is the process of abductive reasoning (also known as hypothetical reasoning or abduction), which underlies — according to the theory — all meaning-making activities. Very simply put, this kind of reasoning consists of generating hypotheses to explain (i. e. to interpret) significant elements of reality around us. Once a hypothesis is generated, it is tested against ready-to-hand evidence. If the evidence contradicts the hypothesis, a new one is generated and tested. If not, the hypothesis is accepted as a general principle capable of binding not only the tested instances

1768

Encyclopedia of Human-Computer Interaction

of representations to the tested objects, but also — and more importantly — future instances and yet-to-be-met objects as well. The principle is true until further evidence is found that calls for revision of beliefs. This indefinitely long process of sign generation is named semiosis. It can, for example, explain Carroll and Rosson’s observations reported in an influential early work in HCI, The paradox of the active user (Carroll and Rosson 1987). The authors remark that users “hastily assemble ad hoc theories” of how the system works and refuse to learn more even though they hold important misconceptions about the interface language semantics, which can lead to inefficient or ineffective interaction. A Peircean account of this phenomenon tells us that, in the process of abductive reasoning, users come upon mediating signs that do not require or motivate further abductive effort. They are content with their inferencing and fixate a generative belief (i. e. an interpretant that will be used in other sense-making situations). Even if from a programmer’s point of view the users’ reasoning and interpretations are totally wrong, for as long as they ‘work’, the users believe that they are right. The meaning of ‘work’, and not so much the meaning of interface elements themselves, is the key to understanding interaction patterns and choices in this case.

The third and last additional concept that we will discuss with respect to Peircean Semiotics is that of a/the true meaning of signs. If human minds can engage in widely (and wildly) different interpretive directions, how is it that two people can understand each other, or at least truly believe that they do so? And, much more importantly, is there such a thing as a/the true meaning of signs?

In a Peircean perspective, meaning emerges in a constantly changing chain of “equivalent [...] or perhaps more developed” signs. Subsequent interpretants are connected, corrected, adjusted, expanded, stabilized and destabilized by the insistence of the real. Without going into the very complex details of Peirce’s theory of truth and true meanings, it is worth highlighting a few points about his particular view of reality and reality’s role in sign-making processes:

25. Semiotics

I.

1769

Sign making (which for the purposes of this chapter is the same as sense making) is a natural disposition of the human species, just like flying is a natural disposition of birds. Humans cannot help but assign meaning to whatever they sense in the world around them. Therefore, reality is always mediated by signs.

II. Because it is an abductive inferencing process, human instinctive sign making is fundamentally prone to error. We constantly make mistakes while interpreting reality. However, our innate interpretive apparatus is such that it can correct interpretations in the presence of new evidence.

This mode of mental operation constitutes the first and more essential step in any sort of human mental activity, from the mundane daily inferences about the freshness of products on market stalls to the scholarly debate about the validity of scientific theories.

III. The principle that makes human knowledge and understanding gravitate towards ‘truth’ is the insistence of reality, the inexorable massive set of evidence provided by the world around us. This is also an indication that, in this theory, truth is not the result of an individual’s introspective activity (as Descartes would wish), but a collective ongoing semiotic process, in which abductive corrections made and expressed by other minds penetrate culture through social processes that eventually affect individual sense-making. Hence, as a social, cultural and historical process, human interpretation gravitates toward truth.

Back to HCI, these three points allow for a reinterpretation of heated controversies. Take for example Jared Spool’s criticism of User-Centered Design. In a CHI Panel in 2005 (Spool and Schaffer 2005), among other things, Spool claimed that “beyond small teams with ‘simple’ issues, formalized UCD doesn’t seem to work” and that “UCD pretends to act like an engineering discipline (formalized methods

1770

Encyclopedia of Human-Computer Interaction

that have repeatable results independent of practitioner), but actually behaves as if it’s a craft (dependent completely on skills and talents of practitioners with no repeatable results).” (p. 1174) Spool’s position is a good example of how the “insistence of reality” is related to, but not the same as, “repeatable results”. A Peircean account of meaning tells us that predictions of how users will interpret interface signs must be taken very cautiously. When making sense about the natural and the cultural world, a user’s abductive reasoning paths will eventually be corrected and refined by the insistence of a massive volume of facts that penetrate the mind as signs contingently related to each other “in some respect or capacity”. Contingency — the craft that lies at the opposite of “repeatable results” expected by Spool — is actually the essence of human sense making.

Computer encoded interface signs, however, are different. They are, and must be, uniquely defined and engineered into software artifacts that continually “repeat” exactly the same mechanical interpretation and generation for each and every representation in the program. These representations are produced by human minds (the software designers’ and developers’) and meant to affect other human minds (the users’). The inexorable repetition of these representations in computer systems’ interfaces is ontologically very different from what Peirce refers to as the insistence of the real. Actually, computer interface signs are no more than the expression of a particular moment in someone’s (or some group of people’s) abductive reasoning path, which is as prone to interpretive error as any user’s abductions while trying to make sense of the interface.

We thus see that Spool’s vigorous criticism of UCD echoes some of the Cartesian notions we discussed above. He, just like UCD adopters, is looking for repeatable semiosis “independent of practitioner”, meaning independent of mind, which should nevertheless be mentally achieved by an individual’s use of rigorous methods of inspection and analysis. The interest of a semiotic perspective on

25. Semiotics

1771

this debate is to show that there are theoretical choices beyond Cartesianism, and that depending on whether the focus is on human semiosis or computer semiosis, some choices are better than others. For illustration, we comment on two alternative paths.

Louis Hjelmslev (Hjelmslev 1961) followed Saussure’s ideas, but gave them a more axiomatic interpretation. In his view, signs could be studied as the correlation between any two autonomous systems. Neither one had to (although they might) be anchored in psychological reality. Charles Morris (Morris 1971), in turn, followed Peirce’s ideas, but gave them a radically behaviorist interpretation. In his view, signs should be defined in terms of stimulus and response. Influenced by Peirce’s notion that habit was a powerful mediating force in sign-making processes, Morris developed a semiotic model that explained all sign processes in terms of reactions caused by the presence of signs under certain conditions. Hjelmslev was the theorist chosen by Peter Bøgh Andersen (Andersen 1990), when writing his Theory of Computer Semiotics, and Morris the one chosen by Heinz Zemanek (Zemanek 1966), when writing his pioneering Semiotics and Programming Languages.

25.2.3 Culture, signification and communication Before we proceed to the next sections, let us revisit the saying that “meaning is in the mind of the beholder” for final remarks about conditions for human mutual understanding. We will borrow Umberto Eco’s Theory of Semiotics (Eco 1976), which brings together Peircean and Saussurean elements in interesting ways. On the one hand, Eco adopts the notion that signs trigger an indefinitely long chain of other signs in the mind (he speaks of unlimited semiosis). On the other, he defines signs as content-expression correlates. The apparently diverging perspectives are reconciled with the introduction of culture in two fundamental semiotic processes: signification and communication.

1772

Encyclopedia of Human-Computer Interaction

Signification is a correlation between content and expression. Signification systems are collections of “discrete units [...] or vast portions of discourse, provided that the correlation has been previously posited by a social convention”. Communication is a process in which “the possibilities provided by a signification system are exploited in order to physically produce expressions for many practical purposes.” (Eco 1976: p. 4) Notice that Eco does not say that in communication one repeats the conventions encoded in signification systems. He says that communicators exploit the possibilities provided by a signification system. In other words, communication may include sign productions that deviate from the system, as well as others that don’t. In fact, Eco’s theory of sign production defines communication as a process that manipulates signs, “considering, or disregarding, the existing [socially conventionalized] codes” (Eco 1976: p. 152). The primacy of social conventions in communication, even when stepping out of existing signification systems, places culture at the center of Eco’s Semiotics, which he actually characterizes as the logic of culture. Semiotics, in his view, is thus a comprehensive study of culturally-determined codes and sign productions at all levels of human experience.

25.3 Semiotics and HCI: Two Disciplines, Two Cultures The work of Hirschheim, Klein and Lyytinen (Hirschheim et al 1995) on the conceptual and philosophical foundations of Information Systems Development (ISD) can further our understanding of why Semiotics and HCI have been keeping apart from each other. Following a still widely-accepted view that information systems are technical implementations of social systems, the authors trace the differences among ISD approaches, concepts, models, methods and even tools back to their proponents’ interpretation of reality. Building on Burrell and Mor-

1773

25. Semiotics

gan’s work in organizational analysis (Burrell and Morgan 1979), they list four paradigms in ISD research and professional practice. The paradigms are simplified world-view models produced by combining the end points of two axes: objectivity-subjectivity and order-conflict (see Table 25.1).

Paradigm

Functionalism

Choice be-

Choice

Assumptions about

tween Ob-

between

Reality

jectivity and

Order and

Subjectivity

Conflict

Objectivity

Order

There is an objective order in social reality, which can be known and described independently of subjective interpretation.

Social

Subjectivity

Order

Relativism

Reality is a social construction, which is the result of interpretations determined by continuous cultural changes.

Radical Structuralism

Objectivity

Conflict

Reality is determined by objective super-structures that are in constant power conflicts.

1774

Neohumanism

Encyclopedia of Human-Computer Interaction

Subjectivity

Conflict

Reality is continually transformed by multiple objective and subjective epistemologies that coexist and contribute to historical evolution.

Table 25.1: Hirschheim and colleagues’ paradigms at the base of information systems development.

Different assumptions about reality, shown in the last column of Table 25.1, can be aligned with distinctions between Saussurean and Peircean perspectives on meaning. This is not to say, however, that the same set of paradigms can be generated by semiotic theories alone, and much less that Saussure or Peirce (would) have embraced the principles of functionalism, social relativism, radical structuralism or neo-humanism. We must only observe the fact that paradigms that preclude subjectivity have higher compatibility with definitions of meaning that do not speak about mental mediation than those that include it, and vice versa.

Likewise, speaking only of the paradigms where order is important, we should observe that the defining characteristics of functionalism in Table 25.1 are more compatible with Cartesian canons than is the case with social-constructivism.

Hirschheim and co-authors note that most of the work in ISD published until 1995 tacitly falls into the functionalist perspective, according to which subjective interpretation doesn’t play a role when systems requirements are elicited and data models are built, for example. Testing hypotheses against empirical data can, in this view, prevent subjective intuitions, beliefs and aspirations from contaminating scientific knowledge of an objective reality. Consequently, this paradigm has long supported a feedback loop in which information and communication technologies (ICT) are viewed and produced as the impersonal (i. e. non-subjective)

25. Semiotics

1775

result of applying empirically-tested facts and principles to solve problems. Computer scientists have thus been increasingly led into empirical research practices looking for more facts and principles, but speaking very little of their interpretations and valued perspectives regarding what these facts and principles mean, why, and under what kinds of conditions.

As a multi-disciplinary field, HCI encompasses almost as much diversity in terms of scientific traditions as the number of disciplines that can be listed among its contributors (e. g. Psychology, Anthropology, Sociology, Design, Ergonomics and Computer Science, to name but only a few). However, most of its success and prestige is associated to ever more sophisticated and pervasive kinds of interactive products coming from the ICT industry. In this context, predictive knowledge, based on sound empirical experimentation, is in high demand. The industry is avidly looking for tested facts and principles to guide its engineering processes and found the establishment of norms and standards. Thus, the pressure of industrial demands on HCI research has been reinforcing functional perspectives on science and restating the belief that scientists and technologists do their work impersonally, backed by statistical inferencing methods applied to objectively collected empirical data.

Semiotics has been successfully applied in marketing and advertising (Umiker-Sebeok 1988; Floch 2001), for example, which suggests that HCI and Semiotics could work together to respond to industrial demands. Nonetheless, the power of opposing scientific traditions and practices has been keeping them on separate tracks. Semioticians of Peircean extraction, for instance, are interested in the rich variety of meanings that emerge in sign production and sign interpretation contexts. They may use extensive interpretive and hermeneutic methods to generate in-depth understanding of situated meaning-related processes, before they engage in statistical inferencing methods leading to general signification facts and principles. In addition, they must not dismiss the fact that in order to establish the

1776

Encyclopedia of Human-Computer Interaction

very object of their investigation (a sign or sign process) they need the mediation of their own minds. That is, in order to take anything as a sign (or evidence) of other signs or sign processes of interest, investigators must carry out interpretation. The process and product of their investigation will depend totally on other signs: their inferences, their validation criteria, their conclusions, and last but not least on the way they signify their knowledge discovery process and communicate it to the scientific community. The main criteria for scientific validity, which must be made apparent in this specific situation, is that newly discovered signs can: be framed into coherent inferential discourse that consistently relates to previously existing knowledge; correct, expand or innovate aspects of previous scientific findings; be exposed to the evaluation of other researchers in the scientific community; and produce, as a result of such exposure, the advancement of collective scientific knowledge.

This view contrasts in important ways with the one that credits scientific validity only to research results obtained by an empirical testing of hypotheses and subsequent statistical inferencing. Firstly, although a semiotic view acknowledges the scientific validity of knowledge produced with these methods, it requires that the choice of hypotheses be accounted for as a signification process. Why have the hypotheses been chosen? Why are they significant per se, or more significant than other possible hypotheses? Secondly, and by consequence, it brings into the territory of legitimate scientific activity the very elaboration of candidate theories and hypotheses that will be tested in the course of empirical knowledge discovery processes. By means of rigorous interpretive methods, Semiotics is ready to inspect and validate the processes by which scientists assign meaning to (i. e. “interpret�) their hypotheses, their testing and their results, against the backdrop of theories that they propose to be building.

These contrastive characteristics can explain why semiotic knowledge is usually viewed by the HCI community as speculative and subjective, an impractical

25. Semiotics

1777

philosophic discussion that cannot usefully inform the design and development of ICT products. By the same token, they can also explain why researchers with conscious or unconscious semiotic awareness tend to be skeptical of HCI theories built upon hypotheses whose selection is not carefully and consistently accounted for. Moreover, motivated by the huge diversity of human meaning-assigning strategies, these researchers may be far more attracted to the possible variations of users’ interpretive behavior in view of even small contextual changes than to regularities found under highly controlled tests in laboratory.

Given the above-mentioned differences in purpose, practice, beliefs and values, the most difficult obstacle for bringing together Semiotics and HCI is probably not the fact that Semiotics requires extensive learning of unfamiliar concepts and methods. This has always been the case in the multidisciplinary research practices that characterize HCI. The real challenge for the two disciplines seems to be how to combine different epistemologies and rise above scientific validity disputes. Only then can Semiotics seed progress in HCI. The rest of this chapter is my personal narrative of a successful case, the theoretical construction of Semiotic Engineering (de Souza 2005; de Souza and Leitão 2009). I hope many others are on their way to correct and refine our thinking, partaking in the ongoing scientific semiosis.

25.4 Computers as Media Today, viewing computers as media is like viewing cars as vehicles. Of course, computers enable, support and enrich individual communication, group communication, and mass communication. We can even put it the other way around: All contemporary media involve computation. But, how obvious was this connection when HCI emerged as a discipline? And, how obvious right now are the consequences of this view for current and future computation?

1778

Encyclopedia of Human-Computer Interaction

Viewing computers as media is the hallmark of all semiotic approaches to computing and human-computer interaction. In 1988, Kammersgaard proposed that human-computer interaction could be characterized in substantially different ways (Kammersgaard 1988). One of them was called the media perspective, which involved not only the fact that computers could be used by people to communicate with each other, but also the fact that computer application programmers could communicate with users through systems interfaces. As the author acknowledges, the latter was influenced by illustrious predecessors:

“I will not go into further detail about this last type of communication, except to mention that Oberquelle, Kupka & Maass (1983) talk about delegation of communicating behaviour from the designer to the machine and then treat the situation as seen from a dialogue partner perspective, whereas Andersen (1985) treats the designer as having the role of one sender in a collective of senders, who makes a contribution to each message sent through the medium.”

-- Kammersgaard 1988: p. 356

Only a few years before, Winograd and Flores had published a notorious book, in which they proposed to shift the focus in computing from pursuing ever-increasing artificial cognitive capacities to supporting pervasive human communication and coordination processes (Winograd and Flores 1986). And in 1990, Andersen would publish the first comprehensive account of computers as media in his Theory of Computer Semiotics (Andersen 1990).

From the late 1980’s to the mid 1990’s, a number of researchers, in different parts of the world, started thinking about how to articulate Semiotics and Com-

25. Semiotics

1779

puting. In 1988, in a 55-page chapter in Hartson’s Advances in Human-Computer Interaction (Hartson 1988), Nadin (Nadin 1988a) explored the semiotic implications of interface design and evaluation. In the same year, he would also publish an article in Semiotica, about a semiotic paradigm to systems interface design (Nadin 1988b). In 1993, Andersen edited a book explicitly called The Computer as Medium (Andersen et al 1993), and we published our first paper on the semiotic engineering of human-computer interfaces (de Souza 1993).

All of this early work explored the power of viewing computer programs and systems — and especially their interfaces — as signs of a very specific kind. They are designed and engineered according to the laws and limits of computation, but what they ultimately communicate is a message meant by humans (systems designers and developers) and to humans (systems users). This is the rationale for viewing human-computer interaction as computer-mediated human communication, the powerful idea underlying Winograd and Flores’s manifesto for a language-action perspective (LAP). LAP attracted many semiotically-inspired researchers in Computer Science (Communications of the ACM, May 2006). Most of them turned towards Organizational Semiotics (Liu 2000), which in general is closer to information systems development than to human-computer interaction design. In Brazil, however, a group of researchers has been exploring the use of Organizational Semiotics concepts in the design of human-computer interaction (Baranauskas et al 2003; Bonacin et al 2004; Neris et al 2011).

Semiotics was also implicit or explicit in the work of other researchers working in HCI. For example, Mullet and Sano (Mullet and Sano 1995) explored semiotic features of representations used in the design of visual interfaces. Their emphasis was on how to communicate the design intent to users through interface signs, an idea that was gaining momentum at the time. In fact, computer systems interfaces provided an excellent context to revisit the concept that there are languages of/in design. The chapter by Rheinfrank and Evenson in Winograd’s Bringing design to software (Winograd 1996) is a great illustration. The authors discuss design as communication, whether intentional or not, and advance the

1780

Encyclopedia of Human-Computer Interaction

idea that when it is intentional, the use of design languages can increase the effectiveness of communication.

As computers became more pervasive and took over a larger portion of social processes, the computers as media perspective gained the interest of researchers investigating the users’ response to intentional communication by service providers (Light 2001) and information appliances in general (Fogg 2002), for example. Fogg’s work, for one, developed into an elaborate study of persuasion through technology (Fogg 2003), which has been interestingly explored in the context of games and culture-sensitive communication (Khaled et al 2006). In a recent work, O’Neill (O’Neill 2005) theorizes about interactive media in view of contemporary technologies that provide new mediation opportunities for human semiotic processes. Research on the convergence of Semiotics, interactive technology and literacy is also noteworthy. Marion Walton’s work (Walton 2008) in South Africa provides insightful elements to the study of computers as media, enriched by empirical evidence collected in African school children’s encounters with computers and the Internet (Prinsloo and Walton 2008).

Inspired by Andersen’s pioneering work in Computer Semiotics, we developed an extensive semiotic account of human-computer interaction, Semiotic Engineering (de Souza 2005). Having started as a semiotic approach to user interface language design (de Souza 1993), over the years Semiotic Engineering evolved into a semiotic theory of HCI with its own ontology and specifically-designed methods of investigation (de Souza and Leitão 2009). This is the theory I will use in the rest of this chapter to illustrate, in practice, the kinds of connections existing between Semiotics and HCI.

25.4.1 Semiotic Engineering Semiotic Engineering picked up the early view that human-computer interaction is in fact computer-mediated human communication. It then defined and articulated a number of fundamental concepts, their relations with each other and their

25. Semiotics

1781

implications not only for the development of the theory itself (the internal motivation), but as a contribution to the advancement of HCI (the external motivation). The most striking distinction proposed by Semiotic Engineering compared to other theories and conceptual frameworks in HCI (Carroll 2003) is to develop the early suggestion put forth by Kammersgaard (Kammersgaard 1988) and postulate that designers of interactive software are active participants in the communication that takes place through user interfaces. They communicate their design vision to users by means of interface signs like words, icons, graphical layout, sounds, and interface controls like buttons, links, and dropdown lists. Users unfold and interpret this message as they interact with the system. In other words, the communication between users and systems is in fact part of a metacommunication process; part of the communication process initiated by designers about how, when, where and why to communicate with the system they have designed.

Since designers cannot be personally present when a user interacts with software, they have to represent themselves in the interface, using a specifically designed signification system, and subsequently tell the users what the software does, how it can be used, why, and so on. The choice of representations is actually wide. For instance, designers may represent themselves as humanoids (e. g. systems interfaces with human characteristics like affect and natural language abilities), as machines (e. g. systems interfaces with press buttons, slide controls, dials, and the like), or even as spaces (e. g. systems interfaces with virtual worlds that users explore to achieve various kinds of effects). Depending on the message that they have to communicate, some representations will work better than others. Humanoids, for example, are likely to communicate explanations and instructions more easily than virtual spaces. Machines, in turn, convey physical affordances more efficiently than the natural language discourse of humanoids.

No matter their self-representation, through systems interfaces designers are telling users:

1782

Encyclopedia of Human-Computer Interaction

ff what they know about the users (who they are, what they know, what they wish or need to do, in which preferred ways, and why); ff how they have responded to the users’ needs or aspirations (what system they have built and how it works) ff what values are encoded in their response (why and how does the system improve the users’ lives).

Here is a very simple sketched example of how the elements above are communicated by designers through the interface. We contrast Microsoft Word (Figure 25.5) and Open Office Writer (Figure 25.6) tools menu, showing how the designers’ message comes across.

25. Semiotics

1783

Figure 25.5: A sketched version of Microsoft’s Word interface unfolding the Tools menu.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

1784

Encyclopedia of Human-Computer Interaction

Figure 25.6: A sketched version of Open Office’s Writer interface unfolding the Tools menu.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

The contrast of both designs shows that they communicate different views on users’ needs and opportunities. The following is a summary of the most striking differences between the two messages communicated through Microsoft Word and Open Office Writer interfaces.

25. Semiotics

1785

25.4.2 Contrast between MS Word and the OO Writer interfaces

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

1786

Encyclopedia of Human-Computer Interaction

Microsoft Word MS Word Tools can be grouped into 4 categories (list items separated by a line): the first with tools related to document content; the second to document manipulation and authoring; the third to mail preparation; and the fourth to interface customization and extensions.

Copyright status: Unknown (pending investigation). See section “Exceptions� in the copyright terms.

25. Semiotics

1787

Open Office Writer OO Writer Tools can be grouped into 7 categories (list items separated by a line): the first with tools related to document content; the second to document structure; the third to inclusion of media files; the fourth to bibliography handling; the fifth to mail preparation; the sixth to data field manipulation; and the seventh to interface customization and extension.

The designers of both applications are telling their users that tools applying to document content are top of the list, and that ‘Spelling and Grammar’ tools are the most important ones. By comparison, we (like users) get the message that MS Word provides more resources than OO Writer in this particular category. In fact, OO Writer tools are a subset of MS Word’s. Similarities in shortcuts are remarkable, suggesting that OO Writer’s designers expect their users to have (or to have had) experience with MS Word at some point in their lives.

Similarities extend also to other categories. In both applications, there are tools to handle mail preparation and to support customization and extensions. The tools in these categories, however, may have different names, and OO Writer provides an additional tool, compared to MS Word: it lets the users customize ‘XML Filters Settings’.

At a closer look, we see that differences in names actually communicate a different perspective on the tools themselves. Whereas in MS Word names tend to express objects (with the exception of ‘customize’), in OO Writer the designers’ communication, when different from MS Word’s, alludes to agents: ‘Mail Merge Wizard’ and ‘Extension Manager’. In these two cases, the communication is clearer (more precise) than in MS Word, which may achieve the effect of encouraging novice users into experimenting new features (since a wizard and a manager will be there to help them).

1788

Encyclopedia of Human-Computer Interaction

Microsoft Word

Open Office Writer

MS Word’s designers communicate

OO Writer’s designers, by con-

very clearly their emphasis on group

trast, do not talk about computer-

collaboration. They are telling us-

supported collaboration. The com-

ers that document preparation is

munication of their design vision

or can be a group activity, requir-

at this point portrays an individual

ing such things as shared work-

user, working on the details of his/

spaces and online communication,

her document. The designers give

in addition to tracking changes and

special

comparing, merging and protecting

structure (numbering and notes)

documents. The communication

and bibliographic reference man-

of their design vision at this point

agement, communicating that they

clearly locates users in a computer-

want to support users in preparing

supported organizational context.

complex extensive documents.

emphasis

to

document

By comparison with OO Writer’s designers’ communication, MS Word’s designers communicate their concern with usability in different ways. Five of their tools come with icon representations that help users identify their presence in toolbars (a more concise and more easily accessible interface control than menus). Three menu options have associated keyboard shortcuts, and four options unfold into submenus, rather than invoking dialog windows (communicated by ‘...’). Together these messages tell the users that MS Word’s designers are concerned with providing fast access and fast learning support for their users. In contrast, OO Writer’s designers favor longer dialogs (‘...’), which actually is in line with longer processes that can also be expected from interaction with a ‘wizard’ and a ‘manager’. Furthermore, by communicating that they expect their users to need a bibliographic database, OO Writer’s designers give us the impression that they have

25. Semiotics

1789

worked for meticulous users engaged in longer-term tasks. By contrast, MS Word designers give us the impression that they have worked for busy users, engaged in broader-context activities of which document preparation is only a part.

In both cases, however, the complete message about which tools are available and how valuable they are expected to be for the targeted users is not likely to be understood unless users engage into interacting with the application. In MS Word, for example, in order to understand what “research” really means in context, users will probably click on the option (i. e. interact with the message itself) and see what comes next. If they do, they’ll find out that the designers are offering the possibility to connect to and search online reference books and other resources. Likewise, in order to understand what “bibliography database” is about in OO Writer, users will also probably click on the option. If they do, they will find out that this option connects them to an existing ‘bibliography’ that is installed with OO Writer, and which can be extended or modified to accommodate the users’ reference items. In both applications, the users can resort to online help resources to learn more about these and other parts of the designers’ message conveyed in the menus above. Thus, we see that communication is not limited to visible signs in isolated screens — communication is also conveyed in the process of subsequent communication with the application and its resources (i.e. online help and documentation).

The summative contrast between communication expressed by the MS Word and the OO Writer interfaces is a good sampler of the essence of Semiotic Engineering, whose main tenets are the following: 1.

HCI is an instance of metacommunication — communication about how, when, where and why to communicate with computer systems. The metacommunication message content can be summarized as follows (note the presence of the designers using 1st person pronouns like “I”, “my”, etc.)

1790

Encyclopedia of Human-Computer Interaction

“Here is my understanding of who you are, what I’ve learned you want or need to do, in which preferred ways, and why. This is the system that I have therefore designed for you, and this is the way you can or should use it in order to fulfill a range of purposes that fall within this vision.”

2. Both designers and users are engaged in metacommunication; the system’s interface represents its designers at interaction time, it is the designers’ deputy. The interface enables all and only the designed types of user-system conversations encoded in the underlying computer programs at development time. The metacommunication message from the designers is unfolded and received as users interact with it and learn ‘what the system means’.

3. There are three classes of metacommunication signs that designers can use to communicate their message to users. Static signs communicate the essence of their meaning with instant time-independent representations (typically representations that can be correctly interpreted in static screen shots). Dynamic signs communicate the essence of their meaning with a series of time-dependent representations (typically representations that can only be correctly interpreted over a number of subsequent screens or states of the system). Metalinguistic signs are static or dynamic signs that differ from either the former or the latter because the essence of their meaning is an explanation, a description, an illustration, a demonstration or an indication of other [interface] signs (typically textual or video material referring to the meaning of some other static or dynamic sign).

25. Semiotics

1791

4. Metacommunication signs in HCI must be produced computationally.

Consequently, this specific kind of computer-mediated human communication introduces critical constraints in sign production processes.

Systems designers must create representations that by necessity have a single definitive encoded meaning — no matter if the designers (and the users) can easily produce evolved meanings for these representations in natural sign-exchange situations. The algorithmic nature of the medium in which metacommunication takes place mechanizes human semiosis, in both directions (designers’ signs produced by the system interface and users’ signs produced with the system interface).

5. The quality required for metacommunication to be efficient and effective is communicability, a system’s ability to signify and communicate the designers’ intent (which is ultimately to satisfy the users). The evaluation of communicability involves a methodical analysis of how the designers’ message is emitted (composed and sent through the interface) and of how it is received (interpreted and followed by physical and/or mental sign-mediated action) by the users.

6. Because the users’ response to the designers’ metacommunication must be mediated by interface signs, the users must learn the interface language, a unique signification system, in which the designers’ message is fully encoded. Users learn this language in the very process of using it in interaction. This process is similar to natural language acquisition, which humans are fully equipped to do, except that the users’ immediate interlocutor (unlike humans) does not reason abductively. Therefore, some of the users’ communicative intent may persistently fail to be interpreted simply because the designers have not anticipated the users’ sign-making strategies. Because this can cause disorientation in the process of interface language acquisition, designers should also ex-

1792

Encyclopedia of Human-Computer Interaction

plicitly signify the communication principles that they have chosen to encode their message.

Semiotic Engineering has been gradually making its way into HCI, especially after Don Norman, one of the leading figures in early UCD (Norman 1986), wrote about advantages that he sees in a semiotic engineering perspective on HCI (Norman 2004; Norman 2007; Norman 2009). In the next section, we begin to conclude this chapter with very brief concrete examples of how this perspective influences the analysis and design of computer-mediated communication in HCI.

25.5 A closer look at computer-mediated communication As mentioned earlier in this chapter, saying that computers are media has become a cliché in the turn of the century. Computers are everywhere, computing is embedded in hundreds of things that we carry and encounter in daily life, and digital information exchanged through computers, with or without our control, affects almost every step we take as individuals and as a society. A relevant question for HCI is: How well are HCI theories prepared to inform and equip HCI designers in this context?

The question is actually not new. In 1986, Winograd and Flores asked it from the Artificial Intelligence community, mainly, hitting additional targets in HCI and in Computer-Supported Collaborative Work (Winograd and Flores 1986). In spite of this history, however, most of the existing HCI theories are dominated by cognitive perspectives. They can and do inform the design of interaction with respect to facilitating the learning, memorization and retrieval of productive interactive patterns, for instance. And nobody disputes that this is a fundamental requirement in HCI design. However, the user-centered approach, as its name so clearly ‘signifies’,

25. Semiotics

1793

intentionally or unintentionally has treated design meanings as if they were mindindependent entities that are somehow elicited from users and then reified in design models and prototypes before they eventually take the shape of a system’s interface.

The designers’ interpretation and signification processes are not accounted for in original UCD (Norman 1986), whose theorists have in general tended to follow the Cartesian tradition of postulating the existence of primary cognitions. These are shared by all humans (and thus by all users), which stimulates research seeking for universal primitives and, based on their consequences, making predictions about users’ behavior.

The problem of UCD in the computer as medium paradigm is that it cannot accommodate computer-mediated communication between systems designers and users. Designers, quite plainly, do not belong in UCD interaction models. Neither of the two most influential historical sources of UCD — the seven-step theory of action (Norman 1986) and the human information processing model (Card et al 1983) — account for the designers’ cognitive processes and meaning-making activity. This part of the story has been covered by methods (e. g. ethnography (Bouissac 1998) and theories (e. g. Activity Theory (Kaptelinin and Nardi 2006)), which have nonetheless failed to form with UCD a seamless body of knowledge that can satisfactorily account for the whole design process in accordance with the computer as medium perspective. The final ad hoc combination of uncohesive parts in actual design processes seems to be what Spool refers to as the “skills and talents of practitioners” (p. 8, above).

Although the new media perspective instantly reminds us of Web applications design and of interaction with or through mobile devices, there is yet more to it. Digital literacy, which is undoubtfully a requirement for the achievement of full citizenship in the 21st century, has evolved to include computational thinking skills without which users (citizens) are not likely to be able to engage in the new cultures of participation (Fischer 2011).

1794

Encyclopedia of Human-Computer Interaction

In the abstract of a thoughtful and provocative article, Jeannette Wing (Wing 2008) expresses her perception of the extent to which changes in current computing practices and possibilities affect science, technology and society:

“Computational thinking will influence everyone in every field of endeavour. This vision poses a new educational challenge for our society, especially for our children. In thinking about computing, we need to be attuned to the three drivers of our field: science, technology and society. Accelerating technological advances and monumental societal demands force us to revisit the most basic scientific questions of computing.”

-- Wing 2008: p. 3717

In the following, I will present three contemporary examples of different levels of computational thinking involved in new kinds of social interactions. In each case, I will underline the potential advantages of a semiotic framing for the illustrated phenomenon. Then, in a separate sub-section, I will talk about research we are doing at SERG, the Semiotic Engineering Research Group of Pontifícia Universidade Católica do Rio de Janeiro (PUC-Rio), partnering with colleagues in Colorado University at Boulder and Universidade Federal Fluminense (UFF), a public university in the State of Rio de Janeiro, Brazil.

25.5.1 Some contemporary examples In the illustrations below, we will see how computer-mediated human communication may involve explicit computational representations of self and message at different levels of complexity. In each case, the purpose of communication is very prosaic — to help someone use his email account. The innovation lies in the

25. Semiotics

1795

chosen form of communication and the wealth of new meanings that it introduces in human experience. We will also see that computing literacy is the key factor for reaping the benefits of this new means of self-expression and social participation. Taking a semiotic perspective on literacy — the ability to interpret and produce signs of socially-valued signification systems in order to achieve social participation and full citizenship — the point of the illustrated cases is to show that being able to program computers has become as important in the 21st century as reading, writing and counting has been in previous centuries.

25.5.1.1 Example 1: Programming via parameter setting

Figure 25.7: Internet Buttons” entry page.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

The first example is Internet Buttons®, a tool that allows you to create a personalized page with buttons that direct you straight to your favorite web sites, services and applications. Internet Buttons was created by a not-for-profit

1796

Encyclopedia of Human-Computer Interaction

company in the UK “to get people from different generations talking more, sharing more and spending more time together.” In Figure 25.7, we show their home page. Notice how they get their message across with different kinds of signs like text (“Create a page of buttons for someone you know [...]” and “Hey Mum, call me if you get stuck [...]”) and image (colorful buttons and personal picture) and think of how obviously the software designers not only participate actively in interaction with the user, but also of how they would completely fail to achieve their intent if they did not have the ability (a “personal talent”?) to express themselves through software.

The beauty of Internet Buttons lies, however, in the recursive nature of the designers’ message. Through their program, they are inviting Internet users to program “super easy” Internet interaction for “someone they know”. The programming paradigm is also an extremely simple form of parametric procedure where all that the users have to do is provide the correct values for pre-selected parameters.

Therefore, the kind of computational thinking required is very basic. In Figure 25.8, we show a step of the programming required to access a Gmail account.

25. Semiotics

1797

Figure 25.8: Creating an Internet Button to access Google’s Gmail login page.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

As the Internet Buttons entry page shows, once the buttons are programmed they go into a personalized web page where the user can place any number of buttons especially created for someone else. The personalized page template allows the user-programmer to add a personal picture and a message directly addressed to whoever he are talking to through software that speaks for him. Therefore, the software carries a representation of “self” in addition to a representation of the “message” to be communicated.

25.5.1.2 Example 2: Programming via macro recording Our next example is IBM’s CoScripter®, a macro recorder for the Web. Unlike Internet Buttons, CoScripter does not focus on personal communication

1798

Encyclopedia of Human-Computer Interaction

and representations of self. However, just like Internet Buttons, the purpose of CoScripter is to make Internet processes easier — for the user himself/herself and for whoever needs help with Web interaction. The system allows the user to record and playback interaction steps, optionally using the values of variables stored in a “Personal Database” in the user’s machine. Recorded CoScripts are stored in an Internet server and can be shared with other users. In Figure 25.9, the arrows that we added to the snapshot show how, in this case, the CoScript instructions (on the left) use the information in the Personal Database (at the bottom left) to command interaction with the Gmail login page.

Figure 25.9: The execution of a CoScript for accessing a Gmail account.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

Although the scripters are not explicitly represented in the CoScript (an interesting sign of the effect of impersonal and implied communication between script designer and script user), the purpose of CoScripter creators is also interpersonal communication through software. In their home page, visited in December of 2011, we read the following message:

25. Semiotics

1799

“CoScripter is a system for recording, automating, and sharing processes performed in a web browser such as printing photos online, requesting a vacation hold for postal mail, or checking flight arrival times. Instructions for processes are recorded and stored in easy-toread text here on the CoScripter web site, so anyone can make use of them. If you are having trouble with a web-based process, check to see if someone has written a CoScript for it!”

-- http: //coscripter.researchlabs.ibm.com/coscripter

The programming paradigm in CoScripter is more sophisticated than in Internet Buttons. One of its creators, Allen Cypher, is a leading figure in programming by demonstration (Cypher 1993). The user-programmer in this case is dealing with a simple programming language, in which demonstrated interactions are automatically encoded. Optionally, the user may manipulate variables to make CoScripts more general and reusable in similar, but not identical, contexts. Mainly because of variable manipulations, the level of computational thinking required to use CoScripter is intermediary. There are a number of interesting semiotic issues to explore with CoScripter. In this chapter, I will only briefly remark that, unlike Internet Buttons, whose messages are computationally encoded with fixed “token-level” semantics (each button means a single Internet address), CoScripter messages can be computationally encoded with fixed “type-level” semantics (each script can be executed with different parameters specified in the end user’s Personal Database). Each CoScript means a range of possible interactions with the same web page, web service or web application. All of the interactions are predicated by the same interactive steps, but there can be variations in contextual parameters. Thus, the communication content that can be expressed

1800

Encyclopedia of Human-Computer Interaction

with CoScripter is considerably more complex (and powerful) than in the previous example. The interested reader can see a deeper semiotic analysis of CoScripter in one of our previous publications (de Souza and Cypher 2008).

Figure 25.10: Programming with “Customize Your Web” Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

25.5.1.3 Example 3: Programming on wheels Our last example is Customize Your Web, a personalization tool that allows users to change the appearance of web pages and add new functionality to them without having

25. Semiotics

1801

to dive too deeply into JavaScript programming if they don’t want or don’t know how to do it. Customize Your Web is an extension to Firefox designed by Rudolf Noe. In Figure 25.10, we show the end user programming environment offered by this tool.

The user can select elements of a Web page (in this example, the “Google Search” button is selected) and make changes to its appearance and behavior (see the menu from which the user is about to select between “Insert JavaScript”, “Insert HTML” or “Insert CSS”). Interface elements can be deleted, inserted and relocated as desired, as long as they can be uniquely identified on the existing page whenever it is loaded.

Compared to the previous example, Customize Your Web opens even more powerful possibilities for computer-mediated communication. In Figure 25.11 and Figure 25.12, we show how facilitating interaction with Gmail is substantially different in Customize Your Web, compared to CoScripter and Internet Buttons. Because the programming paradigm is close to programming on wheels, the possibilities for representation of self and message are limited only by technicalities of the tool (like problems with unique identification of elements on certain Web pages) and the resources of the user. The level of computational thinking required to use this tool effectively is advanced.

1802

Encyclopedia of Human-Computer Interaction

Figure 25.11: Gmail login page modified with “Customize your Web” Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

In Figure 25.11 (see customized text starting with “Hi, Dad” on the left), we can see that the user can script his own presence and participation in someone else’s experience with Gmail. The complexity of self-representation and message in this case is considerably high. Notice that in Figure 25.12, the conversation between the scripter and the targeted user (his Dad) extends over whole interaction spans with Gmail. In other words, a parallel communication about communication with Gmail (i. e. genuine metacommunication about Gmail) is explicitly and intentionally in place. This is one of the best examples of why the computer as media perspective calls for a different breed of HCI theories in order to help end users take the best out of the virtually unlimited possibilities of social interaction and participation now available for them.

25. Semiotics

1803

Figure 25.12: Gmail inbox page modified with “Customize your Web” Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

Designing cohesive and consistent dialog turns throughout a Web process, like shown in the Customize Your Web example, is end user HCI design — not only end user programming and development. However, theories that do not include the designers in the communication process that they are designing have very little to say about how (and why) the current interface of Customize Your Web (see Figure 25.10) must be improved. For example, UCD is not going to help the user design all the conversational paths that his parallel communication with Dad might take in his absence. Dad can decide to use any of the available interface controls during interaction with Gmail. So, how does the user build a representation of conversational context that is consistent with what he is telling Dad at each step of the programmed interaction?

1804

Encyclopedia of Human-Computer Interaction

The answer to these questions can not only show the complexity of programming that end users can engage in at this stage of technological development, but also — and more importantly — it can show what kinds of new dimensions a semiotic perspective can bring to HCI research and practice. In the next sub-section, we will very quickly mention how we are exploring computer-mediated communication issues with Semiotic Engineering.

25.5.2 Going a few steps further One’s level of computing literacy will determine how one will be able to participate in new kinds of social interactions made possible by new ICT. As the examples above have shown, higher literacy levels raise the transformative power of communication to unprecedented standards. The challenge is in the air: “If computational thinking will be used everywhere, then it will touch everyone directly or indirectly. [...] If [it] is added to the repertoire of thinking abilities, then how and when should people learn this kind of thinking and how and when should we teach it?” (Wing 2008: p. 3720) One of the most successful responses to-date is AgentSheets, a visual programming environment designed mainly for teaching computational thinking at schools (Repenning and Ioannidou 2004). The Scalable Game Design Project, carried out by Alexander Repenning and his group at Colorado University at Boulder has been educating school teachers and students in computational thinking for many years now. In 2010, we brought the project to Brazil and started working with a public school in the city of Niterói.

Given our interest in semiotic theories of HCI, we have approached the project with a dual perspective. On the one hand, we want to educate teachers and students in computational thinking. On the other, we want to show them that computational thinking gives them a completely new language for self-expression and communication, with which they can do whatever they can mean (or “signify”, in semiotic terms). Our strategy is to work constantly with the semiotic

25. Semiotics

1805

relations that hold between three distinct signification systems (see Figure 25.13): natural language, game play (i. e. executable games programmed by AgentSheets users) and game code (i. e. the various levels of programming language encodings that make the games run). In AgentSheets, the users can program in Visual AgenTalk, a visual programming language with a textual XML counterpart.

Figure 25.13: Semiotic varieties in a visual programming environment.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

Empirical evidence collected in game design sessions with a group of students in 2010 has shown elaborate and intriguing signification patterns when we compared, for each game, the natural language descriptions provided by the programmers, the Visual AgenTalk code, and the resulting executable game representations (de Souza et al 2011). In particular, we traced interesting entity-naming strategies, token/type relations in representation choices, and curious transitive

1806

Encyclopedia of Human-Computer Interaction

structure changes when contrasting natural language and computer program representations. Subsequent research steps carried out in 2011 with another group of students suggest that semiotic relations between signification systems involved in using AgentSheets can be used to raise the level of computing awareness among learners.

In Figure 25.14, we show successive screen shot snippets from a semiotic exploration of relations between game play signs, game code signs and natural language signs in a very abstract environment used by CS graduate students. The “game” representation (an executable simulation) has only four static signs: an orange circle; three colored squares; and a black background. The dynamic signs are also very few and very simple: if the user presses arrow keys the orange circles moves in the corresponding direction; if the circle moves next to the boundaries of any of the three squares, it is “trapped” by the square and the game is “reset”.

The fascinating aspects of the exploration were: to examine what meanings the participating students assigned to the simulation; and to examine how the apparently identical behaviors of the three squares had been intentionally encoded by the experimenters as completely different representations.

25. Semiotics

1807

Figure 25.14: Screen shots from a semiotic exploration of simulation signs with AgentSheets.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

When asked to write down their explanation (“Explicação” in Figure 25.14) for the orange circle behavior, inferred exclusively from the participant’s interaction with the game, the students used signs referenced to the “protagonist” of the game. For example, the explanation depicted in Figure 25.14 says that “the orange circle cannot collide with any of the obstacles represented by the squares”. A totally different story, however, would be told if the student looked at the underlying program.

The three squares are actually programmed as very different agents. The green square “traps” the circle as it comes near it. The white square is like an attractive area into which the orange circle “throws itself” as soon as it comes near it. And the white square is no more than “a hole in the ground”, an agent-free space in the worksheet, into which the circle may “fall propelled by the ground”. Thus, the designer’s story about this simulation is a totally different one if looked at from the inside or the outside. From the inside, it might go like this, for instance: “The

1808

Encyclopedia of Human-Computer Interaction

orange circle loves the grey square and jumps into it as soon as it sees it; it must escape from the green square that traps it when the circle comes near it; and it must beware of a square-shaped hole in the ground, into which it must not fall.”

An HCI-oriented mind will probably say that for the designer to communicate his story appropriately to the user, the abstract squares should be replaced with icons that adequately represent “trap”, “attractive area” and “hole in the ground”. However, this is actually telling only part of the computational thinking story, of course. The fact that similar effects can be computationally represented by very different data structures and algorithms is one of the most fundamental and powerful principles of Computer Science. Therefore, we should be telling school teachers and students that computational signs are like plastic semiotic material, and stimulating them to play with it in as many ways as they can imagine. This is what will really empower them to express themselves computationally.

In Figure 25.15, we show a sketched version of Exploratorium, a semiotically-rich prototype plug-in to AgentSheets, which we are developing for its users to be able to explore how different game play representations are encoded in Visual AgenTalk. Our intent is to raise the students’ and teachers’ awareness of the various possibilities in computer representations, and to stimulate them to play with these possibilities at their will. The links on the illustrated page show that users can ask a number of things about the representations they see (how they are produced, where they are located, what they stand for and why). The answers are partly generated automatically from Visual AgenTalk representations and partly retrieved from annotations (secondary notations) elicited from the programmers in dialogs about what they meant.

25. Semiotics

1809

Figure 25.15: The Scalable Game Design Brasil Project: A semiotically-rich environment to explore signs in computing.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

In Figure 25.16, we show a sketched version of the Web Navigation Helper (Intrator and de Souza 2009; Intrator and de Souza 2008; Monteiro and de Souza 2011), a user agent to increase Web accessibility that we have been expanding to bridge cultural barriers of various sorts (language barriers, literacy barriers, etc.).

As shown in this illustration, the user agent actually rephrases the designer’s communication originally expressed in a Web page. In Figure 25.16, the user agent is an interpreter for a Brazilian user that is trying to navigate a Web page produced in the USA. Technically, the user agent is a Firefox extension that works in combination with CoScripter. For every interaction that it mediates, the agent requires a recorded CoScript and a full specification of the mediating dialog that will take place at each step of the recorded interaction.

The illustrated dialog, in Portuguese, is telling the Brazilian user how he or she can interact with the Scalable Game Design Arcade page. Notice that the dia-

1810

Encyclopedia of Human-Computer Interaction

log is not a verbatim translation of the page (although this would also be a kind of mediation). The agent is actually guiding a foreigner’s interaction with material produced in and by another culture.

Figure 25.16: An interaction mediator helping a Brazilian user navigate through a website in English.

Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

Guidance is designed using conceptual viewpoint metaphors proposed by Salgado (Salgado et al 2011; Salgado et al 2012). Very briefly, the metaphors allow interaction designers to organize cross-cultural communication from different perspectives, ranging from the strongest form of mediation (where the foreign

1811

25. Semiotics

culture is invisible to the user thanks to communication carried out in his or her own native language, with complete neutralization of all references to foreign cultural practices and values) to the weakest, actually non-existing, form of mediation (where the foreign culture is fully exposed, with no trace of the user’s culture — communication is carried out in a foreign language, with reference to foreign cultural practices and values).

Semiotic Engineering research, as the examples above can show, expands the limits of user-centered HCI and establishes connections with other specialized areas of Computer Science and Information Systems Development, like Programming (through the semiotics of computer representations) and Information Architecture (through the organization of explicit cultural references to be used in computational signification systems).

25.6 So, what’s in it for me?

In his preface to the Encyclopedia of Semiotics Paul Bouissac writes:

“The twentieth century has witnessed an increasing fragmentation of knowledge into a multitude of disciplines and specialties. At the same time, integrative visions have arisen in an effort to make sense of the flood of information generated by the modern intensification of formal and empirical research. Semiotics represents one of the main attempts — perhaps the most enduring one — at conceiving a transdisciplinary framework through which interfaces can be constructed between distinct domains of inquiry.”

-- Bouissac 1998: p. ix

1812

Encyclopedia of Human-Computer Interaction

Only a few years after Bouissac, John Carroll writes in the introduction to one of his books:

“An ironic downside of the inclusive multidisciplinarity of HCI is fragmentation. [...] There are too many theories, too many methods, too many application domains, too many systems. Indeed, fragmentation may be a bit worse than it has to be. Some HCI researchers, faced with the huge intellectual scope of concepts and approaches, deliberately insulate themselves from some portion of the field’s activity and knowledge. This tension between depth and breadth in scientific expertise is not unique to HCI but it clearly undermines the opportunity for multidisciplinary progress.”

-- Card et al 1983: p. 6

So, what’s in Semiotics for you?

I hope that this chapter has provided multiple signs of my personal answer to the question. Semiotics can provide the foundations for new integrative theories of HCI. Now, why should we need integrative theories? One of the answers is explicitly given by John Carroll in the citation above. But this chapter has touched on some of the implications of Carroll’s answers, as well as on some other possible answers to the challenging question.

A semiotic perspective on HCI proposes that meaning is the most fundamental concept in this discipline, and that our inquiry should account for how meaning is communicated through computer systems interfaces, in as many guises as they now occur. It follows from this that HCI must not only care for meaning takers, but

25. Semiotics

1813

also for meaning makers: communication implies an exchange of signs produced by at least two communicating minds. Therefore, to escape the embarrassment of having to postulate that systems have minds of their own, HCI will benefit from theories that can account for meaning-making processes in which HCI designers clearly engage while in the process of building systems interfaces to support productive user interaction with computing artifacts. In fact, computer meaning-making process can engage not only designers, but also programmers and a whole host of developers that encode meaning elements (signs) in software tools with which modern computer artifacts are built. Therefore, a semiotic perspective can potentially track human signification throughout long chains of software instances, in space and time, just like semiotic theories trace meaning in society, culture and history.

Because they push the designers into representing self-images and thinking of their communicative intent and strategies, HCI theories of semiotic extraction can also stimulate the development of a new ethics in HCI design. Although the UCD tradition has clearly established an ethics commitment with the users’ needs and aspirations, it has not contributed much to an ethics of communication since UCD theories only talk about relations and interactions between users and systems — not about users and systems designers. It is remarkable, however, that in his most recent book (Norman 2010), Don Norman, who inspired UCD with his Cognitive Engineering and seven-step theory of action in the 1980’s (Norman 1986) speaks openly of the importance of communication to live with complexity, and goes as far as dedicating a full chapter to “social signifiers” — a semiotic concept. This is itself the sign of that an ethics of communication begins to make its way into design and HCI. This new ethics may play a relevant role in the wake of the 21st century when global awareness of the need to respect and preserve cultural diversity is greater than ever before. A semiotic perspective can help technologists in trying to prevent that their products inadvertently extinguish valued diversity of meanings from user communities (Salgado et al 2012).

1814

Encyclopedia of Human-Computer Interaction

By bringing designers and users together in the process of HCI, a semiotic perspective can naturally promote more reflection in action (Schön 1983). When designers view themselves as talking to users, they can naturally tap on their instinctive abilities to signify reality and prompt communicative reaction from people around them. They are as native in the “human” side of human-computer interaction as the users. Therefore, we need as many good theories about them as we have about users. Reflective theories will help designers to produce computermediated communication in which they can consciously model their own signification and communication A semiotic outlook can also bring about some clarifications at meta-theoretical levels that scientific research should attend to. In this chapter, we have only highlighted aspects of Cartesian influence in the thinking of many HCI researchers and practitioners. We commented on why ultimate primary cognitions, which human minds can access by means of a rigorous inspection method, are not only appealing to computer language theorists, but often tacitly subscribed by members of the HCI community. This notion allows for framing HCI as a formal linguistic exchange between humans and computers. Semiotic theories that are philosophically aligned with this specific Cartesian view will probably reinforce beliefs that there are primitive conceptual meanings that can be known in the process of investigation.

However, even semiotic theories that do not ‘upset the applecart’ in CS, as non-Cartesian theories tend to do, lead researchers into making explicit commitments with the ontological origins of meaning and its variations. For example, a semiotic foundation requires that researchers provide a theoretical explanation of why and how people (in our case users and designers) assign different meanings to identical representational stimuli. This explanation will guide their choices in designing computer-mediated communication between software producers and software users, showing that theoretically-informed design is actually a lot more than a craft. Researchers will also have to deal with hard methodological and

25. Semiotics

1815

epistemological questions. If there are such things as conceptual meaning primitives, what are they? How can they tell true primitives from false ones? How can they explain that sometimes our meanings are right, and other times these very meanings are wrong? How can they explain why, even when some of their meanings are wrong, people occasionally make the right inferences using them? All of these questions are fundamentally important for scientific research in HCI and, as we have seen, Semiotics can play an important role in helping us to formulate them, to understand what they mean, and then to investigate possible answers.

To conclude, it is my belief that, grounded on more explicit philosophical commitments, the most important contribution of Semiotics to HCI is to rehabilitate human talent and intuition from the dark corners where it has been hiding, and to bring it to light as one of the most powerful elements for effective, efficient, creative and enjoyable computer-mediated communication in the 21st century. Our task, as HCI researchers and practitioners, is to generate and use the knowledge that is necessary to turn computers into the richest medium for human signification and communication.

25.7 Acknowledgments I have to thank my sponsors — CNPq and FAPERJ — and many people that have supported, stimulated and developed research in Semiotic Engineering over the last twenty years. In particular, I would like to thank namely those that have contributed more directly to SERG projects mentioned in this chapter: Carla Leitão, Luciana Salgado, Cristina Garcia, Ingrid Monteiro, Juliana Ferreira, Marcelle Mota, Cleyton Slaviero, Eduardo Tolmasquim, Leonardo Faria, Alexander Repenning, Nadia Repenning, Allen Cypher, Gerhard Fischer, CoScripter developers and AgentSheets developers. I would also like to thank Mads Soegaard and Allan Holstein-Rathlou for first-class editorial work. I owe special thanks to Mads for having invited me to write this chapter and for his constant encouragement. It has

1816

Encyclopedia of Human-Computer Interaction

been a much longer project than I first thought it would be, but also a much more rewarding thinking and learning adventure.

For more information about Semiotic Engineering, please visit our website.

25.8  Commentary by Alan Blackwell How to cite this commentary in your report

Alan Blackwell

Š Alan Blackwell

I only have one big research question, but I attack it from a lot of different angles.

The question is representation. How do people make, see and use things that carry meaning? The angles from which I attack my question include various ways in which representations are applied (including design processes, interacting with technology, computer programming, visualisation), vario...

Alan Blackwell Alan Blackwell is a member of The Interaction Design Foundation

25. Semiotics

1817

Clarisse Sieckenius de Souza provides a superbly lucid introduction to the fundamental concepts of semiotics, and their relevance to interaction design, just as we would hope from the author of her influential textbooks introducing Semiotic Engineering. These are principles that are directly relevant to students and professional software designers, principles that many computing specialists find intriguing but also challenging. However her chapter carries a far more profound challenge to readers – a challenge that was already implicit when she juxtaposed the world views of semiotics and of engineering in her book titles. Where engineering methods have traditionally relied on materials whose properties and behavior can be predicted from immutable laws of physics or chemistry, the “materials” of interactive software are socially – semiotically – constructed, and always changing. When engineers are forced to confront computers as media, many engineering fundamentals appear to be threatened at the same time. This tension is likely to remain a feature of HCI research and practice for as long as HCI remains a branch of Computer Science.

But an even larger challenge looms in the way that we all choose to respond in future to the pervasiveness of computing technology. The campaign for Computational Thinking responds, quite rightly, to the demand for educated citizens who can engage in dialogue around computational media, rather than being passive consumers (if any such thing were ever possible in a semiotic world). One consequence of that dialogue is to reconceptualise the notion of “programming”, such that it becomes more than a simple engineering technique, but rather a mode of participation in society. This is a warning for those tempted to assume that computational thinking is simply the evangelical wing in the church of computer science. As shown by the pioneering work in end-user programming that de Souza presents from Cypher, Repenning and others, the ability to configure and control software assists all technology users to recognize the semiotic structures of the society we are building.

1818

Encyclopedia of Human-Computer Interaction

25.9 References (ed.) (1998): The Essential Peirce, Volume 2: Selected Philosophical Writings, 1893-1913.

Indiana University Press Andersen, Peter B. (1997): A Theory of Computer Semiotics : Semiotic Approaches to Construction and Assessment of Computer Systems. Cambridge University Press Andersen, Peter B. (1990): A Theory of Computer Semiotics. Cambridge, UK, Cambridge University Press Andersen, Peter Bøgh, Holmqvist, Berit and Jensen, Jens F. (eds.) (1994): The Computer as Medium (Learning in Doing: Social, Cognitive and Computational Perspectives). Cambridge University Press Andersen, Peter Bøgh, Holmqvist, Berit and Jensen, Jens F. (eds.) (1993): The Computer as Medium (Learning in Doing: Social, Cognitive and Computational Perspectives). Cambridge University Press Baranauskas, M. Cecilia M., Liu, Kecheng and Chong, Samuel (2003): Website Interfaces as Representamina of Organizational Behaviour. In: Gazendam, H. W., Jorna, René and Cijsouw, R. S. (eds.). “Dynamics and Change in Organizations: Studies in Organizational Semiotics (Studies in Organisational Semiotics, 3.)”. Springer Beyer, Hugh R. and Holtzblatt, Karen (1998): Contextual Design: Defining Customer-Centered Systems. San Francisco, Morgan Kaufmann Publishers Bonacin, Rodrigo, Baranauskas, M. Cecili C. and Liu, Kecheng (2004): Interface Design for the Changing Organisation: an Organisational Semiotics Approach. In: Liu, Kecheng (ed.).

“Virtual, Distributed and Flexible Organisations: Studies in Organisational Semiotics”.

Springer Bouissac, Paul (ed.) (1998): Encyclopedia of Semiotics. Oxford University Press, USA Buchler, Justus (ed.) (1955): Philosophical Writings of Peirce. Dover Publications Burrell, Gibson and Morgan, Gareth (1979): Sociological Paradigms and Organisational Analysis: Elements of the Sociology of Corporate Life. Ashgate Publishing Burrell, Gibson and Morgan, Gareth (2007): Sociological Paradigms and Organisational Analysis: Elements of the Sociology of Corporate Life. Ashgate Publishing Card, Stuart K., Moran, Thomas P. and Newell, Allen (1983): The Psychology of HumanComputer Interaction. Hillsdale, NJ, Lawrence Erlbaum Associates

25. Semiotics

1819

Carroll, John M. (ed.) (2003): HCI Models, Theories, and Frameworks: Toward a multidisciplinary science. San Francisco, Morgan Kaufmann Publishers Carroll, John M. and Rosson, Mary Beth (1987): The paradox of the active user. In: Carroll, John M. (ed.). “Interfacing Thought: Cognitive Aspects of Human-Computer Interaction (Bradford Books)”. The MIT Presspp. 80-111 Cypher, Allen (1993): Watch What I Do: Programming by Demonstration. Cambridge, MA, The MIT Press de Souza, Clarisse Sieckenius (1993): The Semiotic Engineering of User Interface Languages.

In International Journal of Man-Machine Studies, 39 (5) pp. 753-773 de Souza, Clarisse Sieckenius and Cypher, Allen (2008): Semiotic engineering in practice: redesigning the CoScripter interface. In: Levialdi, Stefano (ed.) AVI 2008 - Proceedings of the working conference on Advanced Visual Interfaces May 28-30, 2008, Napoli, Italy. pp.

165-172 de Souza, Clarisse Sieckenius and Leitäo, Carla Faria (eds.) (2009): Semiotic Engineering Methods for Scientific Research in HCI (Synthesis Lectures on Human-Centered Informatics). Morgan and Claypool Publishers Descartes, Rene (2004): Discours de la Methode. Project Gutenberg Eco, Umberto (1978): A Theory of Semiotics. Indiana University Press Eco, Umberto (1976): A Theory of Semiotics. Indiana University Press Fischer, Gerhard (2011): Understanding, fostering, and supporting cultures of participation.

In Interactions, 18 (3) pp. 42-53 Floch, Jean-Marie (ed.) (2001): Semiotics, Marketing and Communication: Beneath the Signs, the Strategies (International Marketing Series). Palgrave Macmillan Fogg, B. J. (2002): Persuasive Technology: Using Computers to Change What We Think and Do. Morgan Kaufmann Hirschheim, Rudy, Klein, Heinz K. and Lyytinen, Kalle (1995): Information Systems Development and Data Modeling: Conceptual and Philosophical Foundations. Cambridge University Press Hix, Deborah (ed.) (1988): Advances in Human-Computer Interaction. Norwood, NJ, Intellect Hjelmslev, Louis (1961): Prolegomena to a Theory of Language. Univ of Wisconsin Press

1820

Encyclopedia of Human-Computer Interaction

Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (1979): Introduction to Automata Theory, Languages, and Computation. Addison Wesley Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (2006): Introduction to Automata Theory, Languages, and Computation (3rd Edition). Addison Wesley Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (2000): Introduction to Automata Theory, Languages, and Computation (2nd Edition). Addison Wesley Houser, Nathan and Kloesel, Christian J.W. (eds.) (1992): The Essential Peirce, Volume 1: Selected Philosophical Writings창 (1867-1893). Indiana University Press Intrator, Chantal and de Souza, Clarisse Sieckenius (2008): Using web scripts to improve accessibility. In: Proceedings of the VIII Brazilian Symposium on Human Factors in Computing Systems 2008. pp. 292-295 Kammersgaard, John (1988): Four Different Perspectives on Human-Computer Interaction.

In International Journal of Man-Machine Studies, 28 (4) pp. 343-362 Kaptelinin, Victor and Nardi, Bonnie A. (2006): Acting with Technology: Activity Theory and Interaction Design. The MIT Press Khaled, Rilla, Barr, Pippin, Fischer, Ronald, Noble, James and Biddle, Robert (2006): Factoring culture into the design of a persuasive game. In: Kjeldskov, Jesper and Paay, Jane (eds.) Proceedings of OZCHI06, the CHISIG Annual Conference on Human-Computer Interaction 2006. pp. 213-220 Light, Ann (2001): Interactivity and User Commitment - Relationship Building through Interaction on Websites. In: Proceedings of the HCI01 Conference on People and Computers XV 2001. pp. 459-474 Liu, Kecheng (2000): Semiotics in Information Systems Engineering. Cambridge, Cambridge University Press Monteiro, Ingrid Teixeira and de Souza, Clarisse Sieckenius (2011): Embedded cultural features in the design of an accessibility agent for the web. In: Proceedings of the 6th international conference on Universal access in human-computer interaction design for all and eInclusion - Volume Part I 2011. pp. 295-304 Morris, Charles (1971): Writings on the General Theory of Signs. Mouton De Gruyter Mullet, Kevin and Sano, Darrell (1995): Designing Visual Interfaces: Communication Oriented Techniques. Sunsoft Press

25. Semiotics

1821

Nadin, Mihai (1988a): Interface design: A semiotic paradigm. In Semiotica, 69 (3) pp. 269302 Nadin, Mihai (1988b): Interface design and evaluation - Semiotic implications. In: Hix, Deborah (ed.). “Advances in Human-Computer Interaction”. Norwood, NJ: Intellectpp. 45100 Norman, Donald A. (2009): Systems thinking: a product is more than the product. In Interactions, 16 (5) pp. 52-54 Norman, Donald A. (1986): Cognitive engineering. In: Norman, Donald A. and Draper, Stephen W. (eds.). “User Centered System Design: New Perspectives on Human-Computer Interaction”. Hillsdale, NJ: Lawrence Erlbaum Associatespp. 31--61 Norman, Donald A. (2010): Living with Complexity. The MIT Press Norman, Donald A. (2007): The Design of Future Things. Basic Books Norman, Donald A. (2004). Design as communication. Retrieved 19 March 2012 from JND. org: http://jnd.org/dn.mss/design_as_communication.html Norman, Donald A. and Draper, Stephen W. (eds.) (1986): User Centered System Design: New Perspectives on Human-Computer Interaction. Hillsdale, NJ, Lawrence Erlbaum Associates Néris, Vânia Paula de Almeida, Almeida, Leonelo Dell Anhol, Miranda, Leonardo Cunha de, Hayashi, Elaine Cristina Saito and Baranauskas, M. Cecília C. (2011): Collective Construction of Meaning and System for an Inclusive Social Network. In International Journal of Information Systems and Social Change (IJISSC), 2 (3) pp. 16-35 O’Neill, Shaleph (2008): Interactive Media: The Semiotics of Embodied Interaction. Springer Prinsloo, Mastin and Walton, Marion (2008): Situated Responses to the Digital Literacies of Electronic Communication in Marginal School Settings. In Yearbook 2008 African Media African Children, pp. 99-116 Repenning, Alexander and Ioannidou, Andri (2004): Agent-based end-user development. In Communications of the ACM, 47 (9) pp. 43-46 Saussure, Ferdinand de (1972): Cours de linguistique générale. Payot Schön, Donald A. (1988): Educating the Reflective Practitioner: Toward A New Design for Teaching and Learning in the Professions. San Francisco, Jossey-Bass

1822

Encyclopedia of Human-Computer Interaction

Schรถn, Donald A. (1983): The Reflective Practitioner: How Professionals Think In Action.

Basic Books Souza, Clarisse Sieckenius de de (2005): The Semiotic Engineering of Human-Computer Interaction (Acting with Technology). The MIT Press Spool, Jared M. and Schaffer, Eric M. (2005): The great debate: can usability scale up?. In: Proceedings of ACM CHI 2005 Conference on Human Factors in Computing Systems 2005. pp. 1174-1175 Umiker-Sebeok, Jean (ed.) (1988): Marketing and Semiotics: New Directions in the Study of Signs for Sale. Mouton De Gruyter Wing, Jeannette M (2008): Computational thinking and thinking about computing. In Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366 (1881) pp. 3717-3725 Winograd, Terry (1996): Bringing Design to Software. ACM Press Winograd, Terry and Flores, Fernando (1986): Understanding Computers and Cognition.

Norwood, NJ, Intellect Zemanek, Heinz (1966): Semiotics and programming languages. In Communications of the ACM, 9 (3) pp. 139-143

25. Semiotics

1823

About the author Clarisse Sieckenius de Souza

© Clarisse Sieckenius de Souza

Has also published under the name of: “Clarisse de Souza” Personal Homepage: www-di.inf.puc-rio.br/~clarisse/ Clarisse Sieckenius de Souza is Full Professor of Computer Science at the Departamento de Informática, PUC-Rio, where she has also served as the director of Graduate Studies Program for two terms (2003-2005 and 2007-2009). In 2010, Clarisse was a co-winner of the prestigious ACM SIGDOC Rigo Award (see pictures and more in SERG Website). In 1996 she founded SERG (the Semiotic Engineering Research Group). Among the 15 M.Sc. and 19 Ph.D. students that she has (co-)supervised, more than a half are faculty in various Brazilian universities,

1824

Encyclopedia of Human-Computer Interaction

and nearly one third work in the industry. In 2005 she published her first book, The Semiotic Engineering of Human-Computer Interaction (The MIT Press). Her second book, Semiotic Engineering Methods for Scientific Research in HCI, coauthored by Carla Leitão, was published in 2009, in Morgan & Claypool’s Synthesis Lectures Series.

Your notes and thoughts on chapter 25 Record your notes and thoughts on this chapter. If you want to share these thoughts with others online, go to the bottom of the page at: http://www.interaction-design.org/encyclopedia/semiotics_and_human-computer_ interaction.html

NOTES _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________