CH26 - The Encyclopedia of Human-Computer Interaction,

1825

Chapter

26

Aesthetic Computing

by Paul A. Fishwick.

T

he phrase “Aesthetic Computing� while taken literally applies the philosophical area of aesthetics to the field of computing, and work in the area is broadly

defined as such; however, in my operational definition for the work we do in my research lab and in teaching, aesthetic computing is treated as embodied formal language. The purpose of aesthetic computing is to deliver knowledge and practice of formal languages using aesthetic products as a vehicle. Aesthetic Comput-

ing is founded on an increasing collection of literature on the role of the body in learning, specifically in mathematics. This foundation is then applied to the field of computing whose formal language elements are extensions of mathematics.

There are two questions that this new area raises: ff Q1: How can embodied cognition be situated within formal languages?

1826

Encyclopedia of Human-Computer Interaction

ff Q2: How can embodied cognition result in novel computer interfaces for formal languages?

Q1 surfaces a host of sub-questions revolving around theory, philosophy, and analysis. Asking this question raises issues of motivation: 1) Why am I interested in this topic? 2) How is the area of aesthetic computing built on top of embodied cognition and philosophy? 3) Who has worked in this area (e.g., the literature)?

Q1 is not enough, however. It is one matter to analyze and develop theory, but another to ask oneself, “How can this theory be transformed into practice?” That is the essence of Q2. What should we be doing, practicing, and creating to take embodied cognition of mathematics and computing to the next level? We need to build a new generation of human-computer interfaces that are informed by embodied principles and use these principles as design elements for interacting with formal languages. A potential, and vital, third question would revolve around the effects on such computer interfaces on learning via assessment and scientificallybased research methods. This represents an area that aesthetic computing needs to investigate; however, most work to date is based on theory construction and engineering the novel interfaces.

“The Aesthetic Computing Hypothesis is that given the embodied nature of cognition, we should realize this embodiment through novel human-computer interfaces for learning formal languages.”

Aesthetic Computing

1827

26.1 Context for area of aesthetic computing I pose two questions as a means to provide context for the area of aesthetic computing: 1) Why is the term “Aesthetic Computing” being treated as “Embodied Formal Languages?” and 2) What are “Embodied Formal Languages”? For the first question, we must revisit the roots of the word “aesthetics.” The original Greek definition of aesthetics, αἰσθητικός (aisthetikos), stems from another Greek word aisthanomai, meaning “I perceive, feel, sense.” At the core of aesthetics, then, lies the body, and its interactions in forming concepts and knowledge: a esthetics as embodiment. Aesthetics is, in breadth and depth, a much richer enterprise above this level (Kelly 1998), yet we maintain a view of aesthetics that is body-based, even though Diffey (1995) notes that the term ‘aesthetic’ has largely lost its perceptual sense except in the word ‘anaesthetic,’ but retains its senses of “beautiful’ and ‘artistic.’” As far as to why “Formal Languages” are used to characterize “Computing,” we note that the bulk of theory of automata and computing is situated within linguistics – although a subset of general linguistics that requires a formally well-defined specification and treatment.

Let us now consider the definitions of embodiment and formal language.

Embodiment suggests the perception/action feedback loop present when the body interacts with its environment. So, it seems clear that an embodied approach to anything would involve sensorimotor functions – using the mouse, keyboard, multi-touch displays as well as donning a head-mounted display or using a tactile feedback device. Human-Computer Interaction is chock-full of approaches that leverage such technologies. But, embodiment is a much deeper concept than sensory stimuli and physical manipulation. We have a sense of presence with certain advanced technologies such as multi-user virtual environments (i.e., achieving different types of presence, including social). We also have a sense of presence when reading a book since the book situates our “mind’s body” within the narrative (ref. “narrative psychology” in Beck et al. 2011). Thus, embodiment can be measured

1828

Encyclopedia of Human-Computer Interaction

objectively by hardware used to enable the senses, or subjectively through a presence instrument on the human subject. Embodiment should not be viewed as a rejection of abstraction, but rather as a complement to it (Devlin 2006).

Formal languages define a category of language that is artificial, such as a programming language. These languages stem from formal grammars which can be based on text, shapes, or diagrams. FORTRAN, Java, and Perl are examples of formal languages, but so are the eXtensible Markup Language (XML), Unified Modeling Language (UML), data structures, Morse code, and dynamic model structures used for simulation (Fishwick 1995, Fishwick 2007b). Formal languages are frequently specified using grammars such as the Backus-Naur Form (BNF) and need not be text-based. For example, one can have formal audiovisual languages and also graph grammars. All formal language structures can be defined hierarchically using levels of abstraction (e.g., 3 finite state machine levels governing an underlying set of ordinary differential equations, which in turn are translated into the programming language Java, and then further into byte code).

Languages, therefore, are frequently defined in long chains of specification and translation. Each language has its own target functionality, culture, and adherents. Ghezzi and Jazayeri (1997) provide general concepts of specification for programming languages.

26.2  Personal Experiences and Influences 26.2.1  Art It is easy to take the idea of embodied cognition for granted since it seems like something so natural—that the body plays a central role in cognition. However, an adherence to embodiment tends to change your worldview when looking at objects. As an amateur artist, I collected many posters and prints of historically

Aesthetic Computing

1829

well-known artists. In middle school, I was strongly influenced by Thomas Gainsborough’s work, in particular Figure 26.1.

Figure 26.1: Mr. and Mrs. Andrews, oil on canvas, Thomas Gainsborough, 1750. The National Gallery, London, UK. Copyright: pd (Public Domain (information that is common property and contains no original authorship)).

I imagined that with myself as an avatar , I could enter the painting, walk the wheat field, examine the trees, and engage in social discourse with Mr. and Mrs. Andrews. This led to a series of imaginary conversations and observations “in world.” The key point here is the “reading” of this work as a form of embodied experience. The Gainsborough painting was not a remote object of study for me, but rather an example of virtual reality, a time machine—an illusion that allowed me to immerse myself within the world of 18th century England. This approach is an example of Dewey’s art as experience (Dewey 1934) and relates to Grau’s (Grau 2004) argument about artists as the first virtual reality creators. The approach stresses that when we approach an object, we can interpret it dynamically via a

1830

Encyclopedia of Human-Computer Interaction

bodily simulation with all of the perceptual and motor-based actions that the body affords. This way of thinking and acting can be applied to all objects and media, including mathematics and computing.

26.2.2 Mathematics In elementary school, like hoards of other students throughout the world, I was taught the elements of arithmetic – its methods and laws, with many examples that were exercised using rote memorization and intense practice. Doing mathematics was highly action-based, but the action was limited to solving multiple problems over extended periods of time. After the basic elements of arithmetic came algebra. Let’s consider the following mathematical expression containing arithmetic with a sliver of algebra: “X = 2 * (3 + 4).”

We have all been subject to such mathematical objects as they are critical to an educated public. Learning all components of this equation was not easy – one had to understand the concept of a variable, operations of multiplication and addition, followed by the concept of a parenthetically-delimited group. Order of operations is also critical, as suggested by the group. So, for example, I can add 3 to 4 and then multiply by 2 to obtain 14, which was then set to X as an equivalence.

Certain laws of arithmetic were useful in transforming expressions such as this one. The Law of Distribution states that x(y + z) = xy + xz where x, y, and z are numbers, and the multiplication is implicit rather than being defined explicitly using * as in the above equation. The teacher would define the law of distribution and give us many useful examples as a means to reinforce our understanding of the law and how it can be employed in symbolic manipulation. Such patterns of equivalence drove a static pattern-matching type of approach to mathematics.

However, during the ensuing lessons, I found it convenient to create an arti-

Aesthetic Computing

1831

ficial method of solution that involved treating the numbers and symbols as physical objects. In mathematics education, this kind of process is termed reification (Sfard 1994) and is related to constructivism (Piaget 1950) and constructionism (Papert 1980), where students create their own knowledge through a combination of ideas and life experiences. I used a virtual manipulation of the above expression by representing the distributive law through analogy and metaphor: “Grab the “2” object, which when juxtaposed with the “*” operator, provides a biomechanical state where the “2” is pushed inward toward the group object defined within the parentheses “(...)”. The “2” is pushed gradually and then when it reaches the edge of the spatial boundary denoted by “(“, it moves through it to the other side and splits – in a biological fashion – into two clones that are attached to the “3” and to the “4,” respectively. This cloning activity results in the expression (2 * 3 + 2 * 4).

The sub-expressions 2 * 3 and 2 * 4 are evaluated through

further bodily activity. Pushing the 2 and 3 into the *, for example, results in multiplications. Similar reactions occur to perform the + operation last, as dictated by the learned order of operations. The result is then placed manually in a box with an X printed on it.”

Mathematics then, for me, had become akin to a full-body sport rather than simple operations requiring a collection of static text-based rules and patterns.

The virtual manipulations might involve other embodied activities, where I might have “launched the “ over a wall that bounds the parenthetical expression. While this is a personal experience, it is by no means unique, as Sfard observes in her dialogue with Thompson (Thompson and Sfard 1994), where she notes the propensity for similar mental imagery: “My work with mathematicians brought lots of further evidence that, indeed, the inner world of a mathematizing person may look very much like a material, populated with objects which wait to be combined together, decomposed, moved and tossed around.” Arzarello (2004) explains the difference between natural versus formal mathematical presentations, and sur-

1832

Encyclopedia of Human-Computer Interaction

faces the importance of gesture in using naturalistic explanations and interpretations in addition, or on the path, to the formal. The previous embodied description would be termed natural. Goldin and Kaput (1996) overview the effects of media on mathematical representation by noting “..changes in physical media that permit external representations to be action rather than display representations give these representations one characteristic of powerful internal representations.�

Hadamard (1996) studied mathematical thought which echoed similar cognitive processing. This action-based narrative on mathematical symbols was not limited to the distributive law for me. For example in an expression such as , something interesting happens when moving numbers through the equals sign. There is a virtual line or plane that intersects at a right angle to the . When a number such as is dragged through this vertical plane, the number flips its sign on the other side with a mirror-like effect, resulting in . The laws of commutativity and associativity have similar pseudo-physical, material, behaviors that can be used to understand and process arithmetic expressions.

The problem with my early experiences with embodied sense of symbol manipulation is that none of the books (or teachers) explained mathematics in this way, and I, and likely many others, were forced to keep these somewhat peculiar cinematic episodes to ourselves. Whether this type of thinking is common requires more scientific studies and reflection upon the nature of mathematics.

At the University of Florida, we have developed a web-based interactive tool that allows anyone to manipulate expressions in this fashion. We have also previously explored similar embodied representations involving a sense of presence in a virtual environment (Fishwick and Park 2008a).

My purpose of relaying this experience is to emphasize the importance of the body in understanding formal languages such as mathematics. Lakoff and Nunez (2001) presented a landmark compilation of mathematical metaphors that build on top of the philosophy of embodied cognition (Johnson 1987, Varela et al. 1991, Barsalou 2010). In particular, Johnson’s image schemata such as containment,

Aesthetic Computing

1833

attraction, and equilibrium were integral aspects of my arithmetic experience.

The literature in embodied thinking centers thought and knowledge on the body and is informed not only by areas such as conceptual metaphor (Lakoff and Johnson 1999, Lakoff and Johnson 2003), but also by subsequent empirical studies of the brain (Feldman and Narayanan 2004, Feldman 2006). Even more generally, language-based narratives appear to contain an embodied basis (Speer et al.

2007, Mar and Oatley 2008) defining natural language in terms of simulation.

Reading a story about grasping or running can result in a cognitive simulation of these events and activities, as if the reader had been physically active. Going back in time to when the Method of Loci flourished (Yates 1966), we note that the act of memorizing a set of facts was turned into a rich, embodied process rather than viewed as mere associative retrieval. The area of situated learning and cognition (Brown et al. 1989, Lave and Wenger 1991) meshes well with the embodied approach in terms of its goals and methods: learning by doing.

In closing the discussion of an embodied mathematics, we should note that the concepts of “action”, “interaction”, and “process” can be framed within standard mathematical notation containing explicit aspects of functional composition, dynamics, and procedure (i.e., embodied-types of thought). For example, the aesthetics of geometry and shape can be constructed generatively (Leyton 2001, Leyton 2006) and dynamically via Blum’s wave propagation-based medial axis (Leymarie 2006). We can also use mathematics to create a formal representation of mathematical metaphors (Guhe et al. 2009), thus making a loop: grounding metaphors on mathematical expressions, where the metaphors themselves are formally defined.

The embodied approach has profound implications for mathematics, and by extension for applied mathematics, and computing since computing is a direct outgrowth of mathematics, and formulas such as the one described earlier are common objects found in software “expressions.” If our thought is embodied, then:

1834

Encyclopedia of Human-Computer Interaction

ff We should investigate the variety of metaphors used within mathematics and computing, and also their origins and cultural associations. ff We should leverage the metaphorical, and embodied, substrate of language by creating new human-computer interfaces that reinforce and amplify this experience. ff We should bring to bear other disciplines for whom “the body” is a natural component, such as the arts and humanities (Slingerland 2008), thus forming new interdisciplinary collaborations that span the academy.

26.2.3 Programming The embodied approach was extended from mathematics into learning programming and data structures. Programming, in particular, is known to be rich in metaphor. Loops are just that: patterns of cyclic behavior – small objects moving around a closed path as these objects perform other tasks. Sequential behavior is sometimes a movement along a spatial path, and functions are machines that take product inputs and produce outputs. Papert (1980) in his explanation of the LOGO language reinforces the importance of embodiment in a term he calls “syntonicity“, where he notes “We have stressed the fact that using the Turtle as metaphorical carrier for the idea of angle connects it firmly to body geometry.” Petre and Blackwell (1999) performed studies on programmers, and results indicate metaphorical reasoning involving objects, motion and general embodied interaction. Metaphors such as these are not only present in all programming languages, but also in the theory of computation on which the theory of computing is based. For example, the Turing machine is an excellent example: a machine envisioned by Alan Turing in the 1930s consisting of a tape read/write head and an infinite tape. This metaphor may have been because of the extensive use of magnetic tape at the time. In the

Aesthetic Computing

1835

previous century, Charles Babbage used a “mill” in his computing engine. Interestingly, in the vast history of computing where these historical concepts are discussed (Ifrah 2002), most programming and computing was analog and embodied by definition and implementation. It is only relatively recently that the evolution from analog to digital has simultaneously sped up our computations, facilitated a computer revolution, but also disembodied our relations to computing.

26.2.4 Media Media theorists have provided a host of approaches in understanding the evolution of media. McLuhan (1964) places importance, not only on the message created through a modulated medium, but on the medium itself which affects the message.

McLuhan employs the example of a light bulb which he claims is a “medium without a message.” However, the light bulb can host a binary digit, and perhaps more in the case of multi-way switch bulbs in a means not unlike Morse code manipulated through signal lamps. Bolter and Grusin (2000) present a theory of media forms undergoing gradual alteration, generally technology-driven, causing us to examine issues of immediacy (seeing beyond the medium to the target signified) and hypermediacy (being aware and reflecting on the medium). New media studies place specific importance on materiality, the medium, and embodiment. Manovich (2002, p. 317), when he considers the “loop as a narrative engine,” with a loop being defined as a common programming structure enabling index-based iteration, asks “Can the loop be a new narrative form appropriate for the computer age?”

Popular media have significantly shaped my thought process underlying aesthetic computing. For example, Tron (Kallay 2011), which debuted in 1982, is noteworthy because it was created based on a highly innovative screenplay which included a large piece of software, namely an “operating system,” that could be experienced directly. Programs were bodies, and the operating system was composed of a city-like space with lighted, moving vehicles and interacting programs.

1836

Encyclopedia of Human-Computer Interaction

Tron is fairly unique in this way within the science fiction/fantasy genre. Other more recent cinematic offerings, while impressive and engaging, tend to ignore the “program.” For example, on Star Trek: The Next Generation, we were introduced to the Holodeck where one could experience an ultimate virtual reality with full sensory simulation. A user would stop at the outside of the Holodeck and say “Computer. Load Holodeck Program A-3” or some such phrase, and then the Holodeck would load this program and the user would enter. However, we never actually experienced the program itself – only its inputs and outputs. Similarly, in The Matrix, we have a rich embodied experience of human characters that , in reality, are stored inside of a network of fluid-filled pods.

Despite our familiarity and utility with text-based process descriptions, it is remarkable and ironic that a hyper-real environment such as the Matrix affording real-time synthetic interactions and simulacra would have to be programmed by strange-looking rivulets of green rain, which are not obvious to anyone, presumably except for the operator well trained in this postmodern descendant of cuneiform script. This semiotic condition presents a stark contrast: practically unlimited full-sensory simulation on one hand produced by the program, and what amounts to glorified typewriter symbols on the other defining the program itself. It is as if one provides you with a highly maneuverable hypersonic jet plane to fly with the caveat that you need to pilot the plane by tapping on a straight key to produce Morse code dots and dashes. One would expect that, just perhaps, the capabilities that form programs and data might avail themselves of the practically unlimited human-computer interface that the Matrix provides. Rotman (2000, p. 67) poses the question that forms this concern, “What if language is no longer confined to inscriptions on paper and chalkboards but becomes instead the creation of pixel arrangements on a computer screen?”

Aesthetic Computing

1837

26.3 Aesthetic Computing: Turning Computers Inside-Out Computers have shrunk in size, and increased in number, considerably over the past half-century. We are familiar with news stories about how ever smaller and thinner computers and software are now ubiquitous in our culture to the point where we carry or wear them in our daily routines. The decrease in size and increase in number creates a situation where computing effects most consumer products. For example, the digital video recorder enables time and place shifting for movies and television shows. What is just as interesting is exploring how computing affects us and our thinking. Turkle (2004) explains this psychological phenomenon and closes with the phrase “we are all computer people now.”

Turkle’s argument has significant ramifications for computing, and I would go one step further to suggest that the way in which our thinking is changing culturally surfaces deep abstract concepts in computing to us as we use these devices: from number, to information structure, to process. Digital watches and video recorders (DVRs) are good examples. Most digital watches are multi-function.

These watches contain the ability to act as a way to tell time, set a stop watch, or wake up to an alarm. To use the watch, you have to learn how to navigate a menu by repeatedly pressing a mode button. In each mode, there are sub-functions refining that mode’s interaction. This experience of mode-button pressing directly maps to a fundamental theoretical structure in computing called a finite state machine (Hopcroft et al. 2000). It is not just that the finite state machine is embedded within the watch’s silicon, but also that the human wearing the watch becomes aware of this virtual machine’s structure and its components through the experience of using the watch. The state machine changes how the wearer thinks, even though the wearer is probably unaware of the formal mathematical notation of a state machine. The watch’s software internals become embedded within our psychology and culture. A similar process occurs within most other household appliances such as the DVR, however, the state machines in DVRs are more complex

1838

Encyclopedia of Human-Computer Interaction

than in watches – yet to understand how to navigate the hierarchical menus, one has to become fully aware of a new type of thinking (Negroponte 1996). Effects of computing on thought (e.g., neo-millennial/digital native learning styles) have also been covered in the context of learning (Dieterle et al. 2007).

Experience with computing artifacts is a form of information representation, where the definition of “representation” is expanded as a form of interaction, rather than as a static object in the form of a sign. If the raw elements of computing – information, data, and software – are changing the way that we think and entering into our popular culture, it is natural to suggest that aesthetics of these raw elements can and should play a central role in computing. Aesthetics has evolved from the embodied, sensory, definition to a more comprehensive one offered by Kelly (1998), a “critical reflection on art, culture, and nature.” Aesthetics within computing results in new interaction modalities for computing artifacts such as formal languages. Given the preponderance of new ways to connect human with computer, there are many opportunities for creative representation. We categorize and study these new ways using the phrase aesthetic computing.

26.4 Why Aesthetic Computing?

Representation targets of aesthetic computing include terms such as data, information, software, and code. I use these terms somewhat interchangeably because of semantic overlaps. Data can be atomic or in the form of a structure. Code usually refers to software which encompasses both data as well as process. Information theory tells us that all of this is a form of information since information can be decoded as atomic, structural, or procedural. I prefer terms such as code, software, or information when referring to the “computing” part of aesthetic computing since these terms encompass broader categories of items that can be represented, whereas the term “data” in common parlance tends to denote non-procedural forms of information.

The argument for aesthetic computing involves emerging areas of computing which have changed:

Aesthetic Computing

1839

ff Our relationship to each other and to nature. These aspects include ubiquitous (Greenfield 2006, Gershenfeld et al. 2004) and pervasive computing, customization and personalization of interfaces, and the new modalities for human-nature interaction as mediated through computing (e.g., the virtual reality continuum spanning physical, virtual, and augmented reality). Shared and customized interfaces for information visualization (Viegas et al. 2007), code sharing (Reas and Fry 2007), assisted with “remix culture” (Lessig 2008) create a networked, customized (Pine 1999) representational space. ff Our thought patterns, allowing computing artifacts such as information and software to permeate our experience. Salomon (1990) makes an argument for computing changing thought, resulting in cognitive residues from human-computer interaction. These studies are consistent with Turkle (2005). ff The importance of experience in computing in human-computer interaction (HCI). Cockton (2011) and Hassenzahl (2011) describe the shift in HCI from efficiency, alone, to experience, and Löwgren (2011) emphasizes the importance of interaction – a core aspect of experience. The emphasis on experience is related conceptually to embodiment (Lakoff and Nunez 2001, Johnson 2007) as a basis for cognition. The relevance of aesthetics in HCI is discussed by Tractinsky et al. (2000) and Norman (2004). Dourish (2001) lays out a philosophical foundation for embodiment in HCI through its beginnings in phenomenology. ff Our need as computer scientists to interact more frequently with artists and designers since they represent the creative component of aesthetic inquiry, and so experience-based representations for the diffusing computing artifacts need to be studied

1840

Encyclopedia of Human-Computer Interaction

with the help of artist-scientist collaborations (Buxton 1988, Malina 2011).

26.5 History of the Aesthetic Computing Field I have been teaching a course in aesthetic computing since 2000, and information on the most recent course can be found in (Fishwick 2012). A preliminary paper was published on the concept (Fishwick 2002). A Dagstuhl seminar on Aesthetic Computing (Fishwick and Bertelsen 2002) was co-organized in Germany (Dagstuhl 2011) by myself, Roger Malina, and Christa Sommerer during the summer of 2002. This interaction resulted in several publications (Fishwick et al. 2005, Fishwick 2006, Fishwick, 2007a, Fishwick 2008b). Kelly et al. (2009) represents the most recent published workshop in the area. The use of the word “aesthetics” and “programs” can be found in several contexts, including Mohr (2011) and Nake (2009) who were early investigators in the aesthetics of interaction through the use of computer programs as a means of artistic expression. Knuth (1992) developed literate programming and made note of the importance of aesthetics in programming. Knuth’s interest in aesthetics went beyond the purely cognitive, and included artistic forms of typography and layout design for programs. For Knuth, it would seem that computing was an embodied experience.

Aesthetic computing is unusual in that aesthetics is intended to be applied to computing rather than in the inverse direction: using computing to create artistic products. Examples of aesthetic computing, therefore, capture a kind of “boomerang effect” where elements of computer graphics, ubiquitous computing, and mixed reality interfaces can be used to interactively represent that which formed these technologies – namely the information and software.

In terms of academic curricula, Aesthetic Computing has been taught for a decade at the University of Florida in the form of two classes, which are usually combined: CAP 4403 (undergraduate) and CAP 6402 (graduate). The combined

Aesthetic Computing

1841

classes began as part of the Digital Arts & Sciences (DAS) programs (Fishwick 2012) designed and developed to connect computing with the arts. The class has undergone several stages since 2000: ff (2000-2005) Representational alternatives to software artifacts – from numbers and expressions, to data structures and programs.

There was one physical project, with the other two projects resulting in digital representations. The physical product was exhibited in several gallery areas on and off campus allowing passers-by to comment and explore. ff (2006-2009) Alternative representations for mass media and communications. This emphasis required students to employ representational creativity, but with the idea of starting with a contemporary news story and then mining this issue for the software artifacts in the story that were to be represented. The physical project was eventually dropped since many of the computer science students in the mid to later years had minimal design and art backgrounds. ff (2010-2011) Representation using web mining and APIs. This was an effort to create more automation in the representational process with students finding sources of information and then, mostly using APIs, to translate this information into creative representations. Most students use data as their information, but others used more complex web structures (e.g. XML) as sources. ff (2012) A focus on representation of data structures, mathematical models, and dynamic models and programs. The end product is either an interactive game or video production whose goal is to facilitate education of computing concepts for early-age groups and

1842

Encyclopedia of Human-Computer Interaction

non-computing specialists. This is the current incarnation of the class (Fishwick 2012).

We use the term aesthetics in the spirit of Kelly’s definition, but also extend the concept of “critical inquiry” to include the creative aspect of design and art. This is only natural, for engaging in critical inquiry presupposes and requires the creative act. Studies in aesthetics are numerous (Audi 1999, Kivy 2004) often with underlying attempts to find universal attributes of beauty (Scruton 2011). My view on aesthetics is one that focuses on that which is generated as a result of cultural inquiry, which is to say the vast diversity of design and art forms. This “aesthetics as diversity” approach is similar in spirit to Hogarth (Burke 1943) with the associated phrase, “unity in variety.”

26.6 Toward Software as Embodied Experience 26.6.1 Introduction Partial justification for the use of embodiment as a form of representation is based on educational learning styles (Dede 2005). Also, our ongoing research indicates a significant correlation between presence and memory in a virtual environment (Fishwick et al. 2010) with results currently in the journal submission phase. Recent mixed reality memory studies such as (Ikei and Ota 2008) indicate positive effects on memory in an augmented environment. Instruments and studies on memory performance within virtual environments are being continually refined and investigated. Parsons and Rizzo (2008) introduce a test of validity for a virtual environment cognitive instrument called VRCPAT. Johnson and Adamo-Villani (2010) note significant effects of immersion on short term spatial memory.

Embodied interaction with technology provides us with an understanding of internal logic, software, and process usually through pure experience. For example, we learn the state machine of a DVR through repetitive DVR use. While a large

Aesthetic Computing

1843

population may require this learning, not everyone may be required to take representation to the next step: from interaction to reflection and reification. The latter steps, however, have potential utility in entertainment (the arts, games) as well as in education.

26.6.2 Audiovisual Explorations: Steampunk Obesity Machine Let’s consider one such artifact, which is defined by a system dynamics model found in systems science and simulation. Figures 2 and 3 are two different representations of a System Dynamics flow graph (Forrester 1991) capturing the temporal nature of human metabolism.

Figure 26.2: A System Dynamics flow graph with two levels (i.e., stocks) and three rates.

Copyright © Inderscience Publishing. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

The diagram in Figure 26.2 represents a virtual machine based on the analogy of fluid flow. Fluid starts from source node (left-most “cloud” icon) and proceeds

1844

Encyclopedia of Human-Computer Interaction

to flow through a system of levels separated by rates to a sink node (right-most cloud icon). More generally, the fluid flow can be construed as a kinetic energy flow since fluid velocity is the dominant flow variable. At the start of the machine, at the left, fluids pour into metabolism and food intake to suggest that the more energy, the higher the Fitness Level, but also the higher the Weight. The rate variable, Metabolism, is proportional to a functional combination of Fitness Level, Exercise, and Nutrition. The nature of this precise formula is not present in the model since the model is an abstract representation of the dynamics. The solid curve arrows reflect fluid flow through the system, and the dashed curve arrows reflect control settings to change the rates on the valves. Figure 26.2 is a hypothetical example, and is not put forth as an accurate or valid simulation model of nutrition, but rather to demonstrate that similar diagrammatic models are widely used in science and engineering. These types of models were originally implemented as physical, analog computers although their more frequent existence today is as digital models with a diagrammatic front end authoring capability. The MONIAC, or “Phillips Machine,” is one such example (Swade 2000, Ryder 2009) from the analog computing era.

Figure 26.3 shows the same model which is a synthetic rendition of Figure 26.2, reified using a “steampunk machine” since its structure is reminiscent of the cyberpunk aesthetic that continues to be popular since its inception in Gibson’s work (Cavallaro 2001). Steampunk culture has connotations of “reclaiming tech for the masses” (Grossman 2009). Water is pumped using steam-power underneath the wooden floor. This water shoots out of two brass orifices that represent the two valve-icons in Figure 26.2. Water filled glass containers represent the level quantities, and wood/brass control rods connect everything together as in Figure 26.2. The human avatar on the left is demonstrating the machine in action to us, or we may become the avatar. The natural question is why anyone might want to construct such a machine when Figure 26.2 might do. For the answer to this, we have additional questions to ask, with possible use-cases.

Aesthetic Computing

1845

Figure 26.3: A steampunk obesity machine isomorphic to Figure 26.2.

Copyright © Inderscience Publishing. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

Figure 26.2, and the equations that map to this diagram, are most often used by scientists familiar with the system dynamics method. It is unlikely that these scientists have any interest in structures such as Figure 26.3 mainly because they are comfortable and familiar with more formal representations. However, the vast majority of the population may require additional motivation if they are to understand, and be motivated or influenced by, the more formal representations.

Therefore, the machine for Figure 26.3 is appropriate for education and entertainment. It is easy to imagine the machine in Figure 26.3 being engaging especially with game-like features that required certain goals such as stabilizing the water level in the Weight container.

1846

Encyclopedia of Human-Computer Interaction

26.6.3 Visual Representations of Data There are numerous additional examples of artworks that, if used as guidance, can lead to aesthetic computing products useful for education. The vast majority of examples are the encoding and presentation of data rather than of program or model. It is logical given that data repositories and accessibility are expanding rapidly and that they represent the simplest and easiest to grasp forms of information. Consider the model of a single number shown in Figure 26.4.

Figure 26.4: The relative size of the U.S. debt if it reaches 15 trillion dollars. The large rectangular block represents stacks of one hundred dollar bills.

Copyright © Oto Godfrey. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

This encoding of number as a stack of one hundred dollar bills is given context by familiar objects whose size is known through pictures or experience (e.g., the Statue of Liberty, a football field, a truck). One might take this same approach to representing other analog representations of monetary amounts through choosing different familiar objects. A participant’s engagement can have both artistic and mathematical consequences. For example, we can imagine performing opera-

Aesthetic Computing

1847

tions on numbers in this type of representation much as we have done manually in the past with quipus and abaci.

Consider Huff’s prime number series (Huff 2006) with two example encodings of prime factors shown in Figure 26.5.

Copyright © Kenneth A. Huff. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

1848

Encyclopedia of Human-Computer Interaction

Copyright © Kenneth A. Huff. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

Figure 26.5 A-B: Prime number factorization encodings (EPF: 2003:V:A:997141 and 2000.24).

The two encodings in Figure 26.5 are pieces of fine art, but could also be potentially used to motivate students to appreciate prime factorization through puzzle-

Aesthetic Computing

1849

making. For example, consider where one might provide to someone a visual encoded integer and then ask that person to identify the number and factors. Figure 26.6 shows two additional examples of information presence: Levin’s infoviz graffiti for data, and Living Light. The graffiti is a deliberate mechanism for surfacing numbers of societal relevance in public places. Living Light is a permanent outdoor pavilion in Seoul, South Korea. The pavilion’s purpose is to allow spectators to visualize environment levels such as air quality. As pervasive computing extends into the future, most flat surfaces become display surfaces opening up numerous possibilities for bringing information into our daily lives. Figure 26.7 shows a model of a city which is turned into a computer program-like artifact, or automaton, whose output is a musical score.

Copyright © Golan Levin. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

1850

Encyclopedia of Human-Computer Interaction

Copyright © David Benjamin and Soo-In Yang. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

Figure 26.6 A-B: Leftmost: Infoviz Graffiti/Adjustable Pie-Chart Stencil by Golan Levin. Rightmost: Living Light by David Benjamin and Soo-In Yang.

Figure 26.7: Pianola City Music.

Copyright © Akko Goldenbeld. All Rights Reserved. Reproduced with permission. See section “Exceptions” in the copyright terms.

Aesthetic Computing

1851

26.6.4 Textual Representations The examples so far have been mainly visual; however, it is often desirable to use thinking similar to that described earlier for mathematical expressions, but to extend this to software for example. Emerging areas in the humanities such as software studies (Fuller 2008) and critical code studies (Marino 2006) situate the need for studying formal languages using some dimension of hermeneutics.

These areas also provide opportunity for creation of new human-computer interfaces. For example, we may treat software as a full hyper-mediated structure (Roth et al. 1994, Anderson et al. 2000). One can then, through an embodied approach facilitated by link interaction, treat formal language-based constructs as hypermedia.

26.7 Art and Design as Creative Influences for Embodied Formal Languages The provided examples including Figures 3 through 7 are related to aesthetic computing in different ways. Since aesthetic computing is embodied in formal languages with an educational goal as a final end product, I will overview how these examples might achieve that goal. Table 26.1 contains 5 columns: column 1 refers to a previously described image or product; column 2 is the original medium; column 3 is a hypothesized goal for the last 5 rows (i.e., since the original intention is not known but assumed); column 4 is an example repurposing of the original product for a formal language goal (column 5). Let’s consider the 3rd row. The product has been designed to a highly compelling and attractive display of the national debt. This creative use can be recast as a new way to learn number sense. The formal language products are only examples and have not been constructed by anyone, however, the original art and designs are dually inspirational – for their original goal or purpose, and for a form that leverages their embodied characteristics for the purpose of formal language instruction.

1852

Example

Encyclopedia of Human-Computer Interaction

Origi-

Aesthetic Goal

Formal

Formal

nal

(hypothesized)

Language

Language

Prod-

Product

Goal

uct

(example)

Personal

Typo-

To illustrate ele-

Game with

To teach

Experience

graphic

gance of the math-

moving op-

laws of arith-

(Arithmetic)

Image

ematical form

erators and

metic

operands Steampunk

Raster

To create steam-

Video il-

To teach

Obesity

Im-

punk genre-relat-

lustrating

System

Machine

age  art

ed imagery

functional

Dynamics

(Figure 26.2)

work

mechanism

Methodology

and control US National

Raster

To illustrate the

A tactile set

To teach

Debt

Image

magnitude of the

of blocks

number

(Figure 26.3)

art work

US debt using scale

and objects

sense

Prime Num-

Raster

To celebrate or-

An adven-

To teach

ber Factor-

Image

ganic forms using

ture game

about prime

ization (Fig-

art work

prime number

using en-

numbers and

encoding

codings as

factorization

ure 26.4)

3D puzzles

1853

Aesthetic Computing

Infoviz Graf-

Graf-

To present societal

An alter-

To teachÂ

fiti (Figure

fiti and

information to the

nate real-

concept of

26.5a)

Tem-

public

ity game

percentage

plate in

(hunting for

Outdoor

graffiti)

location Living Light

Outdoor

To present

A kinetic

To teach

(Figure

sculp-

environmental

sculpture

data struc-

26.5b)

ture

information to the

tures

public Pianola City

Indoor

To explore an

Indoor ki-

To teach

Music

kinetic

architecture-

netic object

concept of a

(Figure 26.6)

art work

music interface

data search via sound

Hyper-medi-

To represent cul-

Hyper-

To encourage

tural knowledge

mediated

learning of

compu-

software/

how to code

tational

code

Web-

ated software based engineering

literature Table 26.1:  An aesthetic transformation to formal language learning objectives.

Table 26.1 portrays aesthetic computing through repurposing existing art works, but this procedure is optional. Formal language-based products that capture the essence of embodied interaction can be designed directly from initial design, to detailed design, and onto an implementation. The Steampunk Obesity Machine (Table 26.1, Row 2) is a case in point. Even though a poster board image (Figure

1854

Encyclopedia of Human-Computer Interaction

26.3) was part of a curated art exhibit (Harn 2011), the image was meant as a preliminary design for a virtual machine to teach System Dynamics concepts. The machine has not yet been constructed.

26.8 Embodied Computing using Serious Gaming The discussion of aesthetic computing and the interpretation of it via embodied formal language would be incomplete without reference to video and console game cultures. Two examples are illustrative: logic circuits in the game Minecraft (2011) and the game called Code Hero (PrimerLabs 2011). Minecraft is a “block game” where players move around a space and build blocks using a mining metaphor. Some of the procedural capabilities within the game have engaged members of the community to create basic circuits, leading up to full-fledged computers out of the logic circuitry. Since Minecraft is highly interactive, and invokes a sense of presence to boot, this type of hacking is consistent with the concepts in aesthetic computing: players are working together to form circuits through embodied interaction. Primer Labs recently created a game called Code Hero where the play learns a programming scripting language such as Javascript. It is the means for this pedagogy, however, which places it squarely in the embodied realm: a player has a gun that “shoots code” at a target object, thus causing that object to react to the code. This is, in actuality, a reified form of data flow in a manner similar to the capabilities within lambda calculus and languages based on that formalism such as Lisp (e.g., consider the “map” functions where a function can accept another function as input and then apply that function to arguments, producing output). Figure 26.8 shows a Minecraft arithmetic logic unit (ALU) described by Ganapati (2010), and Figure 26.9 shows a snapshot from Code Hero.

Aesthetic Computing

1855

Figure 26.8: Arithmetic Logic Unit built with “redstone” in an immersive play space using the Minecraft game engine.

Copyright © . All Rights Reserved. Used without permission under the Fair Use Doctrine (as permission could not be obtained). See the “Exceptions” section (and subsection “allRightsReserved-UsedWithoutPermission”) on the page copyright notice.

1856

Encyclopedia of Human-Computer Interaction

Figure 26.9: The Code Hero Game: talking to the Ada Lovelace avatar prior to entering a space to learn how to script code by shooting scripts at objects.

Copyright © Primer Labs. All Rights Reserved. Used without permission under the Fair Use Doctrine (as permission could not be obtained). See the “Exceptions” section (and subsection “allRightsReservedUsedWithoutPermission”) on the page copyright notice.

26.9 Collaborative Roles, Usability, and Experience Aesthetic Computing begins with a formal language construct such as a number, data, model, or software. Then the challenge is to represent this construct through embodiment. We noted that “embodiment” can be as simple as pure reification

Aesthetic Computing

1857

without representation of existing objects when we demonstrated the ability to grab hold of numbers and move them toward operators. However, reification can also suggest object representation as in Figures 4 through 7. I need to address the “who” and “why” aspects of aesthetic computing.

First, who is going to be creating these representations? In the case of collaboration, I recommend teams of humanist scholars, artists, and computer scientists. Humanist scholars bring to bear different philosophies and theories which can help shape the resulting representation. The artist has the creative perspective and tools to create the representation, and the computer scientist can serve two roles: to help construct tools used by the humanist and artist in the extraction of information and in enabling the interaction that ensues through externalizing embodiment in the human-computer interface.

Second, who is going to use the representations? Students in my aesthetic computing class are often initially confused why one would construct anything but diagrams. This confusion is expected, but we must be careful when defining usability: usable for whom and for what purpose? We need to identify 1) the goal of the representation, and 2) the end target users. Goals for the embodied representations are education, arts, and entertainment (e.g., cinema, visual and performing arts, fiction). Target users may be any grade level in school or some segment of the general public. From a psychological perspective, a broad view of “usability” can encompass user goals including: increased valence, motivation, and attitudinal change, as well as improved short or long term memory. Mathematicians and computer scientists are not the target, as these populations are adept at using existing notations. Aesthetic Computing is less stressed on information extraction and more on the use of entertainment, arts, and humanities on formal languages with the largest practical effects being in education. Thus the target users are formal and informal learners of all elements of formal languagebased instruction (e.g., mathematics, computer science).

1858

Encyclopedia of Human-Computer Interaction

The roles of participants in aesthetic computing will likely be different given the interests of each party. For the computer scientist, for example, Figure 26.5 serves as a design template for the creation of special effects and interactive games for the purpose of expressing elements of prime numbers and the factorization process into these numbers. The artist’s work is a medium through which this aspect of formal language is creatively expressed. The goals of the artist and computer scientist are clearly different, but the means (i.e., representations of prime numbers) are common. This difference in ends, with similar means, plays out in the other examples. For instance, Perl poetry (i.e., poetry created using the programming language, Perl) may be an aesthetic product to the writer – a valid end in itself. To the computer scientist, this product represents a medium in which to express a different end – the formal language “message.” Therefore, aesthetic computing by its arrangement of words comprising this phrase is focused on computing – the learning of formal languages. However, aesthetic products play a key role in this learning activity and allow for the artist, scholar, and computer scientist to collaborate with different intentions and goals.

Other areas related to aesthetic computing are information visualization (Card et al. 1999, Ward et al. 2010), and software visualization (Eades and Zhang 1996, Stasko et al. 1998, Zhang 2007, Diehl 2007); however, the goals of these areas are generally quite different than for aesthetic computing. In information visualization, the goal is efficient communication of data and information, whereas for aesthetic computing, the goal is education through highly embodied, and interactive, aesthetic products in the forms of art and entertainment. As such, Aesthetic Computing fosters a deeper experience than building representations meant for immediate consumption (e.g., newspaper diagrams and maps). Readers will observe that the use of metaphor is rich within the high level interactions with computers. We are an interface culture (Johnson 1997). However, the metaphors used on the “desktop,” for instance, have not yet made their way into the core of mathematics and computing. Efforts such as computational thinking

Aesthetic Computing

1859

(Wing 2006) are a move in the right direction.

Laurel (1991) presciently captures a prerequisite for aesthetic computing in her “Computing as Theatre.” However, Laurel was mainly constructing a case for human-computer interaction as a complex theatrical production, involving many of the same elements found in theatre. The use of computing, and its associated interaction phenomena, are like theatre. However, what we find is that as we break open the lid of the black box containing the atomic elements of normally hidden data, formulas, code, and models is that computing is theatre all the way down.

26.10 Toward a Method of Aesthetic Computing While it is interesting to pose ideas and directions, a procedural method is something that can help to forge a discipline even if only as a general guide. Fishwick (2007a) was an initial attempt at this process with a small example of code that was represented as a collection of rooms in a building, complete with a partial narrative for context.

Figure 26.10 serves as a basis for describing the approach used in (Fishwick 2012):

Figure 26.10: Aesthetic Computing Method.

Courtesy of Paul Fishwick. Copyright status: Unknown (pending investigation). See section “Exceptions” in the copyright terms.

We begin (in the top left of Figure 26.10) with a formal language construct that is to be conveyed to non-specialists in mathematics and computing with the goal

1860

Encyclopedia of Human-Computer Interaction

of broadening the exposure of computing concepts. The asterisks denote current emphases in (Fishwick 2012). Target users will depend on the type of formal language. If the goal is number sense, and the numbers are fairly simple, we may be looking at elementary school children. If the formal language is simple algebraic formulas, we may be looking at 8th grade mathematics. More complex mathematical and computing structures may require higher grades, including universities and in postgraduate, informal learning contexts. One of the desirable outcomes of this approach to representation, though, is to expose very young children to seemingly complex data structures and programs by using games and video as motivational media. I expect that the approaches may serve as 1) scaffolding for later, more traditional, instruction and notations, and 2) secondary devices (e.g., puzzles) to reemphasize concepts that some learners find difficult using standard notations. The goal is not to eliminate standard notations as this would be counterproductive. Representation is divided, in Figure 26.10, into two components: methods that achieve representation and technologies that support embodiment.

End products that emphasize, or surface, embodiment can vary. A good piece of fiction can create a strong sense of presence and virtual embodiment, whereas a weak interactive game may be left ignored if not well designed.

26.11 New Connections A primary goal of mine in fostering aesthetic computing is to link disciplines – especially those in computing to the humanities and arts. As evidenced by designers and humanist scholars, artifacts such as “code” and “data” are now being interpreted and recreated. There are many reasons for this. Perhaps, the ubiquitous computing trend is the most significant driver – software is everywhere and so, by natural extension, cultural. I welcome the artists, designers, and humanists into the “formal languages” space and hope that through collaborations and interdisciplinary discussions and critique that we might re-humanize core elements of computing, and perhaps even mathematics.

Aesthetic Computing

1861

26.12  Disciplinary and Technical Challenges The area of aesthetic computing is not without its challenges. The goal is to leverage embodiment theories toward building new computer-based interfaces for learning formal languages. Disciplines that I have covered have sub-areas that are all targeted toward this goal, but significant challenges remain for each area: ff Mathematics: the literature in mathematics education, and in the application of cognitive linguistics within mathematics learning, is well-founded and supports aesthetic computing. This body of knowledge, however, is more focused on analysis and theory construction rather than, through analogy, building new interfaces in mathematics education to take full advantage of the embodiment theories through realization. Some efforts in virtual manipulatives are a good start, but this work should expand to employ the next generation of interface capabilities that stress embodiment (e.g., multitouch displays, body tracking, mixed/virtual reality technology). ff Computing: the literature in computing education provides fairly easy-to-use interfaces for seeing the results of executing programs; however, the programs are often limited to the canonical alphanumeric notation with all of the human interaction being in the program execution rather than inside the program. Efforts at software visualization move in the direction required by aesthetic computing, and yet, there is a much wider set of possibilities for representation if the goal is to teach non-specialists especially through immersion, situated learning, and interactive games. Diagrams are fine for communication, but if the goal is to explore deeply embodied approaches for learning, additional media and newer interfaces – as recommended for mathematics education – should be more thoroughly investigated.

1862

Encyclopedia of Human-Computer Interaction

ff Humanities: the work in cognitive linguistics, and resulting embodiment theories, ground the work in aesthetic computing, but as with the work in the philosophy of mathematics learning, there is little corresponding effort in realizing these theories in a humancomputer interface. Conversely, the work in cultural theory production is recently targeting “code” specifically as a new type of literacy (e.g., critical code studies). And yet, this production tends to avoid linguistic analysis and instead focuses on socio-historical analysis. New, embodied, interfaces for code can build off of the scholarly analysis, but these interfaces should also be informed by key facts of semiotics (e.g., analogy and metaphor) which lie at the foundation of formal languages. There appears to be a bias toward textual notation rather than exploring broader forms of “embodied literacy,” which would include textual notation as one dimension. ff Art & Design: works of art have traditionally treated formal languages as “black boxes,” tools needed to create art or designs. Unlike in the humanities, where code has become subject material, in art, code tends to be treated purely as a tool, whether embedded in package or programmed via a text-based development environment. The only exception to this observation would be in typography within graphic design, where the subject material is the text. More explorations are required so that formal languages become the active subjects of artwork.

Each one of these four areas has some common challenges. Observing that analogy is the engine of metaphor in scientific practice, aesthetic computing products can be created with an increased attention to analogy. Another observation is that with the exception of Art & Design, there is a classical focus on alphabetic notation. Such notation serves us well and has enriched our formal languages.

However, there are other types of notations that exercise more of the body’s sensorimotor functions. Diagrams are a good place to start in seeing this transition

Aesthetic Computing

1863

since with diagrams spatial metaphors for text-based notations abound, but we should not limit our embodied explorations to diagrams.

A primary aesthetic computing challenge is technological. It is still relatively expensive to build new interfaces based on the types of products described by the figures previously shown. “3D modeling” as a real-time technical interface capability is nowhere near the futuristic landscapes of Tron, the Matrix, and the Holodeck. Modeling and animating in three dimensions remains a major challenge compared with diagrammatic approaches, and even diagram-based software modeling (e.g., model-driven architecture) struggles for acceptance in the marketplace of software engineering solutions because of the relative ease of using textual symbols. Human-computer interaction solutions are expanding in scope and capability, but we still are a long way from being able to easily and inexpensively become embodied in our formal language constructs.

26.13 Summary: The Argument for Embodied Formal Language This chapter began with personal experiences in mathematics and then moved on to discussions of embodied cognition, along with some examples of where aesthetic computing could be applied. The area of aesthetic computing rests primarily on the foundation of embodiment – whether we believe that our bodily interactions form our thought. This assumption of embodiment runs deep in philosophy. We all recognize that we have body and mind, and most would agree that the latter is the effect of the former. It is only fairly recently, though, that literature has arisen to indicate a strong relationship to the extent that thought itself, even for abstract objects, is embodied. The theory that undergirds embodiment is compelling, but we have the nagging question about how this theory can change what we do and how we act. If I imagine that I am imagining grabbing and pushing a number through a pseudo-biological membrane during arithmetical operations, I want to build a human-computer interface that reinforces this mental sequence by infusing theory into practice. This

1864

Encyclopedia of Human-Computer Interaction

perceived need matches the aesthetic computing hypothesis stated at the start of the chapter: Given the embodied nature of cognition, we should realize this embodiment through novel human-computer interfaces for learning formal languages.

Achieving this realization involves a more thorough understanding of the interplay among disciplines and how embodiment theories in those disciplines interact and connect. The realization also requires a host of newer “virtuality continuum” technologies that allow us to achieve what Biocca refers to as degrees of progressive embodiment (Biocca 1997). The technologies and their characteristics are overviewed for virtual reality by Sherman and Craig (2002), and by Bowman et al. (2004), and for augmented reality by Bimber and Raskar (2005).

26.14 Where to Learn More about Aesthetic Computing For a thorough understanding of computing as a discipline, and its artifacts which are represented in aesthetic computing, the 1998 ACM Computing Classification System (CCS 1998) serves as a good starting point. Even though my treatment of aesthetics is based on its original, perceptual definition, Kelly (1998) collects that which erupted from this kernel in philosophy and the arts in four volumes. Even though information visualization is centered on efficient communication (e.g., reading the equivalent of a diagram in a newspaper), some archives such as infosthetics curated by Vande Moere (2011) are broader and contain a wide variety of potential use cases – from efficient communication to experience, education, and play. For text-based representations, HASTAC (2011) serves as a high level repository of bloggers and projects, many of which are associated with digital humanities. The reader is encouraged to review articles cited in this chapter.

Aesthetic Computing

1865

26.15 Acknowledgments I would like to first acknowledge all individuals who have participated in this journey from its inception, including my colleagues in the arts, natural and social sciences, computer science, and mathematics. Students in my Aesthetic Computing class have had to put up with these ideas, and they have produced wonderful products that I could never have imagined. Thanks to the following colleagues who took time to make very good critical remarks on earlier forms of this manuscript: Sophia Acord (University of Florida), Michael Kelly (University of North Carolina at Charlotte), Mads Søgaard (Interaction Design), and Kang Zhang (University of Texas at Dallas). I take responsibility for any errors and omissions.

26.16 Commentary by Michael Kelly How to cite this commentary in your report

Michael Kelly

© Michael Kelly

1866

Encyclopedia of Human-Computer Interaction

I am a Professor of Philosophy at the University of North Carolina at Charlotte, and Editor-in-Chief of the Encyclopedia of Aesthetics (Oxford UP, 1998). A new expanded, revised, second edition of the Encyclopedia is forthcoming, and will include entries on aesthetic computing. I organized and participated in a panel discussion on information aesthetics at SIGGRAPH in 2009. And I or...

Michael Kelly Michael Kelly is a member of The Interaction Design Foundation

“The masters of information have forgotten about poetry, where words may have a meaning quite different from what the lexicon says, where the metaphoric spark is always one jump ahead of the decoding function.”

-- J. M. Coetzee, Diary of a Bad Year [1]

26.16.1 Fishwick’s Encyclopedia Entry on Aesthetic Computing Paul Fishwick has a well-developed, impressive research and pedagogical platform at the University of Florida from which he’s been exploring one of the particular versions of aesthetic computing, although, being involved since its inception, he also has a general sense of the field. In fact, there would hardly be such a field without him. The question I want to ask is how much his particular projects are influencing his general conception of aesthetic computing and whether he’s achieved the appropriate, if difficult editorial balance here [2].

Fishwick begins with a pedagogical focus: personal experiences in mathematics that led to discoveries and explorations of embodied cognition. In particular,

Aesthetic Computing

1867

he analyzes “the aesthetic transformation to formal language,” using the concept of embodied knowledge, understood as a perception-action feedback loop based on the idea that embodiment is a form of representation, not just an insignificant step in the process of a strictly cognitive mode of representation. From there he argues, with a rich set of projects, that “The purpose of aesthetic computing is to deliver knowledge and practice of formal languages using aesthetic products as a vehicle.” In short, the examples of teaching abstract mathematical concepts that led Fishwick to aesthetic computing have continued to have structural as well as thematic roles throughout his entry and have largely determined his conception of the field. The result is an excellent but partial picture of aesthetic computing that, if taken for the whole, would be misleading.

Fishwick acknowledges a broader conception of aesthetic computing in the “Why Aesthetic Computing?” section (26.4). But he does not adequately clarify it or show concretely how it informs or otherwise relates to his research. Has he perhaps changed his view of aesthetic computing over the last decade? Back in 2006, he made a number of statements about aesthetics that point to a conception of aesthetic computing broader than what is generally evident in his Encyclopedia entry (Fishwick 2006). For example, he says that aesthetics reaches “beyond classic concepts such as symmetry and invariance” and encompasses “the wide range of aesthetic definitions and categories normally associated with making art.” Yet now he seems to limit aesthetic computing to often classic concepts specifically relevant to his projects. Quoting the Preface to the Encyclopedia of Aesthetics, he embraces the idea of aesthetics as “the philosophical analysis of the beliefs, concepts, and theories implicit in the creation, experience, interpretation, or critique of art.” However, when he references this same idea in the Encyclopedia entry, he seems to pull back from its full implications for aesthetic computing. Fishwick also says earlier that aesthetics has logical as well as material aspects, so it can extend to computing as well as art. Taking discrete mathematics as an example, he claims that aesthetic computing encompasses notions of formal language, ge-

1868

Encyclopedia of Human-Computer Interaction

ometry, and topology, and from such claims he concludes that aesthetic computing corresponds naturally with mathematical formalism. In his current research and Encyclopedia entry, Fishwick focuses mostly on this last sense of aesthetics and develops, albeit very well, only the narrower view of aesthetic computing it implies.

Yet, the case for a broader conception of aesthetic computing can be made from within Fishwick’s own projects because he argues that aesthetic computing rests primarily on the foundation of embodiment, which is itself a very important research topic in aesthetics and a number of disciplines (e.g., cognitive psychology, affective computing, philosophy of mind, etc.). But even here Fishwick’s sense of embodiment is mostly cognitive and pedagogical because it’s linked principally to formal languages. This may seem like an appropriate link because computing is so much about formal languages. But isn’t the whole point of aesthetic computing to develop and sustain a richer conception of computing? With a richer conception in mind, in effect, the art historian and theorist Caroline A. Jones offers a more art-centric and aesthetics-informed account of embodiment that is focused on the impact of computerized technology on the human body, on the “techno-human.” [3]. She begins by arguing that the best way for the critique of our techno-culture to keep pace with “the speed of technological innovation” is “to take up these technologies in the service of aesthetics,” which provides “a site for questioning” how our “bodies are interacting with technologies at the present moment.” Aesthetics provides contemplative space for such a critique because it “buys us time and space” to encounter and reflect “on embodied experience in an ever more technologized world.” That is, aesthetics sets up critique within computing to examine how human-computer interactions impact our bodies. The goal of such critique is not merely to understand all the computer-generated bodily interactions that have been experienced already but to explore which ones could be experienced, and, moreover, which ones we would prefer to experience going forward. In the end, a major advantage of Jones’s account of embodiment is that she

Aesthetic Computing

1869

makes it clear that this kind of critical thinking internal to computing already has a name with a long tradition: aesthetics. By making the links among embodiment, computing, and aesthetics explicit, she offers broader conceptions of computing and aesthetic computing alike.

Jones’s account of embodiment, which is explored by a number of artists and theorists involved in the Sensorium exhibit or catalog, also dovetails well with the aesthetics of participatory art practices that have developed recently in contemporary art, which would also help to broaden aesthetic computing [4]. Participatory art is, in brief, the convergence of various art forms that emerged in avant-garde modernism or contemporary art: interactive art, installation art, performance art, conceptual art, new media art, public art, socially engaged art, etc. Such convergence has altered the aesthetics of contemporary art in ways (e.g., agency is collective, form is participatory, interactions are transformative) that resonate in computing, too, as it becomes ever more ubiquitous, participatory, collaborative, social, and interactive. Since a central concern about aesthetic computing is how aesthetics is relevant to computing, it would help this cause to examine the most recent developments in the aesthetics of contemporary art. This does not mean that the aesthetics of classical or modern art are not relevant, but since participatory art is emerging in part because of the impact that computing has already had on the production and reception of contemporary art, participatory art is an excellent area to explore while developing aesthetic computing.

In addition to the editorial imbalance, my other principal concern with Fishwick’s Encyclopedia entry is that he regards aesthetics primarily as a means (“a vehicle”): “aesthetic computing is embodied formal language with an educational goal as a final end product.” As a result, the critical thinking core of aesthetics seems to be lost. For example, although Fishwick identifies some of his own aesthetic norms in 26.7 (only some of which strike me as aesthetic) and his assumptions about aesthetics (e.g., that a principal concern is still the “universal attributes of beauty”), he doesn’t analyze them critically [5]. For example, the “unity

1870

Encyclopedia of Human-Computer Interaction

in variety” concept he endorses is a strategy in 18th century British aesthetics (developed by Francis Hutcheson and others) to identify a property of an object that accounts for its beauty without violating the shared principle among empiricists and rationalists that beauty itself is not a property of any object. How does such a concept or strategy help to clarify the “diversity” of computing or to negotiate between the conceptual nature of aesthetics and the empirical practices of computing? Moreover, as Jones argues, aesthetics can also help to determine the ends of computing by clarifying and critiquing its aesthetic and related norms, so it shouldn’t be viewed primarily as the means to achieve ends determined before aesthetic computing was introduced.

To be fair, even if Fishwick’s approach to aesthetic computing is narrow in the ways I have described, it may be that the field first has to develop through particular (and thus narrow) projects. Perhaps only then can we initiate a reflective equilibrium between the general field of aesthetic computing and the multivarious, particular projects that Fishwick and others are engaged in. Even though I think the general and particular have to be developed simultaneously from the start, Fishwick has clearly made important contributions to aesthetic computing in this Encyclopedia and his research.

With the same reflective equilibrium in mind, I’d now like to clarify my understanding of a broader conception of aesthetic computing because I appealed to it while critiquing Fishwick [6].

26.16.2 Aesthetics in Computing John Maeda (a computer scientist, designer, and President of the Rhode Island School of Design) once created “Palm Paintings”: small, shallow boxes painted in various abstract styles with a Palm computer built into each one serving as its visible center. His stated purpose was to enable us to “think,” from the inside, “about what the painting signified.” I take it that his point was not necessarily that

Aesthetic Computing

1871

signification is located materially inside the work of art but, more provocatively, that our critical thinking about the work should take place as the work is being made, if the critical thinking is to be truly inside the work and not merely added as an extra after the fact. The mode of critical thinking here is aesthetics since the key normative issues in art are aesthetic, making aesthetics an integral part of (Maeda’s) painting.

In a reciprocal gesture, now imagine that we were to embed aesthetics into the design and production of all the artifacts associated with computers – databases, programs, networks, data visualizations, games, etc [7]. The purpose would again be to think about what they signify and, prospectively, what else we might want them to signify in the future (as well as what other effects besides signification we would like to see). The computing artifacts with embedded aesthetics could be marked in some way to distinguish them from others. We could then hope to learn about ubiquitous computing from the inside, as it is being developed, not merely when it is already being used by people in society.

This reciprocal gesture is not imaginary because, as Fishwick has established, there’s been an “aesthetic turn” in a number of areas of computing, leading to the introduction of new subfields such as aesthetic computing, computational aesthetics, database aesthetics, digital aesthetics, information aesthetics, network aesthetics, or software studies [8]. The diverse names, introduced by collaborative research teams of computer scientists and others (e.g., artists, philosophers, art historians), are distinguished by where or, in the spirit of Nelson Goodman, when aesthetics is introduced into computing [9]. That is, if we think of the computer stack, the various layers of computing (with bits and hardware at the bottom and user interaction at the top), the choice of name here is a function of when aesthetic norms first enter computing. If aesthetic norms are involved in structuring databases, for example, then we have database aesthetics; if they influence how we give form to information, then we have information aesthetics; if they’re part of how we organize networks of people participating in various social media, then

1872

Encyclopedia of Human-Computer Interaction

we have network aesthetics – and so on within the layers of the computing stack.

The lower the layer on which aesthetic norms are implicitly present, the greater the ripple effect the critique of these norms will have on the higher layers of computing [10].

In this light, “aesthetic computing” is the one name among all the options that, in principle, encompasses the entire computing stack and thus best captures the full breadth and depth of the “aesthetic turn” in computing. In exploring more what aesthetics adds to computing, I want to emphasize that aesthetic computing is not merely about the aesthetics of computing (merely the design of programs or products, or merely an external critique of the aesthetic norms of computing).

Following Maede’s “Palm Paintings,” what I envision is aesthetics in computing, albeit with an anticipatory eye to its ethical and social-political impact rather than only its internal structure (i.e., not merely computational aesthetics).

26.16.3 What is Aesthetic Computing?

Aesthetics is critical thinking about the norms, concepts, values, or principles guiding or emerging from the production, experience, or reception of art, culture, or nature. Besides referring to the range of theory and practice associated with computer programming, databases, computation, software, operating systems, and hardware (everything from digits to gadgets), the term “computing” (as distinct from “computer science”) captures the recognition that computer science operates in a broad social (moral-political) context. Aesthetic computing is a preferred way to operationalize this recognition because it is critical thinking about the complex set of norms shaping all layers of computing that are, in turn, shaping this moral-political-social context [11].

Despite all the various names for aesthetic computing, there is a common thread running through all the versions or iterations of it. The thread is the recognition among people involved in computing that there are aesthetic norms im-

Aesthetic Computing

1873

plicit in the decisions or judgments made on all layers of computing. Accordingly, the main tasks of aesthetic computing are (1) to identify the genealogy and current status of the largely implicit aesthetic norms of computing and to render them explicit; (2) to critique the aesthetic norms with an eye to their moral-political-social implications for users; and (3) to help make decisions or judgments in the future about which aesthetic norms to abandon, revise, or sustain in computing, given (1) and (2), and of course given the technical norms within computing.

We can get a clearer picture of the need for aesthetic computing and its tasks by considering Zadie Smith’s review of David Fincher’s film, The Social Network, and of Jaron Lanier’s book, You Are Not A Gadget: A Manifesto [12]. What the film and book have in common, on Smith’s analysis, is the claim that “Different software embeds different philosophies, and these philosophies, as they become ubiquitous, become invisible.” Not only is software not neutral, and not only are there important norms embedded in it (e.g., personhood, privacy, sociality), but software in use also enacts these norms (i.e., puts them in practice). Software does not merely copy our existing norms about ourselves and the world, however, it also enacts new norms and, in doing so, makes a world. The problem, as Smith sees it, is that these invisibly embedded and enacted norms are not discussed critically in advance; rather, they are embedded and enacted by the programmers, most strikingly in the case of Facebook because 800+ million users have had little or no say about its norms.

Smith rightly points out that there are ethical issues involved in this case: Why these norms rather than others? Why this format for connecting people with one another rather than another format? What is the quality of the connection? Why this privacy policy? For example, users are expected to give up their privacy to a large extent, and they seem to do so willingly, albeit while reducing themselves to fit the software they are using, according to Lanier, so much so that their “life is turned into a database.”

Smith’s analysis is relevant to aesthetic computing not only because she points to the invisibility of the norms governing Facebook, the Internet, or the Web, but also because when she develops her critique of these norms, she often

1874

Encyclopedia of Human-Computer Interaction

refers to their “look” or “feel.” For example, while we “know” that it is a mistake to believe that computers can personify human relationships, we know this instinctively only by “feeling” the affective consequences of this mistaken belief, which Facebook embodies: “We know that having two thousand Facebook friends is not what it [friendship] looks like.” What is this look that we feel and that enables us in turn to know that certain norms embedded and enacted in Facebook may be problematic? We come to learn that Facebook is doing something to us through the invisibility of its underlying norms and, if our continued critical reflection is successful, we’ll come to learn what Facebook is doing to us and, moreover, whether there are any alternatives. To succeed, we need to render visible the invisible norms operating in Facebook so that we’ll have “a good reason” for at times feeling “discomfort at the world they’re making [in Facebook].” This kind of critical thinking is precisely what aesthetic computing offers because one of its main tasks is to render explicit the implicit norms of computing.

But let me return to the question: Why aesthetics? We might first ask, why philosophy? Smith answers this second question by emphasizing that “it’s the idea of Facebook that disappoints,” not merely the implementation of its idea. To analyze its idea, we need philosophy to counter what she sees as a general cultural tendency in the Anglo-American world to “race ahead with technology and hope the ideas will look after themselves.” We need to examine the idea of Facebook and all the other ideas enacted on the Web and Internet before, in Lanier’s words, we become “locked in” them, or “entrapped in somebody else’s careless thought,” which means that we are locked into the invisible norms shaping these ideas and, once those norms are enacted on the Web or Internet, shaping our world and us.

But why turn to aesthetics in particular to examine these ideas involving ethics (e.g., security), metaphysics (e.g., personhood or virtual reality), etc.? Returning to Smith’s discussion of the “look” of Facebook, and remembering Jones’s account of embodiment, the closest we come to experiencing the invisible norms that are enacted in software on the Web or Internet is by experiencing the affects they

Aesthetic Computing

1875

create on us, the users. Many of these affects are visible, but they involve all the senses (hearing and, increasingly, the tactile), just as works of art do and just as our aesthetic experiences of everyday life do. Aesthetics brings the affective dimensions of our experiences of computing to the fore, and it does so in a way that provides a basis for critique of the sort that Smith, Jones, and Lanier are exploring. These critiques are examples of aesthetic computing in action.

To take another kind of example clearly internal to computing, there has been an “aesthetic turn” in the area of human-computer interaction (HCI) because some researchers believe it is important to obtain a fuller picture of the “user” now that computer interfaces are more interactive, participatory, immersive, and ubiquitous [13]. In a word, they need to understand the user in affective, moral, and political as well as cognitive terms in order, in turn, to create the right (i.e., effective, usable) interfaces. So aesthetics comes into the picture as the notion of usability becomes normatively more complex. Why turn to aesthetics?

A major reason is that aesthetics has a long history of critiquing the particular kinds of affective and cognitive interactions and modes of participation constitutive of our experiences of art, and these critiques are relevant to the critiques of the affective-cognitive experiences of the user in human-computer interactions [14]. These interactions (with their own modes of participation) also have moral and political dimensions because users have to be treated fairly (e.g., in matters of access, whether for economic or disability reasons) and their political or cultural beliefs have to be respected. Here, too, aesthetics has a history of critiquing works of art in relation to moral-political as well as aesthetic considerations. The aesthetic turn here, whether in HCI or in any other field of computing, is therefore not a narrowing of moral-political-social impact to aesthetic questions; rather, aesthetics provides a philosophical structure for thinking critically about norms that are moral, political, social, and aesthetic at the same time [15].

1876

Encyclopedia of Human-Computer Interaction

26.16.4 Aesthetic Computing and Science Although the critique of aesthetic norms in computing with an eye to their moralpolitical-social impact is a relatively new process, we can more easily appreciate its relevance and importance if we see it as an augmentation of the existing practices of critical thinking in computing [16]. That is, computing has always critically analyzed its normativity, even if the norms have been understood mostly in technical terms (e.g., what is most efficient or effective). The emergence of aesthetic computing stems from the recognition within computing that its norms are more than technical, as we saw in the case of HCI. So aesthetic computing is principally an outgrowth and refinement of the recognition of the complex normativity always already operative within computing. This is an important point to emphasize because some computer scientists may view aesthetics the way they at times view ethical, political, or other issues seemingly external to computing: those issues are not relevant to what they do qua scientists (given their methodologies, aims, etc.) and thus to give such issues methodological credibility can only place constraints on science. However, if aesthetics (and the related normative) questions are understood as emerging from within computing, scientists no longer need to be concerned that aesthetics is constraining computing.

Yet researchers may still worry that aesthetic computing will change computer science in ways that would make it less scientific, especially if Roger Molina is right that the strong claim of aesthetic computing is that it will generate new objectives that “would not naturally have evolved within the computing sciences” and, moreover, that will “redirect the future development of computing.” [17].

That is, the transition from implicit to explicit aesthetic norms on the layers of the computing stack may have the result that we will change technical as well as aesthetic norms and then change the objectives of computing on that basis. But, again, if computing is normative and the self-critique of normativity is part of science, the only real change resulting from aesthetic computing is that the aesthetic norms always already part of computing will now be explicit and critically

Aesthetic Computing

1877

examined. How can computing not benefit from more self-critique, since the revision of its internal norms is part of the engine that has driven progress in modern science, on its own terms? For example, as computing becomes more conscious of the design issues that could contribute to environmental sustainability, that may change certain objectives of computing but it would not make computing less scientific, for if it were to become less scientific, it could not contribute to sustainability. In short, aesthetic computing shows how seemingly external norms are actually internal to computing.

The issue of the status of computing as a science is worth dwelling on even longer because it can stop the discussion of aesthetic computing cold. Some may still worry that aesthetics involves taste and is thus subjective. In this light, to integrate aesthetics into computing would be to introduce subjectivity into an otherwise objective science. However, what is actually happening here is that computer scientists are recognizing (a) that the normative complexity of computing has already shaped their idea of science, making room for a more interdisciplinary approach to computing, and (b) that computing is more than a science, not only because its moral-political-social impact entails too many nontechnical issues that scientists need to understand in order to develop computing internally, but also because the implicit nontechnical norms of computing are already shaping its development in ways that need to be analyzed critically for the sake of computing – as well as for our sakes as we live and work with computers [18]. In short, the aesthetic turn in computing is a way to critique its nontechnical norms in order to strengthen its status as science at this stage of its development. Why aesthetics?

Again, because it is a long-standing field of philosophy that has developed a variety of ways to think critically about aesthetic norms as they are related to moralpolitical-social as well as technical norms.

26.16.5  Open Aesthetic Properties and Objects Now, if aesthetic norms are always already a part of computing, why is aesthetic computing barely a decade old, though aesthetics has its origins in the eighteenth

1878

Encyclopedia of Human-Computer Interaction

century and computers have been around for decades already? One explanation, according to Fishwick, is that computing had to develop to a certain stage before its connection to aesthetics could clearly emerge: “We have had to wait for the technology to become available to leverage the arts,” especially in the fields of HCI, ubiquitous computing, augmented reality, and virtual reality [19]. Yet if aesthetics is so obviously relevant to computing, why was this delay necessary? After all, aesthetics is a form of critical thinking and computing has relied on critical thinking to evolve, so why didn’t aesthetic computing emerge earlier? Another explanation why its emergence has been slow, besides the worries about science being constrained or becoming subjective, is that too many people in computer science seem to have rather narrow, sometimes outdated ideas about aesthetics and thus have not been able to see its relevance to computing or, when they have seen the relevance, they’ve not been able to get from the narrow ideas to what they hope aesthetics could contribute to computing.

Too many people today still assume (and some philosophers still believe) that aesthetics is principally concerned with making disinterested judgments of the quality of beauty inherent in a class of unique and autonomous objects called works of art. However, aesthetics has no unique set of objects, not only because so many “things” can be works of art, as the history of modern art has taught us, but because aesthetics is as much about people, experience, and value as it is about objects or things. And beauty is no longer a principal concern in aesthetics because it’s not a principal concern in art (for a host of reasons analyzed by others elsewhere) [20]. Moreover, aesthetics is not about the fixed properties of any objects, whether works of art, natural objects, or artifacts of computing. This does not mean, intentionally or unwittingly, that aesthetics is merely subjective or that, as we sometimes hear, beauty is in the eye of the beholder. Aesthetics is not merely subjective any more than it is merely objective because beauty (understood not merely as a particular aesthetic property but as a stand-in for the entire

Aesthetic Computing

1879

set of aesthetic properties) is not in the subject any more than it is in the object.

But where is beauty, if it is not a fixed property of any subject or object? In the language of eighteenth-century aesthetics, beauty is a relational property, that is, a property resulting from cognitive and affective relations or interactions among human subjects or between them and an open-ended set of works of art, natural objects, or artifacts of computing. In this light, the task of aesthetic computing is to identify, render explicit, and analyze critically the various conditions – technological, social, ontological, psychological, etc. – that make such relations or interactions possible, not just what makes them more effective, usable, communicable, pleasurable, and the like, though by understanding what makes them possible we’ll presumably be in a better position to address these other concerns. Since the interactions here involve humans, and particularly since the interactions are not only between humans and objects but among humans (hence the need to shift from interaction to participation), aesthetic norms here are also moral and political. Again, aesthetics is able to coordinate all the dimensions of these norms better than either ethics or politics could because aesthetics has a long history of doing just that in the context of art.

On this account, aesthetics is a natural ally of computing because computing also traffics in objects lacking fixed properties, as is evident in Lev Manovich’s discussion of the word “object” in the Introduction to The Language of New Media.

Expressions such as Artificial Intelligence, Virtual Reality, Simulation, and Second Life likewise involve computer-based “realities” and objects that are not fixed.

Also, in the field of scientific visualization involving, say, molecular biology, the data that are visualized are inaccessible to human senses since there is no light at the molecular level. So the data do not constitute objects in the usual sense of the word and their visualizations have no objective correlates. This means that there is no single objective way to visualize molecular data, no essential visualization of them just waiting to be discovered by a computer scientist (though any visualiza-

1880

Encyclopedia of Human-Computer Interaction

tion is always constrained by scientific methodologies and goals). Moreover, this means these “objects� remain invisible even after they have been visualized, so it makes no sense to say that the visualization of molecular data have fixed properties (other than in the broadest sense of data properties – i.e., qua numbers and codes). Looking at this description of scientific visualization, computer scientists working in scientific (and other forms of) visualization should feel at home in aesthetics because molecular (and other) data are very similar to contemporary works of art: they too are not (necessarily) objects; they are more conceptual than sensuous, even when they assume sensuous form(s); and they are not imitations of objective realities against which they can be judged, so they can take numerous forms, subject to the limits of visualization and the methodological structures and goals of science (or art).

Now, if aesthetic computing is as much about human interactions as about objects or properties, a key question here is what makes these interactions aesthetic. How can we delimit the open-ended range of human-computer interactions and isolate those that are specifically aesthetic, especially if beauty is not a fixed property and is actually an effect of these interactions rather than a criterion for identifying them? [21] This question is both easier and harder to answer in the case of aesthetic computing than it might be in aesthetics more generally; easier, because the interactions have to involve some computing activities, artifacts, or the like, which, for the most part, are easier to identify than works of art, which have proven to be very elusive in recent years; yet harder too, because what is aesthetic about human interactions involving computers? The answer to this last question is that the norms implicitly embedded and enacted in the various layers of computing are what introduce the aesthetic dimension (hence database aesthetics, information aesthetics, etc., depending on which layer of the computing stack is involved).

The open-ended nature of aesthetic computing may create consternation among some computer scientists, or at least that has been my experience while re-

Aesthetic Computing

1881

searching, lecturing, or teaching about aesthetic computing. For there is a tendency to expect that aestheticians should provide objective norms (concepts, criteria, or the like) that can then serve as practical guides for researchers in computing (the field of “criticism” in computing sometimes embodies this tendency). If followed, however, this tendency would make aesthetics a field external to computing that is then applied to it. By contrast, I’ve proposed a model of aesthetic computing that operates only within computing by rendering explicit the aesthetic norms that are always already implicit and operative in the layers of the computing stack. Any new norms will have to emerge from within computing practices, just as new norms are introduced within artistic practices. In the end, aesthetics is either internal to computing or has little critical relevance to it.

26.16.6 Endnotes 1.

J. M. Coetzee, Diary of a Bad Year (New York: Penguin, 2008). See also Jaron Lanier: “Information systems need to have information in order to run, but information underrepresents reality” – You Are Not A Gadget: A Manifesto (New York: Knopf, 2006).

2. As the Editor of the Encyclopedia of Aesthetics (New York: Oxford University Press, 1998), I’m not suggesting that the general should exclude the particular – it’s all a matter of balance 3. Caroline A. Jones, “Introduction,” in Jones, Ed. Sensorium: Embodied Experience, Technology, and Contemporary Art (Cambridge: MIT Press, 2006).

4. For more on participatory art, see, e.g., Claire Bishop, Editor, Participation (London & Cambridge: Whitechapel Gallery MIT Press, 2006); and Artificial Hells: Participatory Art and the Politics of Spectator-

1882

Encyclopedia of Human-Computer Interaction

ship (London: Verso, 2012). Nicolas Bourriaud, Relational Aesthetics (France: Les Presse Du Reel, 2002). Rudolf Freiling, Editor, The Art of Participation: 1950 to Now (San Francisco & London: San Francisco Museum of Modern Art and Thames & Hudson). Pablo Helguera, Education for Socially Engaged Art (New York: Jorge Pinto Books, 2011).

Grant Kester, Conversation Pieces: Community and Communication in Modern Art (Berkeley: University of California Press, 2004); and The One and the Many: Contemporary Collaborative Art in a Global Context (Durham: Duke University Press, 2011). Nato Thompson, Editor, Seeing Power: Art and Activism in the Age of Cultural Production (New York: Melville House, 2012); and Living as Form: Socially Engaged Art from 1991-2011 (Cambridge: MIT Press, 2012).

5. To give another example, Lev Manovich argues that people often point positively to the user-generated content available now online (e.g., anime music videos, political mashups) as evidence of artistic freedom or creativity on the internet (even enhanced democracy), yet they fail to reflect critically on the fact that this content follows implicitly embedded and enacted industry templates and conventions or reuses professionally produced content. Manovich, “Art After Web 2.0” in The Art of Participation: 1950 to Now. 6. Since I’m a philosopher, it’s likely inevitable that my perspective on this new field is going to be general. But such generality is also due to the fact that aesthetics is a conceptual and normative field, though it clearly must be linked to the empirical reality of computing if it’s going to have any efficacy as a mode of critical thinking that is internal to computing.

7. Mary Flanagan and Helen Nissenbaum have developed “values at play,” a conception of critical play that identifies and transforms the values embedded and enacted in computer (and other) games. As I see it, their

Aesthetic Computing

1883

approach is a good example of aesthetic computing because they render explicit the implicit norms of games. But they do not appeal to aesthetics. In fact, they seem to shun it, perhaps because Flanagan is an artist and seems to adopt uncritically the anti-aesthetic stance common in contemporary art, while Nissenbaum is a philosopher who doesn’t yet appreciate the critical value of aesthetics. This is unfortunate, I think, because aesthetics provides exactly the kind of conceptual and critical resources Flanagan and Nissenbaum are developing as they analyze and create games that embed and enact transformative values. See Flanagan, Critical Play: Radical Game Design (Cambridge: MIT Press, 2009); and Flanagan and Nissenbaum, Values at Play (forthcoming).

8. Aesthetic Computing began at a conference in Dagstuhl, Germany, in 2002, from which emerged a manifesto published in Leonardo in 2003, and an anthology, Aesthetic Computing, Paul Fishwick, Ed. (Cambridge: MIT Press, 2006) (which I reviewed in Leonardo On-line Reviews (January 2007): http://www.leonardo.info/reviews/jan2007/aest_kelly.html Computational aesthetics, which is also called (or linked to) algorithmic aesthetics or exact aesthetics, has been traced back to the 1930s; see Gary Greenfield, “On the Origins of the Term ‘Computational Aesthetics’”; and Florian Hoenig, “Defining Computational Aesthetics,” in Computational Aesthetics in Graphics, Visualization and Imaging, I. Neumann, M. Sbert, B. Gooch, W. Purgathofer, Editors (2005), pp. 9-12 and 13-18. Database Aesthetics can be traced back to at least 1999; see Victoria Vesna, Editor, Database Aesthetics (Minneapolis: University of Minnesota Press, 2007). For examples of Digital Aesthetics, see Sean Cubitt’s website: http://www.ucl.ac.uk/slade/digita/; and Johanna Drucker, SpecLab: Digital Aesthetics and Projects in Speculative Computing (Chicago: University of Chicago Press, 2009). Information Aesthetics has an active website: http://infosthetics.com/. See

1884

Encyclopedia of Human-Computer Interaction

also the SIGGRAPH Information Aesthetics Showcase in 2009: http:// www.siggraph.org/s2009/galleries_experiences/information_aesthetics/ F or an example of Network Aesthetics, see Warren Sack, “Network Aesthetics,” in Database Aesthetics, pp. 183-210. For an example of Software Aesthetics, see Stephan Diehl and Carsten Görg, “Aesthetics and the Visualization and Quality of Software,” in Fishwick, Aesthetic Computing, pp. 230-37. There are also various websites devoted to this topic.

And there’s also Visual Aesthetics, discussed extensively elsewhere in this Encyclopedia.

9. Goodman, “Art in Action,” in Encyclopedia of Aesthetics, pp. 322-25.

For an account of Goodman’s relevance to aesthetic computing, see John Lee, “Goodman’s Aesthetics and the Language of Computing,” in Aesthetic Computing, pp. 29-42.

10. Manovich speaks of the cultural layer in addition to the computing layer, but I’m envisioning aesthetic computing that integrates rather than separates these layers. See The Language of New Media (Cambridge: MIT Press, 2001).

11. As Fishwick clarifies in his Encyclopedia entry, aesthetic computing is different from computer or digital art, that is, digital technology applied to the arts. “Aesthetic computing” refers to the impact of artistic practices and aesthetic principles on the field of computing, so the influence flows from art and aesthetics to computing. For example, computer scientists are looking to learn from artists how to conduct critiques of their prototypes for new technologies (as artists do of their new works); how best to visualize data in scientific, information, or knowledge visualization; and how to understand the balance between form and function or, more typically in computing, beauty and usability in new technologies, especially those involving user interfaces. As these kinds of influence of art on computing are developed, aesthetics is a natural third party since art always involves some type of aesthetics.

Aesthetic Computing

1885

12. Zadie Smith, “Generation Why?” (Review of The Social Network, a film directed by David Fincher, with a screenplay by Aaron Sorkin; and Jaron Lanier, You Are Not a Gadget: A Manifesto (New York: Knopf, 2010), in New York Review of Books (November 25, 2010): http://www.nybooks.com/articles/archives/2010/nov/25/generationwhy/?pagination=false 13. See, e.g., Olav W. Bertelsen and Søren Pold, “Criticism as an Approach to Interface Aesthetics,” NordiCHI ‘04, October 23-27, 2004; Lars Erik Udsen and Anker Helms Jørgensen, “The Aesthetic Turn: Unravelling Recent Aesthetic Approaches to Human-computer Interaction,” Digital Creativity, 16, 4 (2005): 205–16; Jeffrey Bardzell, “Interaction Criticism and Aesthetics,” Proc. of CHI’09. ACM Press (2009), 2357-66, and Jeffrey Bardzell, “Interaction Criticism: An Introduction to the Practice,” Interacting with Computers, 23 (2011) 604–21. See also Olav W. Bertelsen: “Tertiary Artifacts at the Interface,” in Aesthetic Computing, ed. Paul Fishwick (Cambridge: MIT Press, 2006), pp. 357-368. According to Bertelsen, “human-computer interaction requires understanding of the aesthetics of computing technology,” that is, “how computing technology is experienced and ‘experienceable.’ Input from aesthetic computing is greatly needed in human-computer interaction” (p. 359). In explaining what he has in mind, Bertelsen analyzes the work of Marx Wartofsky, a philosopher of art and science. I think this is a very good article in aesthetic computing, even if one does not accept the Wartofsky framework, because Bertelsen clarifies aesthetics in a way that is philosophically sound, linked to art and science, and relevant to computing.

14. See, e.g., Kirsten Boehner, Rogério DePaula, Paul Dourish, and Phoebe Sengers, “Affect: From Information to Interaction,” CC 05, Proceedings of the Dicennial Conference on Critical Computing (New York: ACM Press), pp. 59-68. See also the MIT Lab for Affective Computing: http:// affect.media.mit.edu/

1886

Encyclopedia of Human-Computer Interaction

15. While any other discourses or disciplines implicated in this normative complex could critique its own type of normativity, only aesthetics is able to critique the normativity in all its complexity. For example, when Ken Goldberg installed “Demonstrate” (2004) in Sproul Plaza on the campus of the University of California at Berkeley, his project raised all sorts of issues and aesthetics is arguably at the center of them all. He set up a robotic webcamera for six weeks (24/7) that could be manipulated (zooming in, taking photographs, and the like) by people in remote locations, allowing somebody in Tokyo, say, to conduct surveillance on people in the Berkeley plaza. Although it was technology that made this installation possible, it clearly was not just an engineering project because of the consequences of remote surveillance on unsuspecting people in an open plaza on the campus of a public university. There were legal issues, starting with the question of the privacy rights of the people under surveillance, in particular because, as I understand it, the camera was not calibrated tightly enough at first so it was able to scan beyond the parameters intended for the project. In addition, because this project was also construed as an art work, there were also issues of artistic freedom, not only on behalf of Goldberg (and perhaps the people conducting the surveillance) but for the people in the plaza; for they were no longer as strictly constrained in their public behavior because they were participating in a work of art (apparently, some people engaged in or at least simulated sex acts under the protection of artistic freedom). Finally, the project commemorated the 40th Anniversary of the Berkeley-led Free Speech Movement, so there were important political issues at stake too because the movement was subjected to surveillance in its time, albeit without today’s more sophisticated technology. Aesthetic critique is able to make sense of the normative complexity (technical, legal, ethical, political) of a project like Goldberg’s Demonstrate because, again, aesthetics has a long history

Aesthetic Computing

1887

of analyzing works of art with this same type of normative complexity.

16. Warren Sack argues, as I understand it, that the recognition of the aesthetic (as well as other nontechnical) dimensions of software and computing was evident from the early days of computing. See his website: http://people.ucsc.edu/~wsack/ 17. Roger Malina, “A Forty-Year Perspective on Aesthetic Computing in the Leonardo Journal,” in Fishwick, Aesthetic Computing, p. 48. The other, weak claim is that aesthetics may help computer scientists “achieve their [existing] objectives more easily, quickly, or elegantly” (p. 47).

18. For similar developments in other sciences, see, e.g., Aesthetic Science: Connecting Minds, Brains, and Experience, Arthur P. Shimamura and Stephen E. Palmer, Eds. (New York: Oxford University Press, 2012).

19. Fishwick, Aesthetic Computing, p. 13.

20. On the fate of beauty in modern art, see, e.g., Arthur C. Danto, The Abuse of Beauty: Aesthetics and the Concept of Art (Chicago: Open Court Press, 2003); Elizabeth Prettejohn, Beauty and Art: 1750-2000 (New York: Oxford University Press, 2005); and Wendy Steiner, Venus in Exile: The Rejection of Beauty in 20th Century Art (Chicago: University of Chicago Press, 2001).

21. There a long-standing discussion of the open nature of art works in the history of contemporary aesthetics. See, e.g., Umberto Eco, The Open Work, tr. A. Cancogni (Cambridge: Harvard University Press, 1989; originally published in 1962).

1888

Encyclopedia of Human-Computer Interaction

26.17 Commentary by Roger Malina How to cite this commentary in your report

Roger Malina

© Roger Malina

Roger F. Malina is an art-science researcher, astronomer and editor. He is a Distinguished Professor of Arts and Technology and Professor of Physics at the University of Texas, Dallas where he is developing an Art-Science R and D and Experimental Publishing program. He is a Directeur de Recherche of the CNRS and former Director of the Observatoire Astronomique de Marseille Provence at A...

Roger Malina Roger Malina is a member of The Interaction Design Foundation Paul Fishwick has been formulating for a number of years his concept of “Aesthetic Computing”, broadly defined as the application of the theory and practice of aesthetics to computing; in the process an “embodied” formal language is advocated. In my view this approach becomes particularly pertinent if, in the process, the methods and content of computing as science and engineering is changed and

Aesthetic Computing

1889

enhanced. I have called this kind of goal the ‘strong case’ for art-science interaction [1] where the interaction goes beyond the demonstrative or pedagogical. It is perhaps no accident that at the Dagstuhl workshop a ‘manifesto’ was issued [2] as there was a general feeling by the participants that the possible contributions of the arts, design and humanities to computer science were not generally accepted outside a group of enthusiasts.

A number of developments, some of which are referred to by Fishwick are mentioned, but here I would like to address a number of issues which are included in the concept of aesthetic computing but go beyond it. Fishwick talks of aesthetic computing addressing the different elements of formal languages which are number, data, model and software.

Here it is perhaps useful to add Denning’s [3] seven principles of computing; these have the advantage of being process oriented and helps focus areas of possible art and design intervention: 1.

Computation: What can and cannot be computed

2. Communication: Reliably moving information between places 3. Coordination: Effectively using many computers 4. Recollection: Representing, storing and retrieving information from media 5. Automation: Discovering algorithms for information processes 6. Evaluation: Predicting performance of complex systems 7. Design: Structuring software systems for reliability This significantly broadens the landscape of possible intervention of aesthetic computing approaches, and indeed many of these areas have been barely engaged by the arts and humanities to date.

1890

Encyclopedia of Human-Computer Interaction

26.17.1 Big Data Transition and the Crisis of Representation.

A major development in the last decade is sometimes referred to as the “big data” transition [4]. As data volumes and rates continue to grow at accelerating rates scientific disciplines go through transformational changes. Astronomy was perhaps the first discipline to make this transition with the emergence of virtual observatory strategies on both data archives and software. Fields such as genomics soon followed and now all areas of business and culture are impacted (see for instance [5]). This has led to what might be called a “crisis of representation” and the emergence of new disciplines such as infoviz and dataviz. It has rapidly become apparent that the problem is no longer one of ‘communication’ of the content of data via illustration techniques (e.g. [6]) but rather the problem becomes one of immersion in data which can no longer be thought of as ‘objects’ but rather as a “fluid”; hence strategies of reification referred to by Fishwick are proving inadequate as we enter media territory. A number of researchers have been seeking to expand the reach of ‘image science’ into this new territory that requires semiotic approaches not yet developed. Most data is never analysed or viewed, and new kinds of “technologies of attention” are required to help navigate and isolate data that has particular content or meaning. This crisis of representation is a good area for the arts and humanities to be involved and will be a long term agenda for aesthetic computing; and as emphasised by Fishwick the key issue is embodiment or how data is put in forms that are apprehensible via the human senses. A number of artists have been prominent in exploratory projects for navigating through data such as Donna Cox [7] and Ruth West [8]. There have been a number of exhibitions that have sought to display the variety of approaches (e.g. siggraph information aesthetics [9]). It seems to me that Fishwick pays insufficient attention to these developments which are rapidly leading to new research areas.

Aesthetic Computing

1891

26.17.2 Artificial Life Art, Visual Mathematics and Embodied Code Paul Fishwick points out briefly the area of serious gaming as one area where code can be seen as being embodied. It seems to me there is a far larger area of algorithmic art, visual mathematics, artificial life art which have much stronger examples of embodiment of code. The development of algorithmic art by computer art pioneers such as Michael Noll, Roman Verostko, Harold Cohen to name just a few has already established a 50 year history of making algorithms apprehensible by human vision and hearing using aesthetic means and objectives. The area of visual mathematics (see Michele Emmer and his two Leonardo books on the visual mind [10]) has already provided success stories of how aesthetic methods have led to scientific, or mathematical, discoveries. Following the Santa Fe Institute workshops that established visibly the bases of artificial life [11], and its more recent applications to synthetic biology, the arts community rapidly picked up the challenge with the development of artificial-life art projects that has led to a proliferation of projects in robotics, virtual worlds, interactive installations and other ways of bringing code into physical contact with the human senses. Recently Leonardo Journal published [12] a selection of works as part of the 10th anniversary of the VIDA artificial life competition, the pre-eminent forum for artificial life art practitioners.

26.17.3 Translation as a possible method for Aesthetic Computing In section 26.10 Fishwick articulates the elements of a method for aesthetic computing.

In recent discussions with colleagues Rainer Schulte and Frank Dufour at the University of Texas, Dallas [13], I have been impressed by their work that seeks to apply the humanities based methods of translation studies to the problems posed by computer generated or mediated forms of creative “writing” (whether text, image, sound or indeed multi-medial and multi sensorial). There are issues of media “essentialism’ as pointed out by Fishwick in his section 26.2.4 on Media, where each medium has specificities that

1892

Encyclopedia of Human-Computer Interaction

allows or prevents certain concepts to be translated from one medium to another. But in addition - as pointed out by Fishwick - the limitations of technology may constrain certain form of embodiment. He gives the example of the experience of early word processing software where the writer had to ‘stop and wait’ for the microprocessor to catch up; similarly now certain microprocessor steps occur at faster rates than the cycle times of the human cortex, so that the act of translation from code to embodied perception requires the slowing down and time stretching of phenomena. The act of translation from the culture of the “formal languages of computing” to the “formal languages of the arts and literature”, or vice versa, requires methodologies from the humanities in order to create meaningful embodiment strategies for aesthetic computing.

26.17.4 Intimate Science Elsewhere I have written [14] of the general problem of how to enable cultural appropriation of scientific phenomena inaccessible to the human senses. I have called this the agenda of ‘intimate science’ that many artists are now involved in.

As pointed out by philosophers many of the ways that we conceive of the world are built from our experiences from birth onwards. Our ideas of causality, or more generally of explanatory systems, are fed by our interactions with the world via our bodies and senses. But science now deals with many phenomena that are not only beyond the “amplification” or “augmentation” of our senses but are inherently ‘non-commensurate’ with the way our senses operate. This problem perhaps emerged most clearly in the case of quantum mechanics where our basic ontologies are no longer applicable (objects can be both wave and particle) and concepts of causality (in the case of entanglement) totally foreign to our experience in the macro world. I would argue that we run into similar issues in the concept of emergence in complexity science. Certainly we run into similar issues in making sense of general relativity and the presence of distortions in space-time and the structure of space itself; there is no way to experience gravity waves as a human being.

It seems to me that Fishwick’s agenda for aesthetic computing in a way transposes

Aesthetic Computing

1893

the problem of making science intimate into the field of computer science. Computers have different internal logics than human bodies, different teleologies, and only through the mechanisms of embodiment that he discusses can we begin to ‘imagine’ the way that a computer “imagines”, The exercise of transposing ‘models’ into ‘maquettes’ is one such process of cultural appropriation.

26.17.5 STEM to STEAM During the last two years the U.S National Science Foundation, in partnership with the U.S National Endowment for the Arts, have been organizing a number of workshops that bring together the research communities in Science and Engineering and the creative communities in Arts and Design [15]. These initiatives were responding to the sociological fact that there is a growing body of research practice that bridges Science and Engineering to Art and Design; in some cases School of Arts and Design find themselves engaged very similar research agendas to Science or Engineering departments with of course different, or overlapping, outcomes in mind. Sometimes unusual trans-discliplinary collaborations are involved, in other cases artists and designers find themselves in the role of inventors and technological innovators. This development was first recognized in information technology. The “Mitchell” report “Beyond Productivity” [15] in 2003 laid out the problems and opportunities. Similarly developments are now occurring in other areas of science and engineering. As a result of these workshops the NSF has put in place two research contracts to stimulate network development via the SEAD initiative [16], and a trans disciplinary documentation platform via the XSEAD contract [17]. An early outcome of these workshops was the acceleration of the concept of ‘turning STEM into STEAM’. After thirty more years of national efforts in the U.S to develop the STEM workforce pipeline, the U.S is faced with shortage of trained scientists and engineers. As articulated by the STEM to STEAM movement (see for instance the congressional testimony organized

by John Maeda, President of the Rhode island School of Design [18]) we need to integrate the Arts and Design, or more generally the Arts and Humanities, into Sci-

1894

Encyclopedia of Human-Computer Interaction

ence Technology Engineering and Mathematics education and research strategies.

Some of the fastest growing computer science related programs are in the computer arts, gaming, social media; and as pointed out by theorists in innovation studies, the process of social and cultural innovation is playing a stronger and stronger role in successful adoption of disruptive technologies. Expertise in visualization and image science areas that are grounded in the arts and humanities are promising areas that emerge from the aesthetic computing agenda that is outlined by Fishwick. As pointed out by Fishwick our institutions of higher education are badly organized to address these research agendas; the program that Fishwick has been developing at the University of Florida is one exemplar of possible approaches.

Fishwick addresses strategies of aesthetic computing for the formal language construct which are number, data, model and software. If we add Denning’s seven principles of computing namely Computation, Communication, Coordination, Recollection, Automation, Evaluation, and Design it is clear that aesthetic computing is part of a larger ensemble of arts and humanities research strategies that offer the opportunity of making major contributions to computer science in the coming decades. As I write these comments, there is a large online discussion on “The New Aesthetics”, a discussion that credits its source as James Bridle’s blog “The New Aesthetics” [20] in May 2011. With a starting point that computing is now culturally integrated into our way of being in the world, the discussion (see for instance Ian Bogost [20], for a rebuttal) has been lively - indicating that we are only at the beginning of aesthetic computing.

26.17.6 References 1.

Malina, Roger (2011), The Strong Case for Art-Science Interaction, Retrieved from http://vectors.usc.edu/thoughtmesh/publish/120.php

2. Paul Fishwick et al., 2003, ‘Aesthetic Computing Manifesto’. Leonardo, 36, Issue No 4,

Aesthetic Computing

1895

3. Peter Denning, http://cs.gmu.edu/cne/pjd/GP/GP-site/welcome.html 4. The fourth paradigm, Data Intensive Scientific Discovery, EDITED BY Tony Hey, Stewart Tansley,and Kristin Tolle, 2009 Microsoft Corporation, ISBN 978-0-9825442-0-4.

5. http://datajournalism.stanford.edu/ 6. Edward Tufte, http://www.edwardtufte.com/tufte/ 7. Donna Cox, http://www.ncsa.illinois.edu/~cox/ 8. Ruth West, http://www.atlasinsilico.net/gallery.html 9. 2003 Siggraph Information Aesthetics Show case, http://www.siggraph. org/s2009/galleries_experiences/information_aesthetics/ 10. Michele Emmer, The Visual Mind, Leonardo Books, 1993, http://leonardo.info/isast/leobooks/books/emmer.html 11. Christopher G Langton (1998). Artificial life: an overview. MIT Press.

ISBN 0262621126 , 12. Leonardo Journal Vida Gallery, http://www.leonardo.info/isast/journal/toc411.html 13. http://translation.utdallas.edu/ 14. http://www.sonicacts.com/portal/index.php/roger-malina-intimatescience-or-artists-in-the-dark-universe/ 15. Beyond Productivity : http://www.nap.edu/openbook.php?record_ id=10671&page=235 16. http://sead.viz.tamu.edu/index.html

1896

Encyclopedia of Human-Computer Interaction

17. http://www.slideshare.net/ironman28/xsead-presentation-scalar-11912 18. http://stemtosteam.org/ 19. James Bridle, http://new-aesthetic.tumblr.com/ 20. Ian Bogost, http://www.wired.com/beyond_the_beyond/2012/04/ianbogost-the-new-aesthetic-needs-to-get-weirder/

26.18  Commentary by Sophia Krzys Acord How to cite this commentary in your report

Sophia Krzys Acord

Š Sophia Krzys Acord

Sophia Acord is Associate Director of the Center for the Humanities and the Public Sphere at the University of Florida. She holds Ph.D. and M.Res. degrees in Sociology from the University of Exeter (UK) and a B.A. from Swarthmore College

Aesthetic Computing

1897

(USA). Acord’s research draws on her background in the sociology of art and science and technology studies in order to explore the impact of digital tec...

Sophia Krzys Acord Sophia Krzys Acord is a member of The Interaction Design Foundation The hand knows what the mind cannot tell. This classic adage about embodied cognition, familiar to artists (cf: Sudnow, 1978) and craftsman (I like: Crawford, 2009), plays an important role in representing the complex ways in which we come to learn and understand the knowledge and substance at work in building our physical worlds. This adage also reveals the shortcomings of the perpetuating Cartesian superiority of mind over body, which privileges language as an authoritative mediator of knowledge. Paul Fishwick’s career work to develop and formulate the novel field of Aesthetic Computing demonstrates an exciting and important way in which notions of sensory and bodily experience can be brought to bear on the most formal of languages: computer code. In this commentary, I will give a short definition of embodied cognition, a brief background on how thinking aesthetically about interaction can reveal important dimensions of knowledgemaking, and then posit a few ways in which aesthetic computing may be transformational for the evolving digital humanities.

Cognition refers to the mental processes involved with gaining knowledge, including those involved in producing and understanding language. To cite an entry on “embodied cognition” from another excellent open-access scholarly encyclopedia, the Stanford Encyclopedia of Philosophy, we can see cognition as embodied when aspects of an individual’s body “beyond the brain” play a significant constitutive role in one’s ability to intake, process, and develop an understanding of new knowledge. New research in the cognitive sciences provides further support for the importance of embodiment in conceptual learning. As philosopher of mind Alva Noë (2006) described in summarizing this work, the sense of touch,

1898

Encyclopedia of Human-Computer Interaction

not vision, should be our model for thinking about perception; we acquire new content through active inquiry and exploration. (This is not a new idea of course, Vichean philosophy — see Vico, 1725 — drew on the work of Aristotle to argue that men can only know what they make.) The understanding that knowledge is an action — something we do in concert with material objects, bodies, and environments — is also supported by much qualitative research in my home field of the sociology of the arts (cf: Acord and DeNora, 2008; Sutherland and Acord, 2007).

Seen from this vantage point, aesthetic computing is a move that fits into a broader theoretical paradigm interested in exploring the non- and quasi-cognitive aspects of behavior, knowledge-production, and interaction, as well as the important roles played by materials, technologies, and objects in the worlds we make. As MIT social scientist Sherry Turkle describes in her 2007 edited volume, Evocative Objects, the physical objects in our lives are anchors of our memories, thoughts, and action; how we interact with them demonstrates that thought and feeling are linked. Similarly, music sociologist Tia DeNora (2000) pointed out that aesthetic materials, like the songs we hear, are accomplices in our everyday lives; they allow us to undertake tasks that we could not accomplish without them. (Any Zumba instructor will be familiar with this power of music.) Even earlier studies in science and mathematics support this point. Looking at how shoppers in the grocery store use mathematics, Jean Lave (1988) demonstrated that cognition is an interactive process between persons acting and the settings in which their activity is constituted. This has also been discovered in a range or professions, including: design engineers (Henderson, 1999), cookie manufacturers (Streeck, 1996), and ship navigators (Hutchins, 1995). As Fishwick rightly points out, incorporating aesthetic encounters into learning software design reunites the mind and body of the computer scientist such that the physical coding (or serious gaming) experience can build understanding of more abstract analytic concepts. Creating embedded virtual experiences for learning code, or otherwise bringing real world bodily metaphors into software design, is significant in the student interaction

Aesthetic Computing

1899

with formal language. The resources we have at our hands with which to make meaning influence what we can know.

In this chapter, Fishwick identifies the challenge of aesthetic computing as “connecting humans with computers”. In an evaluation of one of his University of Florida aesthetic computing courses, Fishwick, et al. (2005) discovered that many of the undergraduate students felt that aesthetic computing was time consuming, but particularly useful in explaining computing concepts to non-engineers.

In digging more deeply into the theory behind aesthetic computing, however, I posit that there are additional important opportunities to connect humans and computers here to advance knowledge in computer science and also the humanities disciplines.

As sociologists of science show, producing scientific knowledge requires moving from ‘dirty’, ‘fuzzy’, and hands-on experiences to abstract and codified representations (cf: Latour and Woolgar, 1979). As a result, scientific results and findings are translations of our human experiences that may distort what it is that we really know. Ong (1982) makes a similar argument about the technology of written human languages: learning a written language entails a transformation of consciousness; we begin thinking with words, rather than speaking our thoughts.

While Fishwick cites Mark Johnson and George Lakoff’s good work to show that our embodied experiences are present in language through metaphor, Ong also demonstrates that written languages risk eliminating processes of embodiment by positioning words and written language notations as artificial mediators of what we know and how we can express it. How can we exit language to study language concepts?

I see aesthetic computing as offering an opportunity for computer scientists (and their students) to engage differently with software design by creating a new and embodied experience to ‘play’ (cf: Huizinga, 1944) with the concepts upon which formal language notation is built. By side-stepping formal language no-

1900

Encyclopedia of Human-Computer Interaction

tation as a mediator of our knowledge of mathematical relations, and engaging with the body as a different kind of mediator, aesthetic computing may enable new ways of thinking about software design. Briefly, let us consider the case of art as an aesthetic activity, in which art-making or participation is a way to externalize and reflect upon ‘felt’ experiences in order to grasp (but also to extend) linguistically mediated situations. Art can be a place to work through alternative constructions and implementations of our understandings; in this way art may enable healing, conflict resolution, and social movements (Acord and DeNora, 2008). Similarly, as education scholar Donald Schön (1987) observes of architecture students, engaging in the hands-on ‘making’ and interrogation of more abstract architecture concepts through building physical models creates opportunities for the on-the-spot experimentation, problem-solving, and tinkering that he terms “reflection-in-action”. Importantly, this embodied tinkering can create opportunities for questioning and altering language design concepts that may result in amendments to formal language systems. (The hand may feel something differently than how the mind classified it.) In this chapter, Fishwick observes, ‘as we break open the lid of the black box containing the atomic elements of normally hidden data, formulas, code, and models, we find that computing is theatre all the way down.’ If computing is theatre, the trick is to treat it as such: a place to engage in embodied play to not simply repeat well-rehearsed formal language concepts but also to tinker with or improve upon them. This potential for ‘improvisation’ is, to me, one of the most exciting potentials of aesthetic computing.

Finally, Fishwick describes aesthetic computing as a terrain to bring together computer scientists, artists, and humanities scholars to extend our models of meaning-making in the digital age. Earlier in the chapter, however, Fishwick draws on the work of Sherry Turkle and the example of the digital watch to discuss how deep abstract concepts in computing (as a formal language) affect the thinking of users. The idea that formal language shapes (and, by inference, limits) how we think is particularly frightening to some scholars in the humanities who would

Aesthetic Computing

1901

like to use computing technology to expand interpretive possibilities (not delimit them). (As Jaron Lanier, 2010, notes: we should use gadgets, not the other way around.) Johanna Drucker, in particular, has written about the tensions between formal languages based upon mathematics and their use with images and digital humanities data that question formalized and established regimes of meaning (cf: Drucker, 2001, 2009, 2011). Aesthetic computing, which enables questioning and play with formal language concepts themselves (not simply their written notation as code), may well be the solution to creating genuinely new and open-ended interpretive interfaces. As Dexter, et al. (2011) argue in a recent special issue in Culture Machine on the digital humanities, “The functional role of aesthetics plays out most richly when contextualized as part of the creative process of software development� (pp. 16-17). In other words, opportunities to transform interactive computing environments are enhanced when programmers see themselves as embodied. I posit, then, that to advance the digital humanities we must not only think about code aesthetically or symbolically, as does the research area of critical code studies, but take a step further and think differently about the mental/ physical constructions upon which coding is based. In this way, aesthetic computing provides the physical and bodily tools to dramatically rethink possibilities for cultural computing in the digital age.

26.18.1  References ff Acord, Sophia Krzys and DeNora, Tia (2008). Culture and the arts: From art worlds to arts-in-action. The Annals of the American Academy of Political and Social Science, 619(1): 223-237. ff Crawford, Matthew (2009) Shop Class as Soul Craft: An Inquiry into the Value of Work. New York: The Penguin Press. ff DeNora, Tia (2000) Music in Everyday Life. Cambridge: Cambridge University Press.

1902

Encyclopedia of Human-Computer Interaction

ff Dexter, Scott, Melissa Dolese, Angelika Seidel, and Aaron Kozbelt (2011) On the embodied aesthetics of code. Culture Machine, 12. ff Drucker, Johanna (2001) Digital ontologies: The ideality of form in/and code storage: Or: Can graphesis challenge mathesis? Leonardo, 34(2): 141-145. ff --- (2009) SPECLAB: Digital Aesthetics and Projects in Speculative Computing. Chicago, London: University of Chicago Press. ff --- (2011) Humanities approaches to interface theory. Culture Machine, North America, 12. ff Fishwick, Paul, Timothy Davis, and Jane Douglas (2005) Model representation with aesthetic computing: Method and empirical study. ACM Transactions on Modeling and Computer Simulation, 15(3): 254—279. ff Henderson, Kathryn (1999) On Line and On Paper: Visual Representations, Visual Culture, and Computer Graphics in Design Engineering. Cambridge, MA: The MIT Press. ff Huizinga, Johan (1944) Homo ludens: A study of the play element in culture. Translated by G. Steiner, 1970. London: Paladin. ff Hutchins, Edwin (1995) Cognition in the Wild. Cambridge, MA: The MIT Press. ff Lanier, Jaron (2010) You Are Not a Gadget: A Manifesto. New York: Alfred A. Knopf. ff Latour, Bruno and Steve Woolgar (1979) Laboratory Life: The Construction of Scientific Facts. Beverly Hills: SAGE Publications. ff Lave, Jean (1988) Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life. Cambridge: Cambridge University Press.

Aesthetic Computing

1903

ff Noë, Alva (2006) Action in Perception. Cambridge, MA: The MIT Press. ff Ong, Walter J. (1982) Orality and Literacy: The Technologizing of the Word. New York: Methuen. ff Schön, Donald A. (1987) Educating the Reflective Practitioner. San Francisco: Jossey-Bass Publishers. ff Streeck, Jürgen (1996) How to do things with things: Objects trouvés and symbolization. Human Studies, 19: 365-384. ff Sudnow, David (1978) Ways of the Hand: The Organization of Improvised Conduct. Cambridge, MA: Harvard University Press. ff Sutherland, Ian and Sophia Krzys Acord (2007). Thinking with art: From situated knowledge to experiential knowing. The Journal of Visual Art Practice, 6(2): 125-140. ff Turkle, Sherry (ed.) (2007) Evocative Objects: Things We Think With. Cambridge, MA: The MIT Press. ff Vico, Giambattista. (1725) Scienza Nuova. (The First New Science, edited and translated by Leon Pompa, Cambridge: Cambridge University Press, 2002.)

1904

Encyclopedia of Human-Computer Interaction

26.19 Commentary by Kang Zhang How to cite this commentary in your report

Kang Zhang

© Kang Zhang

Kang Zhang is Professor and Director of Visual Computing Lab, Department of Computer Science at the University of Texas at Dallas. He is also a Board Director of Vital Art and Science Inc., USA. He holds a B.Eng. degree in Computer Engineering from University of Electronic Science and Technology of China, a Ph.D. degree from University of Brighton, UK, and an Executive MBA degree from th...

Kang Zhang Kang Zhang is a member of The Interaction Design Foundation Fishwick’s edited book published in 2006 had formally introduced the concept of Aesthetic Computing [1]. Based on the then definition, aesthetic computing addresses the question “how can the theory and techniques in the traditional visual art help beautify modern technology outputs and products and enhance their

Aesthetic Computing

1905

usability?”. It includes the aesthetic design of computer algorithms, simulation, visualization [4], human-machine interfaces, and high-tech products, so that users are highly engaged and thus usability is enhanced. An interesting example of aesthetic computing is to apply Kandinsky’s aesthetics to Java programming [3]. Malina [2] highlighted the aesthetic computing activities published in the Leonardo journal over the last forty years. Fishwick’s new chapter on aesthetic computing takes a more specific and operational view on aesthetic computing, focusing on the notion of embodied formal language. In analogy to artists who were considered to be the first virtual reality creators, computer scientists may interpret objects dynamically via bodily simulations, possibly within virtual reality environments.

A related new discipline, but conceptually in reverse direction, is computational aesthetics that aims at answering the question “how can the computer automatically generate various forms of visually aesthetic expressions?” [6]. In other words, computational aesthetics investigates how the modern technology helps arts. The technology serves to create tools that can enhance the expressive power of visual art and heighten human understanding of aesthetic evaluation, perception, and meaning.

The viewpoints and concepts expressed in this chapter, some of which bear deep roots in arts, science and technology, are stimulating and of great interest to anyone who is interested in both computing (or mathematics) and art. Technology has advanced to such a level that art and design become increasingly important and relevant to science and technology, whereas the importance of our technical knowledge is decreasing. This trend will continue and the argument is indirectly supported by a recent discovery that the Internet and search technology are changing our brain and also how we think, since we no longer need to memorize, just need to know how to search to find information we need.

1.

P. Fishwick (Ed.) Aesthetic Computing, MIT Press, Cambridge, 2006.

1906

Encyclopedia of Human-Computer Interaction

2. R.F. Malina, A Forty-Year Perspective on Aesthetic Computing in the Leonardo Journal, in: P. Fishwick (Ed.) Aesthetic Computing, MIT Press, 2006, 43-52.

3. C.B. Price, From Kandinsky to Java (The Use of 20th Century Abstract Art in Learning Programming), ITALICS, Vo.6, No.4, October 2007, 35-50.

4. K. Zhang, From Abstract Painting to Information Visualization, IEEE Computer Graphics and Applications, May/June 2007, 12-16.

5. K. Zhang, Book Review: Aesthetic Computing edited by Paul Fishwick, MIT Press, 2006, ISBN 0-262-06250-X, Journal of Visual Languages and Computing, Vol.18, No.6, December 2007, Elsevier Science Inc., New York, 613-616.

6. K. Zhang, S. Harrell, and X. Ji, Computational Aesthetics - On Complexity of Computer-Generated Paintings, Leonardo Journal, MIT Press, 2012, Vol.45, No.3 (in press).

Aesthetic Computing

1907

26.20 Commentary by David J. Therriault How to cite this commentary in your report

David J. Therriault

© David J. Therriault

Dr. Therriault is an Associate Professor in the School of Human Development and Organizational Studies in the College of Education at the University of Florida.

He received his undergraduate degree in psychology from the University of New Hampshire and his M. A. and Ph.D. in cognitive psychology from the University of Illinois at Chicago. He was formerly a Postdoctoral Fellow at Florida ...

David J. Therriault David J. Therriault is a member of The Interaction Design Foundation In his chapter on Aesthetic Computing, Paul Fishwick has created a very accessible and compelling argument for applying embodiment to human-computer interfaces. Fishwick’s case is built by examining the different levels of presence in

1908

Encyclopedia of Human-Computer Interaction

embodied examples. Can you imagine yourself in the landscape of a painting, or virtually manipulating objects while doing math, or becoming a character in a textually described world? Indeed you can, and more importantly there may be learning benefits to doing so. Fishwick provides rich examples of the use of embodiment in software (e.g., a steampunk obesity machine crafted from barrels of water to explain dynamic systems) that provide us with a glimpse into the potential of aesthetic computing. In the balance of this commentary, I briefly discuss the cognitive history and my own experiences researching embodiment, and argue that Fishwick’s insights provides readers with a true glimpse into the future of programing.

Traditional views of cognitive psychology assume that information processing makes use of abstract symbols. Since the 1950’s, the manipulation of abstract/amodal symbols has become the cornerstone of theories examining memory, reading, and thinking. Let us take as an example the psychology of reading. Kintsch’s (1974, 1998) influential research on reading comprehension was built from propositions (i.e., abstract idea units). Kintsch also provided evidence for the psychological reality of propositions, largely shaping our current understanding of what happens in our minds when we read. However, there was always a slice of representation missing from our reading experiences (often referenced as the symbol grounding problem)--a sense of presence or embodiment. How do we explain how propositions acquire meaning, how do we truly experience what we have read? In many ways, Fishwick is tackling the same symbol grounding problem (but within a programming framework).

Embodiment most simply put is attempting to understanding mind through the experiences and perceptions of our bodies (e.g., perceptual symbols, or grounded cognition). Interestingly, research examining how the body can influence our understanding has been most prominent in discourse psychology, specifically text comprehension. Researchers such as Barsalou (1999), Glenberg (1997), Lakoff and Johnson (1999), and Pecher & Zwaan (2005) have all argued that an embodied approach to cognition may have advantages over traditional views of mental representation.

Aesthetic Computing

1909

My own introduction to embodiment came during the earlier 2000’s working as a post doc with Rolf Zwaan. It was a wonderful experience, one were we spent the bulk of our time talking about how we would test the links between cognition and action, over good coffee or even better scotch, and see these ideas come to life in the lab. Prior to starting my position, other faculty members had cautioned me that researching grounded cognition was “kooky” and that attention on the topic would soon pass.

Since that time, there have been a host of compelling demonstrations of the psychological reality of embodiment. For example, we know that when listeners hear a story with their eyes closed they move them as if viewing the story in the real world (Spivey, Richardson, Tyler, &Young, 2000), that areas of the brain employed when doing a physical task are the same used when reading about that task (Feldman & Narayanan, 2004), and that even our judgments of morality can influence our perception, such as the perceived level of light in a room (Banerjee, Chatterjee, & Sinha, 2012). Our own research (Kaschak et al. (2002)) provided evidence that the perception of motion makes use of some of the same neural machinery needed to understand a verbal description of motion.

Research exploring an embodied view of cognition continues to flourish but there is currently no unified theory. Most researchers strongly advocate for either a symbolic or embodied view of cognition, but the field is moving away from this dichotomy. For example, Louwerse (2007) argues for exploring the relative contribution of both symbols and embodiment. Fishwick’s chapter represents a truly novel approach to embodiment: applying it to create a better understanding of such things as system dynamics, number sense, or programming. This view is also refreshing, in that, embodied computing isn’t intended to supplant the rich symbols tradition in coding. But embodiment may positively augment how we interact with computers in the future, and who wouldn’t want that?

In sum, I find Fishwick’s chapter a successful venture and one that has implications beyond undergirding (forgive the pun) the study of aesthetic comput-

1910

Encyclopedia of Human-Computer Interaction

ing. Programming becomes a more attractive domain for folks in my discipline to continue the study of mental representation.

26.20.1  References ff Banerjee, P., Chatterjee P., & Sinha, J. (2012). Is it light or dark?

Recalling moral behavior changes perception of brightness. Psychological Science, 23 (4), 407-409. ff Barsalou, L. W. (1999). Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577-660. ff Glenberg, A. M. (1997). What memory is for. Behavioral and Brain Sciences, 20, 1-55. ff Kaschak, M. P., Madden, C. J., Therriault, D. J., Yaxley, R. H., Aveyard, M., Blanchard, A., & Zwaan, R. A. (2005). Perception of Motion Affects Language Processing. Cognition, 94(3), B79-B89. ff Kintsch, W. (1974). The representation of meaning in memory.

Hillsdale, New Jersey: Erlbaum ff Kintsch, W. (1998). Comprehension: A paradigm for cognition.

New York: Cambridge University Press ff Lakoff, G. & Johnson, M. (1999). Philosophy in the flesh: The embodied mind and its challenge to Western thought. New York: Basic Books. ff Louwerse, M. M. (2007). Symbolic or embodied representations: A case for symbol interdependency. In T. Landauer, D. McNamara, S. Dennis, & W. Kinsch (Eds.). Handbook of latent semantic analysis (pp. 107-120). Mahwah, NJ: Erlbaum. ff Pecher, D. & Zwaan, R. A. (Eds.) (2005). Grounding Cognition: The role of perception and action in memory, language, and thinking. New York: Cambridge University Press.

Aesthetic Computing

1911

ff Spivey, M. J., Richardson, D. C., Tyler, M. J., & Young, E. E. (2000).

Eye movements during comprehension of spoken scene descriptions. Proceedings of the Twenty-second Annual Meeting of the Cognitive Science Society, (pp. 487-492), Erlbaum: Mawhah, NJ.

26.21 References Anderson, Kenneth M., Taylor, Richard N. and Whitehead, E. James (2000): Chimera: hypermedia for heterogeneous software development environments. In ACM Transactions on Information Systems, 18 (3) pp. 211-245 Audi, Robert (ed.) (1999): The Cambridge Dictionary of Philosophy. Cambridge University Press Barsalou, Lawrence W. (2010): Grounded Cognition: Past, Present, and Future. In Topics in Cognitive Science, 2 (4) Beck, Dennis, Fishwick, Paul A., Kamhawi, Rasha, Coffey, Amy Jo and Henderson, Julie (2011): Synthesizing Presence: A Multidisciplinary Review of the Literature. In Journal of Virtual Worlds Research, 3 (3) Bimber, Oliver and Raskar, Ramesh (2005): Spatial Augmented Reality: Merging Real and Virtual Worlds. A K Peters, Ltd Biocca, Frank (1997): The Cyborg’s Dilemma: Progressive Embodiment in Virtual Environments. In Journal of Computer-Mediated Communication, 3 (2) Bolter, Jay David and Grusin, Richard (2000): Remediation: Understanding New Media.

The MIT Press Bowman, Doug A., Kruijff, Ernst, LaViola, Joseph J. and Poupyrev, Ivan (2004): 3D User Interfaces: Theory and Practice. Addison-Wesley Professional Brown, John Seely, Collins, Allan and Duguid, Paul (1989): Situated Cognition and the Culture of Learning. In Educational Researcher, 18 (1) Burke, Joseph (1943): Hogarth and Reynolds: A contrast in English art theory. Oxford University Press Buxton, Bill (2008): The Role of the Artist in the Laboratory. In: (ed.). “Meisterwerke der Computer Kunst”. pp. 29-32

1912

Encyclopedia of Human-Computer Interaction

Card, Stuart K., Mackinlay, Jock D. and Shneiderman, Ben (eds.) (1999): Readings in Information Visualization: Using Vision to Think. Academic Press Cavallaro, Dani (2001): Cyberpunk & Cyberculture: Science Fiction and the Work of William Gibson. The Athlone Press Cockton, Gilbert (2012). Usability Evaluation. Retrieved 6 June 2013 from [URL to be defined - in press] Dede, Chris (2005): Planning for Neomillennial Learning Styles Neomillennial Learning Styles and Mediated Immersion. In EDUCAUSE Quarterly, 28 (1) pp. 18-21 Devlin, Keith (2006): The Useful and Reliable Illusion of Reality in Mathematics. In: Towards a New Epistemology of Mathematics. Workshop at GAP.6 Freie Universität Berlin 2006.

Dewey, John (1934): Art as Experience. Perigee Trade Diehl, Stephan (2007): Software Visualization: Visualizing the Structure, Behaviour, and Evolution of Software. Springer Dieterle, Edward, Dede, Chris and Schrier, Karen (2007): “Neomillennial” Learning Styles Propagated by Wireless Handheld Devices. In: Lytras, Miltiadis D. and Naeve, Ambjorn (eds.). “Ubiquitous and Pervasive Knowledge and Learning Management: Semantics, Social Networking and New Media to Their Full Potential”. pp. 35-66 Diffey, T. J. (1995): A note on the meanings of the term ‘aesthetic’. In The British Journal of Aesthetics, 35 (1) pp. 61-66 Dourish, Paul (2001): Where the Action Is: The Foundations of Embodied Interaction. MIT Press Eades, Peter and Zhang, Kang (1996): Software Visualization. Series on Software Engineering and Knowledge Engineering. World Scientific Pub Co Inc Feldman, Jerome (2006): From Molecule to Metaphor: A Neural Theory of Language. A Bradford Book Feldman, Jerome and Narayanan, Srinivas (2004): Embodied meaning in a neural theory of language. In Brain and Language, 89 (2) pp. 385-392 Fishwick, Paul A. (2012). Aesthetic Computing Class. Retrieved 1 January 2012 from University of Florida: http://www.cise.ufl.edu/~fishwick/ac/2012

Aesthetic Computing

1913

Fishwick, Paul A. (ed.) (2006): Aesthetic Computing. The MIT Press Fishwick, Paul A. (2007a): Customized Visual Computing: The Aesthetic Computing Method. In: Ferri, Fernando (ed.). “Visual Languages for Interactive Computing: Definitions and Formalizations (Premier Reference Source)”. pp. 425-435 Fishwick, Paul A. (ed.) (2007b): Handbook of Dynamic System Modeling. Chapman and Hall - CRC Fishwick, Paul A. (2002): Aesthetic Programming: Crafting Personalized Software. In Leonardo, 35 (4) pp. 383-390 Fishwick, Paul A. (1995): Simulation Model Design and Execution: Building Digital Worlds.

Prentice Hall Fishwick, Paul A. (2008): Software Aesthetics: From Text and Diagrams to Interactive Spaces Paul A. Fishwick. In International Journal of Arts and Technology, 1 (1) pp. 1-13 Fishwick, Paul A. (2012b): A Decade of Digital Arts and Sciences at the University of Florida. In Leonardo, pp. 211-216 Fishwick, Paul A. and Bertelsen, Olav W. (2002). Dagstuhl Seminar Report Number 348. http://www.dagstuhl.de/Reports/02/02291.pdf Fishwick, Paul A. and Park, Yuna A. (2008): A 3D Environment for Exploring Algebraic Structure and Behavior. In: Ferdig, Richard E. (ed.). “Handbook of Research on Effective Electronic Gaming in Education Set of 3”. Information Science Referencepp. 546-559 Fishwick, Paul A., Coffey, Amy J., Kamhawi, Rasha and Henderson, Julie (2005): An experimental design and preliminary results for a cultural training system simulation. In: Simulation Conference WSC, Proceedings of the 2010 Winter 2005. pp. 799-810 Fishwick, Paul A., Diehl, Stephan, Prophet, Jane and Lowgren, Jonas (2005b): Perspectives on Aesthetic Computing. In Leonardo, 38 (2) pp. 133-141 Fuller, Matthew (ed.) (2008): Software Studies: A Lexicon (Leonardo Book Series). The MIT Press Gershenfeld, Neil, Krikorian, Raffi and Cohen, Danny (2004): The Internet of things. In Scientific American, 291 (4) pp. 76-81 Ghezzi, Carlo and Jazayeri, Mehdi (1997): Programming Language Concepts. Wiley Goldin, Gerald A and Kaput, James J (1996): A Joint Perspective on the Idea of Representa-

1914

Encyclopedia of Human-Computer Interaction

tion in Learning and Doing Mathematics. In: Steffe, Leslie P., Nesher, Pearla, Cobb, Paul, Goldin, Gerard A. and Greer, Brian (eds.). “Theories of Mathematical Learning”. Routledgepp. 397-431 Grau, Oliver (2004): Virtual Art: From Illusion to Immersion (Leonardo Books). The MIT Press Greenfield, Adam (2006): Everyware: The Dawning Age of Ubiquitous Computing. New Riders Publishing Grossman, Lev (2009). Steampunk: Reclaiming Tech for the Masses. Retrieved 19 April 2012 from Times Magazine: http://www.time.com/time/magazine/article/0,9171,1945343,00. html Guhe, Markus, Smaill, Alan and Pease, Alison (2009): A formal cognitive model of mathematical metaphors. In: Proceedings of the 32nd annual German conference on Advances in artificial intelligence 2009. pp. 323-330 Hadamard, Jacques (1996): The Mathematician’s Mind. Princeton University Press Hassenzahl, Marc (2011). User Experience and Experience Design. Retrieved 6 June 2013 from [URL to be defined - in press] Hopcroft, John E., Motwani, Rajeev and Ullman, Jeffrey D. (2000): Introduction to Automata Theory, Languages, and Computation (2nd Edition). Addison Wesley Huff, Kenneth A. (2006): Visually Encoding Numbers using Prime Factors. In: Fishwick, Paul A. (ed.). “Aesthetic Computing (Leonardo Books)”. The MIT Press Ifrah, Georges (2002): The Universal History of Computing: From the Abacus to the Quantum Computer. Wiley Ikei, Yasushi and Ota, Hirofumi (2008): Spatial Electronic Mnemonics for Augmentation of Human Memory. In 2008 IEEE Virtual Reality Conference, 53 (14) pp. 217-224 Johnson, Mark (2007): The Meaning of the Body: Aesthetics of Human Understanding. University of Chicago Press Johnson, Steven A. (1997): Interface Culture. Basic Books Johnson, Mark (1987): Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason. University of Chicago Press Johnson, Eric and Adamo-Villani, Nicoletta (2010): A Study of the Effects of Immersion on Short-term Spatial Memory. In Virtual Reality, 71 pp. 1-44

Aesthetic Computing

1915

Kallay, William (2011): The Making of Tron: How Tron Changed Visual Effects and Disney Forever. William Kallay Kelly, Michael (ed.) (1998): Encyclopedia of Aesthetics. Oxford University Press, USA Kelly, Michael, Vesna, Victoria, Fishwick, Paul A., Moere, Andrew Vande and Huff, Kenneth (2009): The state of aesthetic computing or info-aesthetics: curated panel discussion. In: ACM SIGGRAPH 2009 Art Gallery 2009. pp. 43Kivy, Peter (ed.) (2004): The Blackwell Guide to Aesthetics. Wiley-Blackwell Knuth, Donald E. (1992): Literate Programming. Center for the Study of Language and Information Lakoff, George and Johnson, Mark (2003): Metaphors We Live By, 2nd edition. University of Chicago Press Lakoff, George and Johnson, Mark (1983): Metaphors We Live by. New York, Basic Books Lakoff, George and Johnson, Mark (1999): Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought. New York, Basic Books Lakoff, George and Nunez, Rafael (2001): Where Mathematics Comes From: How the Embodied Mind Brings Mathematics into Being. Basic Books Laurel, Brenda K. (1991): Computers as Theatre. Reading, MA, Addison-Wesley Publishing Lave, Jean and Wenger, Etienne (1991): Situated Learning: Legitimate Peripheral Participation (Learning in Doing: Social, Cognitive and Computational Perspectives). Cambridge University Press Lessig, Lawrence (2008): Remix: Making Art and Commerce Thrive in the Hybrid Economy.

Penguin Press HC, The Leymarie, Frederic Fol (2006): Aesthetic Computing and Shape. In: Fishwick, Paul A. (ed.). “Aesthetic Computing�. The MIT Presspp. 258-288 Lowgren, Jonas (2008). Interaction Design - brief intro. Retrieved 6 June 2013 from [URL to be defined - in press] Malina, Roger (2011). The Strong Case for Art-Science Interaction. Retrieved 19 March 2012 from thoughtmesh: http://vectors.usc.edu/thoughtmesh/publish/120.php Manovich, Lev (2001): The Language of New Media. The MIT Press

1916

Encyclopedia of Human-Computer Interaction

Mar, Raymond A. and Oatley, Keith (2008): The Function of Fiction is the Abstraction and Simulation of Social Experience. In Perspectives on Psychological Science, 3 (3) pp. 173-192 Marino, Mark C. (2006). Critical Code Studies. Retrieved 18 April 2012 from Electronic Book Review: http://www.electronicbookreview.com/thread/electropoetics/codology McLuhan, Marshall (1964): Understanding Media: The Extensions of Man. The MIT Press Mohr, Manfred (2011). Réflexions Sur Une Ésthetique Programmée. Retrieved 18 March 2012 from Bitforms Gallery, New York: http://www.bitforms.com/press-releases/4052011-manfred-mohr.html Nake, Frieder (2009): The semiotic engine: notes on the history of algorithmic images in Europe. In Art Journal, pp. 76-89 Negroponte, Nicholas (1996): Being Digital. Vintage Norman, Donald A. (2004): Emotional Design: Why We Love (Or Hate) Everyday Things.

Basic Books Papert, Seymour (1980): Mindstorms: Children, Computers and Powerful Ideas. New York, Basic Books Parsons, Thomas D and Rizzo, Albert A (2008): Initial validation of a virtual environment for assessment of memory functioning: virtual reality cognitive performance assessment test.

In CyberPsychology and Behavior, 11 (1) pp. 17-25 Petre, Marian and Blackwell, Alan (1999): Mental Imagery in Program Design and Visual Programming. In International Journal of Human-Computer Studies, 51 (1) pp. 7-30 Piaget, Jean (1950): The Psychology of Intelligence. Routledge Pine, B. Joseph (1999): The Experience Economy: Work Is Theater & Every Business a Stage.

Harvard Business Press Pine, B. Joseph and Davis, Stan (eds.) (1999): Mass Customization: The New Frontier in Business Competition. Harvard Business School Press Reas, Casey and Fry, Ben (eds.) (2007): Processing: A Programming Handbook for Visual Designers and Artists. The MIT Press Ritchie, Andrew C. (1968): English Painters, Hogarth to Constable. Ayer Co Pub Roth, Tina, Aiken, Peter and Hobbs, Scarlette (1994): Hypermedia Support for Software Development: A Retrospective Assessment. In Hypermedia, 6 (3) pp. 149-173

Aesthetic Computing

1917

Rotman, Brian (2000): Mathematics as Sign: Writing, Imagining, Counting (Writing Science). Stanford University Press Ryder, William (2009): A System Dynamics View of the Phillips Machine. In Proceedings of the 27th International Conference of the System Dynamics Society, Salomon, Gavriel (1990): Cognitive Effects With and Of Computer Technology. In Communication Research, 17 (1) pp. 26-44 Samuel P. Harn Museum of Art (2011). Art in Engineering. Museum Nights at the Harn Museum of Art. Retrieved 19 April 2012 from Samuel P. Harn Museum of Art: http://www. harn.ufl.edu/octmuseumnight.pdf Scruton, Roger (2011): Beauty: A Very Short Introduction. Oxford University Press Sfard, Anna (1994): Reification as the birth of metaphor. In For the Learning of Mathematics, 14 (1) pp. 44-55 Sfard, Anna and Thompson, Patrick W. (1994): Problems of reification: Representations and mathematical objects. In Proceedings of the Annual Meeting of the International Group for the Psychology of Mathematics Education North America Plenary Sessions, 1 Sherman, William R. and Craig, Alan B. (2002): Understanding Virtual Reality: Interface, Application, and Design (The Morgan Kaufmann Series in Computer Graphics). Morgan Kaufmann Slingerland, Edward (2008): What Science Offers the Humanities: Integrating Body and Culture (New Approaches to European His). Cambridge University Press Speer, Nicole K., Zacks, Jeffrey M. and Reynolds, Jeremy R. (2007): Human brain activity time-locked to narrative event boundaries. In Psychological Science, 18 (5) pp. 449-455 Stasko, John T., Domingue, John B., Brown, Marc H. and Price, Blaine A. (eds.) (1998): Software Visualization. The MIT Press Swade, Doron (2000): The Phillips Machine and the history of computing. In: Leeson, Robert (ed.). “A. W. H. Phillips: Collected Works in Contemporary Perspective�. Cambridge University Presspp. 120-126 Thompson, Patrick W. and Sfard, Anna (1994): Problems of reification: Representation and Mathematical objects. In: Proceedings of the Annual Meeting of the International Group for the Psychology of Mathematics Education - North America 1994, Baton Rouge, USA. Tractinsky, Noam, Katz, A. S. and Ikar, D. (2000): What is Beautiful is Usable. In Interacting with Computers, 13 (2) pp. 127-145

1918

Encyclopedia of Human-Computer Interaction

Tufte, Edward R. (1983): The Visual Display of Quantitative Information. Cheshire, CT, Graphics Press Turkle, Sherry (2004): How Computers Change the Way We Think. In Higher Education, 50 (21) Varela, Francisco J., Thompson, Evan and Rosch, Eleanor (1991): The Embodied Mind: Cognitive Science and Human Experience. Cambridge, Massachusetts, MIT Press ViĂŠgas, Fernanda B., Wattenberg, Martin, Ham, Frank van, Kriss, Jesse and McKeon, Matt (2007): ManyEyes: a Site for Visualization at Internet Scale. In IEEE Transactions on Visualization and Computer Graphics, 13 (6) pp. 1121-1128 Ward, Matthew O., Grinstein, Georges and Keim, Daniel (2010): Interactive Data Visualization: Foundations, Techniques, and Applications. A K Peters Ltd Wing, Jeannette M. (2006): Computational thinking. In Communications of the ACM, 49 (3) pp. 33-35 Yates, Frances A. (1966): The Art of Memory. University of Chicago Press Zhang, Kang (2007): Visual Languages and Applications. Springer

Aesthetic Computing

1919

About the author Paul A. Fishwick

© Paul A. Fishwick

Has also published under the name of: “Paul Fishwick” Personal Homepage: cise.ufl.edu/~fishwick/Welcome.html Current place of employment: University of Florida Paul Fishwick is Professor of Computer and Information Science and Engineering at the University of Florida, Director of Digital Arts & Sciences Programs, and CEO of Metaphorz, LLC. He has six years of industrial experience at Newport News Shipbuilding & Dry Dock Co (now: Northrup Grumman). and at NASA Langley Research Center in Virginia. He obtained the PhD from the University of

1920

Encyclopedia of Human-Computer Interaction

Pennsylvania, is a Fellow of the Society for Computer Simulation International, and is highly active in modeling and simulation research, with emphasis on virtual environments, scientific modeling and workflow environments, GPU-based simulation acceleration., social network acquisition interfaces, and model-based training environments for culture and medicine. Commonly used technologies include immersive visualization, portable devices, 3D animation, and mixed/augmented reality.

Your notes and thoughts on chapter 26 Record your notes and thoughts on this chapter. If you want to share these thoughts with others online, go to the bottom of the page at: http://www.interaction-design.org/encyclopedia/aesthetic_computing.html NOTES _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________ _______________________________________________________