Interactive Design

Behnaz Farahi, Neil Leach (eds.)

Towards a Responsive Environment

125 Active Matter Interview with Skylar Tibbits by Behnaz Farahi

137 Activating the Physical Toward Material Experience Design

Yasuaki Kakehi

151 Architected Morphing Matter: The Confluence of Geometry and Hidden Forces

Lining Yao, Harshika Jain

Part IV: Transdisciplinary Approaches

181 Cyberphysical Architecture in the Era of Digital Eclecticism: Transdisciplinarity and Conflation of Technological Milieus

Güvenç Özel

193 A New Assemblage: Interactive Art Experiences of a Human-Machine Reality

Weidi Zhang

203 Trans-Disciplinary Interactions Ruairi Glynn

213 Demystification: The Evolving Design Process of Interactive Architecture

D

235 By Way of Conclusion, we Have a Panel Discussion Between Philip Beesley, Guvenc Ozel, and Ruairi Glynn

Led by Behnaz Farahi

Welcome to Interactive Design: Towards a Responsive Environment. We are thrilled to share with you our fascination with the relationship between human beings and their environment. Through our exploration of various ways of using technology to create responsive environments that adapt to the needs, emotions and behaviours of their users, we have compiled this book as a culmination of research and projects by some of the most prominent practitioners and thinkers in the field of interactive design.

Our goal is to provide a theoretical and practical framework that designers and researchers can use to create meaningful and engaging interactive experiences. We draw upon various fields of design, including architecture, product design and urban design, to illustrate the key concepts and techniques.

This book is intended for anyone interested in the intersection of technology and design, including designers, researchers, educators and students. Whether you are a seasoned professional or a curious beginner, we hope that this book will inspire you to explore the possibilities of interactive design and create responsive environments that enhance the human experience.

We would like to express our gratitude to all our colleagues, collaborators, and mentors who have supported us in this research. We would also like to thank CSULB and Tongji for their generous support towards the production costs of this book.

We invite you to join us on this exciting journey and hope you will find this book informative, inspiring, and thought-provoking.

Behnaz Farahi and Neil Leach

A Brief History of Interactive Architecture

Behnaz Farahi, Neil Leach

on innovation and novelty, with the past serving largely as a repository of earlier experiments that help to define whether a contemporary project is itself original or not.

The avant-garde designs of Archigram during the 1960s depicted a vision of the future in which architecture is interactive and responsive not only to its surroundings but also to human needs and desires. In projects such as Plug-in City, The Walking City, and Instant City, Archigram envisioned the transformation of architectural spaces by embracing technology and ideas from cybernetics. In his article “Living 1990: Archigram Group” Warren Chalk (1967) notes, “The push of a button or a spoken command, a bat of an eyelid will set these transformations in motion – providing what you want where and when you need it. Each member of a family will choose what they want – the shape and layout of their spaces, their activities and what have you”. It was radical ideas such as these that launched the field of interactive architecture.

It may seem a little counterintuitive to attempt to write a history of interactive design, however cursory, since most aspects of computational design are premised on looking forward, rather than backwards. Indeed, in its present condition, research in interactive design and architecture seems to be governed by an experimental outlook premised

Moreover, as American media theorist Benjamin Bratton once commented, we are still in the “silent movies” era of computation (Leach, 2022). The history of digital design is indeed brief, but the history of interactive design is even briefer. One reason for this is that some of the technology intrinsic to interactive systems was not accessible until relatively recently. For example, computer-aided design software was popular in architectural offices from the 1980s onward, whereas Arduino, the microcontroller that has helped to make interactive design so accessible, was not introduced until 2005.

Nonetheless, the future is inextricably linked to the past. “Tomorrow”, as the saying goes, “today will be yesterday”. As such, past and future should be seen within a continuum, and an understanding of the past – however cursory –should be an essential component in addressing any futuristic venture, such as interactive design and architecture.

Cybernetics

The origins of interactive design in general and interactive architecture in particular can be traced back to the theory of cybernetics. The discourse of cybernetics emerged as a result of the Macy conferences from 1946 to 1953 (Pias, 2016). These conferences aimed to open up a new interdisciplinary field by connecting various disciplines, such as mechanical and electrical engineering, neurobiology, evolutionary biology, communications theory and psychology. Engineers, mathematicians and physiologists, such as

“Technology is the answer, but what was the question?”

(Cedric Price, 1966)

Norbert Weiner, Claude Shannon, Warren McCulloch and William Ashby, met and shared their ideas at these events (Pias, 2016).

In 1948, two years after the first Macy conference, Norbert Wiener coined the term “cybernetics” in his book Control and Communication in the Animal and the Machine (1948). Derived from the Greek word, kybernetike, meaning “governance”, cybernetics looks at information feedback in order to “steer, navigate or govern a goal” in any intelligent system including biological, social and mechanic processes. As Weiner (1948) explains it, “Cybernetics combines under one heading the study of what in a human context is sometimes loosely described as thinking and in engineering is known as control and communication. In other words, cybernetics attempts to find the common elements in the functioning of automatic machines and of the human nervous system, and to develop a theory which will cover the entire field of control and communication in machines and living organisms”.

As Dubberly and Pangaro (2005) point out, both living and non-living systems (machines) can have a purpose and therefore operate according to cybernetics principles. “Cybernetics focuses on the use of feedback to correct errors and attain goals. It has roots in neurobiology and found practical application during World War II in the development of automatic controls for piloting ships, airplanes, and artillery shells”. Having said that, the scope of cybernetics is broad; it attempts to embrace not only the way in which humans interact with machines and systems, but also the way in which humans interact with one another.

The next phase came from 1968 to 1975 when Heinz von Foerster (2003) formulated his theory of second order cybernetics as a way of moving beyond

the cybernetics of Wiener. It was an attempt to explore the role of the observer in the formation of systems through positive feedback. Although first order cybernetics was tied to the image of the machine, second order cybernetics more closely resembled organisms and biology by using noise as positive feedback. With positive feedback, a distorted message would reinforce the system’s organisation and sometimes help the system to self-organise. As von Foerster (2003) notes, “A brain is required to write a theory of a brain. From this follows that a theory of the brain, that has any aspirations for completeness, has to account for the writing of this theory. And even more fascinating, the writer of this theory has to account for her or himself. Translated into the domain of cybernetics; the cybernetician, by entering his own domain, has to account for his or her own activity. Cybernetics then becomes cybernetics of cybernetics, or second-order cybernetics”.

Although the science of cybernetics has had a significant influence on many researchers and practitioners, it is important to note that cybernetics has also faced considerable criticism. In The Cybernetic Brain: Sketches of Another Future (2011), Pickering responds to some of this. One of the criticisms is that cybernetics is often thought of as a militaristic science; responsible, for instance, for the development of autonomous anti-aircraft guns. Responding to this criticism, Pickering (2011) notes, “First, I think the doctrine of original sin is a mistake – sciences are not tainted forever by the moral circumstances of their birth – and second, I have already noted that Ashby and Walter’s cybernetics grew largely from a different matrix: psychiatry”. Another criticism of cybernetics concerns the notion of control, which has caused a certain degree of

suspicion. Pickering thinks that this is more a problem of modernity than cybernetics. Pickering (2011) answers, “Perhaps we have gone a bit overboard with the modern idea that we can understand and enframe the world”. He thinks that British cybernetics was not “a scientized adjunct of Big Brother”, but argues instead that we have to see it as an ontology of “unknowability” and “becoming” (2011).

At the heart of the theory of cybernetics was the notion of interaction. The English scientist, researcher and cybernetician, Gordon Pask, who had been introduced to Wiener’s ideas in the early 1950s, developed his Conversation Theory (1970) in an attempt to model how humans and machines learn and construct knowledge through interaction. As Usman Haque (2007) notes, “It was a framework that accounts for observers, conversations, and participants in cybernetic systems”. Pask was particularly interested in grounding conversation theory in authentically interactive systems that are not predefined but can develop a unique interaction with individuals. The bottom line in his theory is that this process is a circular one with potential for feedback correction and evolution. Otherwise, the system is just reactive. He goes on to differentiate between

single-loop and multiple-loop interaction. Haque (2006) explains, “Multiple-loop interaction does not depend upon complexity; it depends upon the openness and continuation of cycles of response. It also depends on the ability of each system, while interacting, to have access and to modify each other’s goals”. In his paper “The Architectural Relevance of Cybernetics” (reprinted in this volume) Pask (1969) goes on to explore the connection between cybernetics and architecture.

In this context, it is important to note that “interactive architecture” needs to be distinguished from simple “responsive architecture”. For a genuinely interactive architecture there has to be a two-way process or feedback loop. In other words, there needs to be a form of “interaction” and not merely a “reaction” or “response”. The important word here is “conversation”, within the context of conversation theory implying that the user and the device respond to one another through a form of feedback loop. For example, an appliance such as a refrigerator, where the light comes on automatically when the door is opened, could be described as merely “responsive” while many of the projects and papers presented in this book are examples of interactive design.

Fig. 1 Spring Dragon Tail by Philip Beesley Architect Inc. The permanent acquisition by the Shangdu Li corporation opened in October 2015 as part of the Interactive Watertown exhibition in Shanghai. Interactive systems within the Chun Long Tiao sculpture feature a new generation of proprioreceptive sensors, enabling internal feedback within networked Teensy microprocessor control systems.

Fig. 2

Light Sculpture by teamLab. Laser beams are used to create a collection of light planes constructing a spatial object where visitors will find themselves immersed in a complex, three dimensional reconstruction of space.

A teamLab Planets TOKYO

teamLab Planets TOKYO, a massive Body Immersive space consisting of a collection of immersive installations by teamLab, opened in Toyosu, Tokyo for a limited period of two years from July 7, 2018 to Fall 2020. Planets TOKYO spans across 10,000 m2, immersing the body entirely to dissolve the boundaries between the viewer and the work.

teamLab has been producing artworks with the Body Immersive concept for many years. teamLab Planets is a collection of installations in which the entire body becomes immersed, and the boundaries between the viewer and the work become more abstract. Body Immersive works immerse the body entirely through the use of digital technology, which allows for the artwork to be separated from the canvas that mediates it. This allows for continuous dynamic behaviour, visual phenomena, and the ability to transform the canvas. By doing so, the boundaries between the body and the work become ambiguous, thereby prompting people to think about their relationship with the world. As the artwork changes, due to your existence and that of others in the work, people dissolve into the artwork world and the relationship between people changes.

Here are three different exhibitions:

Floating Flower Garden

The flowers float up above people, and when the people move, the flowers descend again. The artwork space is completely filled with flowers, but as they float up, spaces with people at the centre are created. Because of this, people are able to freely wander around the three-dimensional flower mass space. If you encounter other people within the artwork, your space will connect with theirs and become one single space.

Zen gardens are said to have been created as a place for groups of Zen priests to carry out training in order to become one with nature. There is a Chinese Zen kōan (a question or story that is part of Zen priests’ theological training) called “Nansen’s Flower”. A man named Rikukô Taifu, while talking with Nansen, said: “Jô Hoshi says, ‘Heaven and I are of the same root. All things and I are of the same substance’. How wonderful this is! Nansen, pointing to a flower in the garden said, ‘People these days see this flower as if they were in a dream’.” In this work, people immerse themselves in flowers, becoming one with the garden. When people become one with the flowers and look at them, the flowers look back. People may truly look at flowers for the first time. The flowers blooming in the air are epiphytic orchids. Epiphytic plants are extremely common in the orchid family, and epiphytic orchids are able to grow without dirt by absorbing water from the air. The flowers in this artwork are alive, growing, and blooming with each passing day. Orchids are said to be the most recent plants to appear on earth. The ground was already covered by other plants, and orchids evolved to live on rocks and trees where other plants could

not survive. Orchids adapt and diffuse in a short period of time, and it is said that there are 25,000 to 30,000 kinds of wild species alone. It is believed that there are more types of orchid than any other plant. However, many species are endangered due to the loss of their habitat or overexploitation due to development. Orchid seeds are as fine as dust, with only immature embryos, no endosperm, and few storage nutrients. In nature, germination of the seed requires symbiosis with a specific fungus, and the symbiotic fungus supplies the nutrients. The seeds have no reserve for germination and cannot sprout themselves, which would seem to contradict the very notion of what a seed is. Seeds should also be a nutrient storehouse for seedlings to germinate, yet the last orchid species to appear on earth has abandoned this aspect. It causes us to wonder why evolution takes certain paths. Orchids are known to have co-evolved with certain pollen-carrying insects, and it is believed that they continue to evolve rapidly today. The time at which the flowers in the artwork space strengthen in aroma varies according to the time at which these partner insects are active. Because of this, the scent of the artwork space changes each moment between morning, day, and night.

The Infinite Crystal Universe

Pointillist painting uses an accumulation of distinct dots of colour to create a picture; here light points are used to create three-dimensional objects. This interactive artwork expresses the universe through accumulated light points that spread infinitely in all directions. People

can use their smartphones to select elements that make up the universe by dragging them and releasing them into The Infinite Crystal Universe. Each element released influences that of other elements and is influenced by the presence of people in the space. The work is created by people in the space and is thus continuously changing forever.

B Border Tuner

Rafael Lozano-Hemmer

“Border Tuner” is a large-scale, participatory art installation designed to interconnect the cities of El Paso, Texas, and Ciudad Juárez, Chihuahua, Mexico. Powerful searchlights make “bridges of light” that open live sound channels for communication across the US-Mexico border. The piece creates a fluid canopy of light that can be modified by visitors to six interactive stations, three placed in El Paso and three in Juárez.

Each of the interactive “Border Tuner” stations features a microphone, a speaker and a large wheel or dial. As a participant turns the dial, three nearby searchlights create an “arm” of light that follows the movement of the dial, automatically scanning the horizon. When two such arms of light meet in the sky and intersect, automatically a bidirectional channel of sound is opened between the people at the two remote stations. As they speak and hear each other, the brightness of the “light bridge” modulates in sync – a glimmer similar to a Morse code scintillation. Every interactive station can tune into any other, so for example a participant in Mexico can connect to the three US-based stations or to the other two in Mexico, as they wish.

“Border Tuner” is not only designed to create new connections between the communities on both sides of the border, but to make visible the relationships that are already in place: magnifying existing relationships, conversations and culture. The piece is intended as a visible “switchboard” of communication where people can self-represent. The project seeks to provide a platform for a widerange of local voices and an opportunity to draw international attention to the coexistence and interdependence between the sister cities that create the largest binational metropolitan area in the western hemisphere.

C Epiphyte Chamber

Philip Beesley Architect Inc.

Epiphyte Chamber, presented at the Museum of Modern and Contemporary Art, Seoul, was an interactive environment composed of hundreds of thousands of individual laser cut acrylic, Mylar, glass and aluminium elements. The suspended structural scaffold was composed of vertically aligned hollow diagrid acrylic and stainless-steel structural components, employing novel laser-cutting and thermal and mechanical forming processes that made tulip-shaped diagrid spar forms with attachment points permitting assembly into a dense, foam-like aggregate matrix. This densely repeating system created a hovering building system full of interlinking voids, akin to the spaces of sinuses or termite mounds.

Lining one side of the structure were arrays of tentacle-like lashes, organised in triple sets that extended the lower tips of the structural spars. These tentacles were derived from the “breathing pore” shape-memory alloy mechanisms

described earlier within this paper. Infrared proximity sensors were mounted on each tentacle, configured to provide reflex-like curling reactions. Reacting to the stimulus of viewers, composite chained reactions from adjacent mechanisms created motions that ripple out in peristaltic waves. Secondary reactions were configured to follow these primary reflexes, including the operation of highpower LED lights inserted within liquid-filled glass flasks positioned immediately above each tentacle cluster.

In parallel, separate chains of LED lights within glass vessels were positioned high around a central grotto-like space, each with their own IR proximity sensor configured to respond to the movement of viewers providing local reflex reactions and related ripples of responsive light. This separate grouping used shift-register microprocessor controls, permitting control of massive numbers of individual LED lights, and

employing pulse-width modulation envelopes for smooth transitions in rising and fading levels of illumination. Softly rolling clouds of delicate light were emitted around these centrally located chains.

Lashes extending from shape-memory alloy actuated mechanisms intermittently brushed against adjacent IR proximity sensors, creating cycles of self-triggered signalling and motion that propagated in turbulent cycles. When occupant activity heightened, the structures of this space became saturated with turbulence combining both physical triggers and behaviour caused by software-based communication. In a second series of liquid cells also located within the central area of the environment, organic batteries made of glass flasks holding vinegar with copper and aluminium electrodes generated tiny amounts of electricity. The trace currents generated by this battery system functioned as triggers for acoustic modules that produce subtle, drifting whispers of sound emitted from cycles of MP3 samples.

The organisation produced deeply interwoven fields of reaction that combined complex combinations of human interaction and emergent machine-based cycling, approaching “subsumption” in which nested reflexes are built within the system. Alongside the mechanized component systems, a wet system was introduced into the environment that supported simple chemical exchanges in the same way renewing functions of the human lymphatic system operate. Thousands of primitive glands containing synthetic digestive liquids and salts were clustered throughout the system. The adaptive chemistries within the wet system captured traces of carbon from the vaporous surroundings, translating this into inert carbonate precipitates located within the fluid cells suspended within the system. Engineered protocells – liquid-supported artificial cells that share some of the characteristics of natural living cells – were arranged in a series of embedded incubator flasks. Bursts of light and vibration triggered by viewer movements influenced the growth of the protocells, catalysing the formation of vesicles. The growth of skin-like layers growing within the flasks suggests the possibility of architectural environments clothing structures in generative skins.

Fragile Architecture: Notes for a New Model of Interaction

Philip Beesley

When the ancient Roman author Lucretius watched motes of dust quivering and darting within the sunbeams of his Roman window, he saw atoms play. Rivers of motion took the particles in laminar flows, bringing degrees of certainty into the sight of barely tangible things. Darting and wavering, the dust spoke of decay and loss; possibility; specious circumstance in flux: corrupted, damaged, and dying swerves. And a vague, shaded shift of life arising too – the rising semiquaver of living seeds. In the footsteps of Lucretius and earlier ancient thinkers seeing life arising from the chaos-borne quickening of air, water, and stone, a renewed architecture can follow the perception of life arising out of material.

Lucretius dwelt on the measurement of such flux. His writing speaks of an approximate geometry within curves shearing away from lines, calibrated within the infinitesimally small angle called a clinamen. A clinamen is the angle that occurs when a straight line meets the tangent of an arc. This essay employs the

conceptual terrain of the clinamen as a launch into the realm of hyperbolic and reticulated forms, pursuing the opposite of polarised boundaries. By following deliberately unstable, open boundaries, new architectural form-languages can offer delicacy and resonance, and, above all, fragility – the very antithesis of the firmitas that has defined Western architecture since Vitruvius uttered that famous paradigm. Architecture could be founded on adaptation and uncertainty where acquiring and shedding heat play in uneven cycles. Writ large, these qualities speak of involvement with the world. A building system using an expanded range of reticulated screens and canopies is implied, constructed from minutely balanced filtering layers that can amplify and guide convective currents encircling internal spaces. Within this vanguard city fabric, the thermal plumes surrounding clusters of human occupants offer a new form of energy that could be ingested, and diffused, and celebrated.

Historical Humanism has tended to encourage the opposite of these qualities. An ethic of strength and clarity prevails throughout much of the cultural history of our Western legacy, prioritising clear boundaries and the explicit negotiation of limits. With a legacy that has lasted through to the present day, Alberti (1988) wrote in his famous 1450 treatise De Re Aedificatoria:

Fig. 1 Piero Della Francesca, Flagellation of Christ, 1444

… just as the head, foot, and indeed any member must correspond to each other and to all the rest of the body in a living being, so in a building, and especially a temple, the parts of the whole body must be so composed that they all correspond one to another, and any one, taken individually, may provide the dimensions for all the rest … Beauty is a form of sympathy and consonance of the parts within a body, according to definite number, outline and position, as dictated by concinnitas, the absolute and fundamental role in Nature.

Architectural design within this prevailing tradition has tended to be based on distilled, meticulously balanced constructions framed by rigid surfaces supporting free human action. Piero della Francesca’s remarkable canvas “The Flagellation of Christ”, painted five years after Alberti published his thesis, provides a vivid depiction of social space that the architecture following the principles of De Re Aedificatoria might foster. Framed underfoot by a precisely rendered perspectival grid, a group of Florentine citizens is depicted deep in conversation. Nuance and grace underlies their bodily postures, their gestures, and their expressions. Certainly, the citizenship manifested here commands the highest respect. If the

viewer were to reflect only on the animated, highly involved depiction of conversation and reflection evoked by the sidelong exchanges shared by these figures, the painting might be taken as an epitome of human consciousness. Yet something is surely lost as well in this. Like a stripped stage set that has been cleared for action, the gridded stone floor under their feet makes the free placement of these figures possible. The floor stretches backward to include the obligatory religious scene of the Flagellation that defines the topic of the painting, and reaches starkly defined walls that bound the social space of the citizens, making a clearly bounded sanctuary. The sky and ground remain inert, silently framing the scene. Almost all expressive gesture within the composition is lateral, focusing attention on social and quite exclusive dimensions. This version of an ideal city requires secure walls and floors. The balance of the framed space is achieved by reduction and distillation.

As if a direct provocation to this kind of unified picture of quite exclusive citizenship, painted just a few years later within the same cultural circle, Giovanni Bellini’s 1480–85 “St. Francis in Ecstasy” depicted that famous saint standing emphatically outside the walls of a city. The latticework shelter that supports St. Francis relates to countless other “primitive huts” that speak to the origins

Fig. 2 Thermal imprint with liquid nitrogen erasure – high-definition stereo thermal cinematography. Liquid nitrogen is visible as radiating dark trails erasing the bright imprint of a dancer’s body on the stage floor of the ORA production set. Nitrogen is used for flash erasure of heat traces, clearing the stage in preparation for new recording.

Fig. 3 Arrayed dancers – high-definition stereo thermal cinematography. Varying patterns of thermal energy from inner metabolisms of dancers are vivid in this production still.

Fig. 4 Philippe Baylaucq, Stills from ORA, 2011. Dancers were recorded using high-definition stereoscopic thermography. An expanded physiology of traces leading into surrounding surfaces of floors and walls was revealed.

of architecture in cultural history. The painting depicts a primary architecture of intertwined branches. The simplicity of this open filter belies its prodigious reach far outward to soaring skies, massively deep underground, and richly embroidered fields of agriculture. Taken as a kind of primary architectural position, the occupied space described here seems based in the kind of radical exfoliation and ecstatic embrace of the myriad dimensions of the world.

Dissipative geometries offer a formlanguage that fulfils the vision evoked by Bellini. Following the need of sheltering enclosures to alternately shed heat and to cool, the kind of diffusive form-language embodied within snowflakes offers a paradigm of involvement with their surroundings. Rather than prioritising enclosed territory and maximum defence, a form like that of a snowflake seems instead to seek a maximum of involvement through its expanded perimetres, increasing its possible exposure and engagement with the world. This kind of optimum then seeks the utmost possible involvement with its surroundings with minimum defence. From a design perspective it promises an efflorescence of involvement and exchange between body and environment. Perhaps those formlanguages of radical exfoliation might inform involvement with the world, and for example, make more efficient batteries, or perhaps instruct the design of new biogenerators modelled after the reticulated interior membranes of mitochondria in human cells. At the scale of architecture, such principles might offer alternatives to the conception of compacted and distilled enclosing walls and roof surface. Instead, those surfaces could be reconceived as deeply reticulated heat sinks, and as layered interwoven membrane curtains that modulate the boundaries between inner and outer environments.

This kind of diffusive architecture pursues qualities similar to those found in veils of smoke billowing at the outer reaches of a fire; the barred, braided fields of clouds; torrents of spiralling liquids; mineral felts efflorescing within an osmotic cell reaction. Such sources are characterised by resonance, flux, and open boundaries. A new form language of maximisation and engagement implies that design may in turn embrace a renewed kind of stewardship. Such a role replaces the sense of a stripped, Platonic horizon with a soil-like generation of fertile material involvement with the world.

Hybrid design methods can support the composition of dissipative interactive architecture. Thermography, the visualisation of flows of heat and cold, offers a potent design tool. High-resolution helium-cooled thermal cameras such as those first used by the Quebec filmmaker and Hylozoic Series collaborator Philippe Baylaucq are emerging as tools for designers. Typical thermography reveals form by filtering out the atmosphere. Baylaucq’s cinematography goes further, however, implying an expanded range that includes atmospheric interactions between wall, floor and ceiling surfaces and their occupants. If “notch” filters applied to thermal detectors are tuned to precisely address gaseous material, minute air currents and concentrations of carbon dioxide and oxygen can be seen concentrating and dispersing around our bodies, extending in laminar flows along inner and outer surfaces of building envelopes. The series of images shown here (see Fig. 1) uses image analysis to reveal breath from a viewer passing over a filter frond. The thin tips of the frond quickly absorb the energy of the breath, passing it into the thickened centre of the filter which is fed by the dense surround of multiple tines. There is a tangible sense

Fig. 5 Philip Beesley, Sibyl, Hylozoic Series, Sydney Biennale, Australia, 2012. Sibyl integrates next-generation details from the Hylozoic series including massed ventilating bladders, scent-glands, and vibrating fields of filter mechanisms.

Fig. 6 ris van Herpen, Voltage haute couture, 2013. The studios of Van Herpen and Philip Beesley collaborated in developing diffusive frond fabrics used for the Voltage series, presented in Paris in January 2013.

Balance, Body Awareness, and Movement in Interactive Experiences

Elyne Legarnisson

“Bodily sensations greatly contribute to our minds being embodied and embedded in the world. It is how we experience our bodies and how our bodies experience the world”

(Eccleston, 2015).

As embodied cognition emphasises, our bodies inform how we think and perceive the world. We construct reality through body action in and on the world. Shusterman (2006) puts it as “our bodies are our indispensable tool of tools, the necessary medium of our being, perception, action and self-presentation”. Going further in the importance of body movement, Somaesthetics theory highlights links between body awareness, unusual movements and perception of the world. Shusterman says “to improve our body awareness, we need to move and furthermore move in ways that shifts us out of our habitual movements and response patterns”.

Today, the Western world is slowly welcoming the idea that body awareness and body movement play key roles in our physical and mental wellbeing. However, we increasingly evolve towards spatial contexts and interactions which tend to limit and control our moving bodies in our everyday lives while dispersing our

attention through constant visual stimuli. Most interactive experiences in virtual reality (VR) are no exception, as they primarily play off the visual sensory system, disengaging participants from their own bodies.

I therefore see a need to foreground the notion of embodiment in humanmachine and human-space interactions and ask: How could we imagine interactions encouraging body awareness and movement?

It is this question that I explored through my work, (un)Balance (Figs. 1 and 2). (un)Balance is an interactive XR (extended reality) experience which provokes its participants to move in unusual ways through raising body awareness and shifting world perception. It does so by playing on the sense of balance in different ways.

Balance, Body Awareness, and World Perception

Balance is the action of “maintaining postural equilibrium _ centring and maintaining the body’s mass in relation to gravity” (Eccleston, 2015). Through balance, we control our bodily positions while moving; whether it be walking, running, or dancing. It is a continuous and automatic activity which accompanies every movement of the body. It allows us “to take a stand in relation to the world” (Lopez et al., 2008). The perception of balance is achieved by processing sensory input from the visual, vestibular and proprioceptive and haptic systems and their cortical integration. Proprioception, which is first described by Sherrington (1906) as “the ability to feel the position of the body in space and limbs of the body in relation to the rest of the body” also plays a key role in our perception of balance.

The perception of balance therefore relies on information regarding the body as well as the world, and their respective stabilities. When attending to our balance, we are aware of our bodies and their relationship to the environment. However, this only happens momentarily as the sense of balance operates automatically. In most contexts it does not require constant attention (Eccleston, 2016).

With (un)Balance, I looked at whether one could provoke unusual movements and raise body awareness through interactions channelling and maintaining attention towards our sense of balance. The XR experience combines a VR layer with two analogue tools, a tilting platform and some apparel. The designs of each of those tools, as well as the way they interact with each other, use one or several principles aiming at bringing the participant’s attention towards their sense of balance. Those principles, which I will explain below, are: pushing balance to its limits, sensory augmentation of balance, and predictive processing.

Pushing Balance to its Limits

Being embodied also means being limited. While the sense of balance operates automatically, when we live experiences pushing our balance to its limits we attend to it. Those experiences can be negative, such as falling or feeling dizzy, or experiences associated with activities requiring a high level of balance control, such as surfing or ballet dancing (Eccleston, 2016). Therefore one way to raise body awareness is to bring attention towards the sense of balance through experiences pushing the sense of balance to its limits. This is a technique used for example in the practice of yoga, “where a

large part of the skill lies in sensing just how far to move into a pose. If one does not move far enough, there is no challenge for the muscles; however, going too far may result in pain or injury”. Therefore, the practice of yoga involves learning to listen to the sensations of the body while willingly pushing it to its limits (Shiffman, 1996).

In (un)Balance, the design of the tilting platform (Fig. 3), which we will describe in further detail below, is designed around this principle. Its primary aim is to push the participant’s balance to its limits.

Sensory Augmentation of Balance

Another approach to channelling attention towards balance is to augment its perception. For example in the apparel worn by Rebecca Horn (1970) in her performance “Unicorn”, the wearer describes the experience as “the mythical hybrid feels the pull of gravity and must concentrate on balance, pace and head position”. Here the apparel augments balance perception through amplifying sensations of weight and gravity, which brings the performer to attend to her sense of balance. Augmenting a sense can also be done through using different sensory input(s) than the one(s) usually involved in this sense. This process can also be referred to as feedback. For example, sound feedback can be used to augment proprioception (Kurihara et al., 2012).

As I will explain below, in (un)Balance, multiple sensory inputs (visual, auditory, and haptic) are used to augment the participant’s perception of balance.

Fig. 1 (un)Balance (London: Elyne Legarnisson, 2018). Showing three different prototypes of the apparel.

Fig. 2 Showing the physical layer of the experience, and in particular its tilting platform.

Fig. 3

Showing its apparel.

Fig. 4

Showing the virtual layer of the experience as seen from an audience point of view.

Active Matter

Interview with Skylar Tibbits

by Behnaz Farahi

Skylar Tibbits is one of the pioneers in the field of active, programmable matter. He is a designer and computer scientist whose research focuses on developing self-assembly and programmable materials within the built environment. Tibbits is the founder and co-director of the Self-Assembly Lab, and an Associate Professor of Design Research in the Department of Architecture at MIT. He is the author of Self-Assembly Lab: Experiments in Programming Matter (2016), editor of Active Matter (2017), and coeditor of Being Material (2019).

BF: Let’s start our conversation with the issue of how we engage with materials of the built environment according to their “affordances”. For instance, different materials with different properties could afford different actions; you can see through a glass versus you cannot see through concrete. As someone who has been a pioneer in the field of active programmable materials, could you perhaps start by telling us what “active matter” means to you? And how does active matter transform our interactions with the built environment? And, in your work do you engage with the notion of interactivity?

ST: I define active matter in the most literal way – matter that is active. That could be matter that is assembling itself, it could be matter that’s transforming, whether in its physical properties, appearance or physical behaviours, and so on – literally making materials active. And when I talk about programmable

materials, that’s maybe one step lower, addressing how we embed information and agency into materials. How do we give them whatever instructions are needed and tap into their inherent capabilities to sense, respond, transform, et cetera? So, oftentimes in our work we’re doing that without electronics, but that also could be done with various forms of electronics, sensors, batteries, and motors. From our perspective, we’re usually looking at the material properties, different forms of fabrication and that becomes both the information – like Braille or zeros and ones are A, C, T, and G in DNA – both geometry and material property as information – through those material properties that gives them agency to become active.

We held a symposium on active matter, and then published a book and it’s broad enough that now it includes an entire field. There are lots of researchers, designers, architects; but also scientists, physicists, material scientists, and all these other people working on matter that is active. But you can also look at the history of it. Before the Industrial Revolution, when there was craft-based production, matter was always seen as active and that was a positive thing. You could look at shipbuilding or alcohol barrels or Japanese joinery or metals or textiles or a number of other materials. The craftsperson really understood the inherent active nature of those materials and listened to the material properties, and it became a collaboration with materials and their agency. But then in industrial production in order to make it scalable, fast, repeatable, you can’t be spending the time listening to and collaborating with materials – you need to make static, stable, standardised materials. So, a lot of that craft-based knowledge and truly listening or collaborating with materials was stamped out, and it became about

production, scalability, and repeatability. If you look at wood, for example, that’s when we get two by fours, we get standardised lumber and that ignores all the anisotropic properties of wood. And so now we’re sort of shifting back, which is super exciting; we have scalable production and fabrication, but we can tap into the material properties again. Whether that’s because of printing or knitting or various forms of digital fabrication, we are inherently able to bring back that knowledge and collaboration with materials. So, now it’s an interesting moment to unleash those capabilities.

You were also asking about how we think about interactivity at the lab. On one hand, you could look at interactivity, being the human and the material, and so we’re always designing and collaborating with materials in that way. The designer is also embedding some information and that’s usually some goal or some performance or what we want to happen. What’s the general application or behaviour that we are looking for? And the material is then responding to the environment, responding to that information that’s been embedded and has its own agency. We’re going back and forth, listening, understanding, adjusting, maybe tweaking the environment, tweaking the parameters; we’re in this dance and that’s truly like an interactive collaborative moment between the designer and

the material. There’s also the non-designer interacting with the material. Whether we’re working on apparel or fashion or working on automotive or whatever, it’s the person using it, who didn’t necessarily design or didn’t embed information. That kind of interactivity is also interesting. How is their performance changing if it’s about some performance increase or how is their comfort changing? Or, how is their awareness changing? There’s an interaction there, which is pretty interesting. You could also talk about robotics and fabrication and materials. I think about the three-way relationship between designers, materials, and robots, in that the material is collaborating with the environment or responding and interacting with its physical environment in terms of its physical and material properties. The robot is great at repeatability and precision; it doesn’t get tired, and it’s able to do things precisely all the time. The human is great at creativity and flexibility and interacting between all three of those. And so, if you have those three working together, it’s a pretty nice relationship. Oftentimes, we start to shift roles, the human takes over the job of the robot or the robot takes over the job of the human. Architecturally, or even in manufacturing, we’ve gotten to the point where we think about materials as static traditionally and our job is to force them into place.

126

Fig. 1 Programmable Materials by MIT Self-Assembly Lab, Christophe Guberan, Erik Demaine, Carbitex LLC, Autodesk Inc. Fig. 2 Programmable Materials consist of material compositions that are designed to become highly dynamic in form and function, and yet they are as cost-effective as traditional materials, easily fabricated and capable of flat-pack shipping and self-assembly. These new materials include: self-transforming carbon fibre, printed wood grain, custom textile composites and other rubbers/plastics, which offer unprecedented capabilities including programmable actuation, sensing and self-transformation, from a simple material.

A lot of manufacturing is all about precision, it’s all about control, it’s all about making sure the material does what we want it to do. And it’s the same with structural engineering; it’s often about minimising any movement, stopping things from being active, stopping vibrations and resonant frequencies, stopping swelling, controlling the material; and that became the standard regime for architects to engineers and in manufacturing. Then there was the Louis Kahn comment, “What does the brick want to be?”1

That one, I think is the right question, but the wrong answer, like oh, the brick wants to be the arch, which is sort of a mistake. Actually, it was only Louis Kahn who wanted it to be in an arch – the brick didn’t necessarily want to be in an arch. And now I feel like we’re really getting into what the material is, what it truly wants to be and how we could celebrate that. Because maybe it’ll be different from what I expected and maybe that will be awesome, and we can truly collaborate based on what it wants to do.

BF: Most of the time, in your work, this type of active programmable matter senses environmental data; for instance, air, water, air pressure, or temperatures. Do you imagine they ever respond to other aspects in relationship to the human body or in

general; do you see the role of sensing beyond environmental sensing data? Is your work moving in that direction at all?

ST: Yeah, you’re totally right. Many of the materials are responding to moisture, temperature, sunlight, pressure, these kinds of fundamental activation energies that are around us every day. Part of that probably evolved because of very pragmatic things. When I was a graduate student, I was with Neil Gershenfeld’s lab, working on a DARPA programmable matter grant and that was all robotics. Over a decade ago, everyone was doing robots as programmable matter. And frankly, I wasn’t very good at that. These robots were failing all the time, and they’re super expensive, and they became clunky, and it was just always nonstop dealing with these issues. So, it was a very pragmatic decision to drop all the robots, just keep the programmability, keep the material behaviour, transformation, all that stuff and ask how can we do this without robots? And that became the very beginning of a lot of this work. What’s the most simple, fundamental, elegant way to embed information? In my master’s thesis that was through geometry. It was called Logic Matter, and it was purely shape as information, and that’s basically like Braille. But over time, it also was material properties. What are the other ways to activate something? Maybe

Fig. 3 Transformable Meeting Spaces by Self-Assembly Lab, MIT + Google + Michelle Kaufmann. These transformable woven structures can smoothly transform with lightweight and soft materials/mechanisms. A series of prototypes were built at 10cm, 1.5m, 3m, 6m and 20m demonstrating articulating woven structures for various applications.

1 Huxtable, Ada Louise. Seeking the Father, Finding the Architect. Accessed from https:// www.wsj.com/ articles/ SB107775941946739614

Fig. 1

Cypher, 2018, Image courtesy of Ozel Office

framework. This particular problem is especially prevalent in architects and artists working with machine learning and digital animation.

At Ozel Office as well as the University of California, Los Angeles IDEAS Program Technology Studio, where we conduct such transdisciplinary research with a highly technological focus, we aim to avoid such pitfalls by collaborating with innovative industries rather than working in isolation. By engaging experts in the fields of aerospace, entertainment, robotics, communication, transportation, fabrication, and software, we aim to not only benefit from and enhance their work but also seek alternative milieus to expand the reach of architectural practice. These collaborations range from contextual design exercises to hands-on research projects focusing on particular problems where spatial thinking and architectural experimentation can be useful.

Cyberphysical Architecture as Matter and Data

With such collaborations at work, the promises and perils of technology have been the main focus of my work in practice and academia. For the last ten years, pop-scientific discourse in technological singularity, transhumanism, artificial intelligence, and artificial life have become the dominant themes in my research, with the objective of exploring and formulating a renewed role for architecture as a technological interface that intelligently negotiates between human presence and the environment. Envisioning architecture as a technological actor and prosthetic enhancement to human influence in space was the main objective in my 2013 installation Cerebral Hut, where the user could control the physical

boundaries of the environment through an EEG brainwave interface. Scientific and science fiction writing provided the language to describe such an architecture; cyborgs, androids, AI and automata all became the useful terminology to describe such heightened modes of interaction between humans and spaces through design. Analysed in this regard, Cerebral Hut was described as a cybernetic system of human and machine, operating within the philosophical framework of transhumanist discourse to envision a new framework of spatial exploration. Since then, my work has evolved into a more critical approach toward architecture and technology. Politics of big data in the context of privacy and surveillance, perception of reality as a construct and a deep questioning of architecture as a material practice in the context of anthropogenic climate change have become the new dominant themes of exploration.

This formulation of architectural space as cybernetic assemblage not only calls into question the necessary skillset that an architect needs to acquire in order to design such highly technological spaces of human-machine networks, but also deeply challenges the agency that the architect has over the design of such spaces. These new and emerging technological paradigms force us to revisit the fundamental historic objectives of architecture. As a consequence of twentieth-century industrial production, buildings are no longer designed to be permanent structures made to last. Just like other objects we create, spaces are made for a limited lifecycle of consumption and are eventually transformed or discarded. Further challenging this notion, spaces no longer rely on material organisation to be conceived. Through the proliferation of sensor technologies working in tandem with digital

183

simulations and extended reality (XR), spaces can also be experienced in virtual milieus. These emerging notions of space as digital/physical hybrids prioritise motion, transformation and interaction over materiality, stasis, and permanence. In this context, architecture as an exclusively material practice becomes a wasteful and irresponsible act.

In addition, as demonstrated in Ray Kurzweil’s (2005) theory of technological singularity1 and Kevin Kelly’s (1995) definition of the “Biology of Machines”, the distinction between the manmade and the natural no longer exists: “the realm of the born – all that is nature – and the realm of the made – all that is humanly constructed – are becoming one”. Instead of subscribing to dystopian visions of machines taking over human civilisation, Kurzweil predicts a moment in history where human intelligence is artificially enhanced to create a form of intelligence that is not the direct consequence of natural evolutionary processes, but rather that of manifest destiny. Kurzweil describes this post-Darwinian moment necessary for overcoming our own shortcomings in adapting to the rapid technological progress. It is an intended and designed form of human evolution, both cognitive and material, triggered by our own inability to comprehend the tools we have created.

In parallel with the evolving definition of what is considered as intelligent and human, machine learning/ AI-based image production techniques call into question the authorship of emerging design trajectories. What we previously called “inspiration,” where a creative evolves an existing genre of work, is now being automatically iterated by algorithms. In this new type of image-production workflow, the architect and artist becomes a curator of datasets driven by cultural and aesthetic criteria.

Modes of Presence in Cyberphysical Environments

In the world of cybernetics, telepresence is defined as a means to be able to experience and initiate change in an environment through technology where the subject is not physically present (Minsky, 1980). In order for a telepresence scheme to be functional and convincing, it needs to bridge experiential visual and sensory simulations along with an accurate depiction of the context either through machine vision or with real-time virtual simulation where changes in both contexts are tracked and mirrored. In most scenarios, a telepresence scheme needs to synchronise robotic devices that can either control actuators or camera systems with immersive media (Kristoffersson et al., 2013). Therefore, telepresence is different from virtual presence in the sense that the former depicts a cyberphysical environment whereas the latter could be imaginary without any immediate real-world implications.

Contemporary applications of telepresence normally require an XR system in the form of virtual or augmented reality to “teleport” the user to the target context. In the target context, the objective could either be observational and experiential or active. In order for active schemes of telepresence to function, the robotic system should be operating under a particular goal, which could be in the context of human-robot collaboration for manufacturing objectives, could allow for human collaborations to flourish across geographic boundaries.

Since Marvin Minsky coined the term in 1980, the majority of telepresence schemes either focused on low-level teleconferencing applications or required expensive robotic body-rig systems that track and duplicate the position and morphology of the human motion. Deeply

184

Kurzweil extensively discusses his “Law of Accelerating Returns”, which claims that the speed of technological growth is exponential rather than linear.

rooted in theories of embodied cognition in its essence, telepresence is as much of an architectural problem as it is an anthropomorphological one (Brooks, 1995). The presence of the duplicated body in a target location is required to create a level of immersion and synchronicity with the original body so that the user can feel “present” enough to initiate change in that environment accurately (Adalgeirsson & Breazeal, 2010). This level of immersion requires a very precise depiction of the architectural features of both contexts. In this regard, an architectural system that accommodates telepresence needs to operate as a cyberphysical ecosystem of technologies.

Such fundamental re-questioning of architecture’s relationship to technology, politics, culture and reality created the backbone of the themes explored in my current research. Technologies such as gaming engines, machine learning, robotics, and virtual reality platforms are explored simultaneously and with equal footing, treated as cumulative technological ecosystems that work in synchronisation with each other, paving the way for a cyberphysical architecture. Based on this premise, two case studies below illustrate a spectrum of experiences in their formulation and interaction within the framework of cyberphysical environments in the context of presence: telepresence and virtual presence.

Telepresence: Cypher

Cypher is an architectural installation that creates an interactive experience through robotics, virtual reality, sensor interaction and machine learning. By combining a responsive soft robotic body with a virtual reality interface, Cypher establishes a bridge between the physical and digital worlds, collapsing them into

the same experiential plane by synchronising a virtual reality simulation with human-robot interaction.

Through an infrared sensor array and a LIDAR (similar to technologies in autonomous vehicles), the sculpture has an ability to detect the proximity of the audience and change its shape accordingly. The virtual reality headset tethered to the sculpture teleports the user to its interior, radically shifting the scale of experience from object to space. While in virtual reality, the user has the ability to change the shape of the simulation through natural hand gestures. As the user changes the shape of the virtual reality simulation, the robot moves real-time, aligning the physical and digital transformations. The relationship between virtual reality and robotics is further negotiated through machine learning algorithms, allowing the sculpture to develop natural motions by learning to predict the way in which people are interacting with it. The AI component allows for the sculpture to get more “intelligent” the more it is exhibited, using the number of interactions it has with the audience to cumulatively shape its motion and behaviour through time. Through the synthesis of these multiple technologies, Cypher challenges the notions of what is real vs. virtual, allowing the viewer to travel between multitudes of realities simultaneously.

The sculpture is built through a combination of multiple digital fabrication techniques. Mounted to an aluminium t-slotted frame that is assembled through unique 3D printed steel joints, there are 36 individual inflatable soft robotic clusters. Each cluster is attached to a computer-controlled solenoid valve. The lower part of the sculpture is made of carbon fibre-infused 3D printed panels. Spanning between the soft robotic clusters and 3D printed carbon fibre pieces are large-scale silicon panels. These

panels are actuated by five computercontrolled linear actuators to create an overall mass deformation. The computer and the air compressor are located at the centre of the aluminium frame, as well as additional physical computing components. The computer is responsible for not only controlling all physical computing components such as relays, actuators, solenoids, and additional IOs2, but also it synchronises the virtual reality content with the robotic components real-time. A LIDAR attached to the top of Cypher detects the presence of any object or person up to a 40-metre range. Running on a custom-made software, the LIDAR collects and stores periodic point cloud data from its environment. The software not only uses this data to change the overall mass of Cypher based on the proximity of the people around it, but also has integrated machine learning so that Cypher can develop more natural motion patterns through time. The same software is also used to “evolve” the geometry of the virtual reality scenes. In addition, Cypher has an array of infrared sensors embedded in its silicon skin. These sensors have a range of 30 centimetres, and allow for a more intimate interaction with the sculpture. Each sensor directly controls a solenoid, which feeds air into the individual soft robotic silicon clusters, allowing the clusters to pulsate.

The helmet is made up of a combination of 3D printed carbon fibre thermoplastic and silicone, fabricated through the same process as the sculpture. The helmet inflates and deflates due to the actions triggered by the user in the virtual reality environment, fusing the user into the spectacular motion of the sculpture. A virtual reality headset with inside-out tracking is integrated into the helmet that is tethered to the sculpture. This setup allows for the user to be

“teleported” inside Cypher, radically shifting the scale of experience. Through this virtual reality interface, Cypher blurs the boundaries between architecture, sculpture and fashion, allowing them to be experienced interchangeably.

The gaming engine Unity is used in order to synchronise all the virtual reality, physical computing and additional custom software. This approach allows the computational system to develop behaviours. This method provides a platform to collapse physical and virtual actions into a streamlined interface, creating a continuity of experience between the digital and physical worlds. The experimental machine learning app allows for Cypher to process and make decisions on proximities of multiple users. Proximity and speed of the user are taken into consideration to smoothen out reaction speeds and distances of linear actuators and inflations. With this combination of multiple technological systems working seamlessly, Cypher exists simultaneously in the digital and the physical worlds. It has an ability to respond to changes in its environment both as simulation and as material. By merging the worlds of virtual reality and robotics, Cypher has an ability to translate concepts and experiences that are traditionally seen as opposite domains: architecture vs. sculpture, object vs. space, digital vs. physical, real vs. virtual, visual vs. tactile, machine vs. organism.

Virtual Presence: Deep City

In the world of surveillance capitalism designed to collect and monetise data, digital representations of ourselves and our environments are harvested by closed-circuit television (CCTV) cameras, drones, Internet bots and social media platforms for social, economic, and

186

Fig. 2

Deep City, 2019, Image courtesy of Ozel Office.