A Virtual Synaesthesia - East of Eden by Holly Baldwin

THE EARLY DAYS OF A BETTER FUTURE A VIRTUAL SYNAESTHESIA EAST OF EDEN

The Greenwich Peninsula, or North Greenwich, comprises a piece of land bound on three sides by the River Thames to the east of the city of London. It has been the subject of development plans for the last 200 years, since its origins as marshland for farming and river industries, through the industrial heyday of east London in the 19th century, to the more recent residential and commercial development of London Docklands in the 1980s.

EAST OF EDEN GREENWICH PENINSULA

The Peninsula has gone through a variety of ownershipsâ&#x20AC;&#x201D; institutional and charitable organisations dominated up to the 19th century focussing on encouraging and promoting local industries. With the political and economic issues in England in the 1980s, the government sold off cheaply large areas of east london to a number of redevelopers with the hope of transforming the area,

and boosting public funds. However due to the areas poor connectivity to central London, a large quantity of public money was pumped into North Greenwich to provide new infrastructure such as the Blackwall Tunnel, the Jubilee Line Extension and North Greenwich Bus Station. This, alongside the public expense of the Milennium Dome, has caused huge political and economic unrest surrounding the peninsula, and many years of neglect and dereliction.

Today the area is owned by Knight Dragon and Greenwich Peninsula Regeneration Limited, and outline planning permission was gained in 2004 for 1,683 residential units, hotel and office space over six plots. The construction is now being carried out. The commercial and entertainment hub of Peninsula Square, with the O2 arena, and the initial residential development of the Millennium Village have already been largely completed as the first stage of the redevelopment.

The Greenwich Peninsula is bound on three sides by the River Thames, with the fourth side comprising the Millennium Village Development and the roads leading back into East Greenwich. The heart of the peninsula is Peninsula Square, the urban space linking to the O2 Arena, which holds the commercial uses and activity. In contrast the periphery is still under construction and left exposed to the river and road traffic. This makes it feel barren and open to the elements. Some industrial elements have been retained, such as various wharfs, jettys and the gas cylinder, which is enhanced by two art pieces by Anthony Gormley and Richard Wilson.

GREENWICH PENINSULA JOURNEY AROUND THE PERIPHERY

Initially I recorded sounds and movements in short films and timelapse sequences to document my journey around the Greenwich Peninsula, focusing on the aspects that caught my attention. These four clips show the journey around the periphery of the peninsula, from the sounds of the grass, the wind, the mechanical movement of the cable car and the dominance of construction work.

At the heart of the peninsula is Peninsula Square, the public open space that is enclosed by the bus/tube station to the west, the O2 dome to the north, Ravensbourne college and high rise buildings to the east and the round buildings housing marketing suites to the south. Peninsula Square is the heart of activity on the peninsula, with shops, restaurants and cafes spilling out into it. The buildings at the edges are covered in advertisements, screens and icons, with materials such as glossy plastic and mirrored glass that intensify this experience.

GREENWICH PENINSULA PENINSULA SQUARE

The sounds and images recorded here comprise the noise of people, music and advertisements, and the short timelapses show fast paced movement and light.

The visually intense experience in Peninsula Square is dominated by graphics, adverts and symbols of the interested parties and investors as well as residents around the square. The complex political and economic history of the peninsula in recent years, is omnipresent with the dominance of private investment and big brands, although the adverts try to remind us of the more human and personal motivations behind the development— ”village life in the city”, “bloom”, “celebration”. This combination of pre-conception and expectation of a place, or specific elements within it, running in parallel with the real-time experience of the space is a theme which interests me. The idea of the virtual city as being a ubiquitous element within the Postmodern city, that constantly affects and is affected by real-time experience.

GREENWICH PENINSULA FIRST CONTACT

Introductory Project about Greenwich Peninsula, introducing and beginning to explore project themes such as space through experience, relationships between the physical, emotional and virtual realities of space, Correalism and the ‘dynamics of continual interaction’ and architecture as the physical embodiment of ownership, identity and mass culture.

FIRST CONTACT THE EARLY DAYS OF A BETTER FUTURE

“Real life has become indistinguishable from the movies” “The culture industry remains the entertainment business” “The culture industry perpetually cheats its consumers of what it perpetually promises” “The triumph of advertising in the culture industry is that consumers feel compelled to buy and use its products even though they see through them” Text from “The Culture Industry: Enlightenment as Mass Deception”, by Theodor Adorno and Max Horkheimer

“postmodern hyperspace…has finally succeeded in transcending the capacities of the individual human body to locate itself, to organise it’s immediate surroundings perceptually, and cognitively map its position in a mappable external world…the incapacity of our minds, at least at present, to map the great global multinational and decentred communicational network in which we find ourselves caught as individual subjects” “The newer architecture therefore…stands itself as something like an imperative to grow new organs, to expand our sensorium and our body to some new, as yet unimaginable, perhaps ultimately impossible dimensions” Text from “Postmodernism, or the Cultural Logic of Late Capitalism”, by Fredric Jameson

INITIAL PROJECT THEMES FREDERIC JAMESON | MAX HORKHEIMER AND THEODOR ADORNO

“that euphoria or those intensities which seem so often to characterise the newer cultural experience” “strange new hallucinatory exhilaration”

“urban squalor can be a delight to the eyes when expressed in commodification”

Photographs from New York (top) and Berlin (bottom) that express the themes of Jameson’s Postmodern City—intensity, hallucinatory, commodification, photorealist, and extraordinary surfaces.

INITIAL PROJECT THEMES THE POSTMODERN CITY | PERCEPTION

“extraordinary surfaces of the photorealist cityscape”

“Perception is our sensory experience of the world around us and involves both the recognition of environmental stimuli and actions in response to these stimuli. Through the perceptual process, we gain information about properties and elements of the environment that are critical to our survival. Perception not only creates our experience of the world around us; it allows us to act within our environment.”

- http://psychology.about.com/od/sensationandperception/ss/ perceptproc.htm

The difference between what we actually ‘see’ and what we perceive is fascinating and complex. Scientists have tried to understand this through experiments into eye tracking during watching a scene (top) and computer simulations of vision (middle). The bottom image shows a representation of how much of a scene our eye can focus on at any one time. In order to understand the complexities of the Postmodern city, the process of human perception, in particular visual perception, is an important starting point.

INITIAL PROJECT THEMES PERCEPTION AND THE PERCEPTUAL PROCESS

Poster displaying research into the perceptual process, particularly relating to vision. >>

The concept of human perception is developed and extended into the realms of philosophy, psychology and metaphysics. Perception involves all the senses in order to make sense of the electro-magnetic waves, pressure fluctuations and chemical interactions that form objects and events, the physical basis of the world. Stimuli are assessed on the basis of modality, intensity, location and duration, and integrated or segregated according to the structural and spatial congruence of these different sense-data.

The brain then builds up a version of reality by combining these temporarily coincident sensory signals, the anchors or samples of our world, alongside our expectation for reality. Expectations are based on cognitive, cultural, and contextual information which allows us to create a unified and coherent perception of our surroundings.

However, given that our expectations are constantly changing and shifting according to feedback loops, it is possible to prime or influence people into having certain perceptions. Consider here optical effects, illusions, magic tricks etc. Further to this, with multi-modal perception, we can layer incongruous forms of sense-data together, and alter reality. Examples of this are ventriloquism, the Rubber Hand Effect, McGurk Effect, and Double Flash Illusion.

This being true, perception, and therefore ‘reality’ can be said to be a construct of consciousness, a realm that is only present when perceptual phenomena, bundles of sense-data that exist in time and space, are manipulated or constructed through perception. In metaphysics the ‘Measurement Problem’ led to a similar conclusion in the early 21st century, where a quantum particle transforms from the ‘fuzzy quantum world to the sharp reality of common experience’ only when measured.

“A fundamental conclusion of the new physics also acknowledges that the observer creates the reality. As observers, we are personally involved with the creation of our own reality. Physicists are being forced to admit that the universe is a “mental” construction. Pioneering physicist Sir James Jeans wrote: “The stream of knowledge is heading toward a non-mechanical reality; the universe begins to look more like a great thought than like a great machine. Mind no longer appears to be an accidental intruder into the realm of matter, we ought rather hail it as the creator and governor of the realm of matter.” (R. C. Henry, “The Mental Universe”; Nature 436:29, 2005) construct of human consciousness.

If the observer creates the reality, and all reality is a construct of consciousness, then changing the way we perceive our surroundings will in turn change our reality.

This research into human perception explores the idea of multimodal perception, where different senses combine to formulate a more complete idea of the world. These senses can aid one another or confuse and trick us into making decisions about the location of a noise, or the source of a particular sensation, constructing our own reality.

INITIAL PROJECT THEMES FROM MULTI-MODAL PERCEPTION TO THE ILLUSION OF REALITY

If multi-modal perception could include the combination of two different visual stimuli, instead of being restricted to one visual and one oral for example, then how could this alter our perception of the world. This collage combines a view of the Millennium Village with a photograph taken of an island in Indonesia, a previously perceived place.

INITIAL PROJECT THEMES FROM MULTI-MODAL PERCEPTION TO THE ILLUSION OF REALITY

In order to collect visual data from a scene, the eye flickers around fixating on certain areas to obtain ‘anchors’. The eye never takes data from the whole scene, instead filling in the blurry areas using expectations in the brain. These images explore breaking down a scene from the peninsula into particular fixations and blurred out areas. “Typically fixations last about half a second, so babies have on the order of 10 million glimpses of the world in their first year of life. That’s an enormous amount of information” Separating the image into a grid of fragments of visual stimuli, the image is then organised into squares that are in focus and squares that are blurred, in a similar, albeit simplified, version of sensory perception. By attempting to very simply map my thought process over the image I identified anchors and samples that could be used to recognise and decipher, demand attention or dismissal.

INITIAL PROJECT THEMES PERCEPTION | FRAGMENTS, FIXATIONS, COGNITION, ILLUSORY CONTOURS

Illusory contours or subjective contours are visual illusions that evoke the perception of an edge without a luminance or colour change across that edge. “The border of this invisible surface is known as a subjective or illusory contour, one that is not physically present in the stimulus but is filled in by the visual system” (Kanizsa, 1979) “Features include spots and edges, colours and shapes, movements and textures. These are all attributes that are not in themselves objects, but in combination they can define the objects we see. They are the building blocks of perception.”

This experimental collage looks at the potential of recreating the blurred out areas of the image with imagined and associated fragments and components, a reconstructed personal reality.

INITIAL PROJECT THEMES CONSTRUCTING REALITY

VIRTUAL AS CONCEPT

The virtual as CONCEPT; relationship between the human and the virtual world, the disorientation or distortion of space, mirrors, commodity, and the impact of simulation or representation in the ‘Hyperspace’ of the postmodern city. (Baudrillard, 1981). Baudrillard, Jameson, Deleuze, Adorno, Bergson

“Culture is dominated by simulation…the real has been bypassed…the image has supplanted reality” (Baudrillard, ‘The Precession of Simulacra”)

VIRTUAL AS MEDIA

The virtual as MEDIA; the digital image, gaming, cinema, the concept of the screen as a fixed viewpoint, new technology and theory of the post-perspectival.

The virtual as IMMATERIAL; cyberspace, data/information culture, intangible, communications culture.

Crary, Virillio, Freidberg

Virillio, Poster, McLuhan

“It presumes an ideal state in which one could redistribute one’s attention so that nothing would be shut out, so that everything would be in low level focus but without the risk of schizophrenic overload. It is a full reversal of the “searchlight” hypothesis of attention, for that kind of illuminating “selection” carries the risk of only finding only “what one already knows”. Freud sought to fashion himself into an apparatus capable of engaging a seemingly random sequence of signs (whether language, gestures, intonations, silences) and yet distracting from that disjunct texture some interpretative clarity. My interest here is not in any specific psychoanalytic implications, but rather in the larger cultural significance of a technique designed to impose a measure of cognitive control on an assimilable excess of information or on the apparently chaotic syntax of dreams. At stake is a calculated demotivation of attention, an automation of it, as a way of decoding the historical genesis of an indecipherable present in terms of an individual human subject”.

“An erosion of the physical, to the point where the loss of material space leads to the government of nothing but time...speed distance obliterates the notion of physical dimension”

(‘Suspensions of Perception’, Jonathan Crary, 2001).

The research into the construction of alternative realities brings back into focus the notion of the virtual. With the advancement of technology and the ‘second media age’ (Poster, 1996), the virtual has become part of the everyday, an ‘immaterial space’ where we spend increasing amounts of time, changing the way we negotiate and interact within architecture and within the city as a whole. This initial research into key theorists of recent years sets out the current thinking on the virtual, and its importance within architecture and the city.

CONSTRUCTING REALITY THE VIRTUAL CONTINUUM

VIRTUAL AS IMMATERIAL

(notes on Paul Virillio from ‘‘Rethinking architecture: a reader in cultural the-

ory”, Neil Leach (1997))

As a combination of the work to date, short scenes were developed aiming to represent the virtual continuum on Greenwich Peninsula. Using the idea of fixations and anchors each scene was constructed out of flickering fragments of visual data overlaid into film from the peninsula (top images). A second concept of the virtual is reflection and the simultaneous perception of two or more different view points. This was explored in short scenes using overlay and combination of film (bottom images).

CONSTRUCTING REALITY DISTORTION | FRAGMENTATION | IMPACT OF THE VIRTUAL ON THE ‘ACTUAL’

This still shows an exaggeration of the intensified experience in Peninsula Square, combining reflections and multi-perception with fragmentation and fixation present in visual perception.

CONSTRUCTING REALITY THE VIRTUAL CONTINUUM, GREENWICH PENINSULA

“Unreality

no longer resides in the dream or fantasy, or in the beyond, but in the real's hallucinatory resemblance to itself” - Baudrillard, Precession of Simulacra

The current virtual continuum on the Peninsula is centred around two dimensional overlays on to the existing environment that alter and enhance the experience but do not necessarily create new spaces within it. The separation between the ‘actual’ and the virtual is a highly theorised issue. Jean Baudrillard called this idea ‘hyperreality’: “a world of self-referential signs...the real has been bypassed, the image has supplemented reality”. In order to build up an idea of the hyperreality that already exists on the Peninsula, sensorial data from the physical environment across different time frames and perspectives is combined alongside fragments of virtual data and cognitive connections or interpretation. The initial project themes film will aim to explore the visible, invisible and subconscious qualities of our environment, transporting the viewer from the known reality, into an almost entirely virtual reality, that can be explored and inhabited in a different way.

CONSTRUCTING REALITY THE VIRTUAL CONTINUUM, GREENWICH PENINSULA

Initial sketch chronogram showing the journey of the film and the specific views to be created.

Developing the chronogram for this initial film which will outline the project themes of Hyperreality on the Greenwich Peninsula. The film will take the viewer on a disjointed journey around the periphery of the peninsula and into the sensorially-saturated Peninsula Square. “The effect of new media such as the Internet and virtual reality, then, is to multiply the kinds of realities one encounters in society” (Mark Poster ‘Postmodern Virtualities‘)

THE VIRTUAL CONTINUUM FROM PERIPHERY TO HYPERREALITY

By slowly introducing Peninsula Square as glimpses in the background of the beginning scenes, the film explores techniques to display the notion of virtual reality in the existing postmodern city – technological overlays, popular culture, fragmented images. These effects are intended to be subtle additions suggesting the layers of experience within the existing site, and the dominance of images and social network interfaces in contemporary media culture – however these need further interrogation and refinement. As the film moves into Peninsula Square, it is intended that a fully immersive virtual reality should be presented, displaying all the different virtualities that affect human perception of environment – cognitive, cultural, temporal, social, personal. The film also aims to begin to display signs of what virtual reality on the Peninsula could be in the future. As architecture moves from a physical manifestation into an increasingly virtual environment, material space becomes less relevant. The physical city becomes the framework within which to develop alternate virtual and augmented realities. In the film this is implied by the construction site and half finished nature of the planned development.

THE VIRTUAL CONTINUUM FROM PERIPHERY TO HYPERREALITY

The intensified Peninsula Square, saturated with virtual fragments from past events, commercial graphics and logos and social media posts, as well as film and photographs from different times and views around the peninsula. The dominance of the square as a fragment is unsuccessful however due to its association with the screen and its limitations of being a two dimensional object.

THE VIRTUAL CONTINUUM PENINSULA SQUARE

‘My 90-year old grandma tries the oculuc rift’ Youtube; Still from Inception; Audio visual installation ‘320° Licht’ by URBANSCREEN; Keiichi Matsuda for Dezeen.

Mapping Research from my initial study into the scientific process of perception of our environment to the implications of technology and new media on this concept of perception and on the postmodern city. My project begins to explore the increasing role of virtual reality in architecture and the city. This can be explored as a contributor to postmodern hyperspace as a further layer of perceptual information in the city, but it can also be seen as an effect of the new media age, directly influenced by postmodern image culture. Virtual or augmented reality is often considered only as an overlay on the existing environment, a recognisable interface dominated by the graphical norms of computer games, social media and popular advertising, or as a dislocated experience, imagined through computer screens or via headsets that are separate from the environment that is being conveyed. Using the concepts of sensory perception (from the gathering of sensory information, through transduction, to reception), alongside advances in new technologies, virtual reality will be able to manifest itself directly into the city, and enable us to interact with and transform all aspects of our environment in real-time and space.

PROJECT THEMES FROM THE PERCEPTUAL PROCESS TO VIRTUAL REALITY

Up to this point, the main project theme has been to combine the virtual and the actual, to create a new and enhanced experience of the Greenwich Peninsula. The virtual continuum should not only be an image overlay, but also a spatial experience within and around the current physical environment. The virtual as transformative, ephemeral virtual spaces that enhance and distort the existing city. Spatially and temperally augmented. Creating new spaces, to fill and sit between the physical environment.

THE VIRTUAL AND THE ACTUAL VIRTUAL SPACES AND THE IN-BETWEEN

Collage of a view into Peninsula Square with virtual spaces emerging within it.

THE VIRTUAL AND THE ACTUAL VIRTUAL SPACES AND THE IN-BETWEEN

Initial ideas about the experience of being within a virtual space; ambiance, colour, distortion, light.

THE VIRTUAL AND THE ACTUAL VIRTUAL SPACES AND THE IN-BETWEEN

Vortex structures as installations breaking into and out of existing spaces. Sophia Chang, Arnie Quinze and Art League Houston.

The spaces become vortexes that emerge from between the buildings and structures around Peninsula Square, expanding and spilling out into the public spaces. These initial model images show these almost parasitic structures which reflect and distort the environment. The periphery of the vortex melds into the actual with fragmented and blurred edges. The insides of the structures become intensified, multi-faced cavelike spaces.

THE VIRTUAL AND THE ACTUAL THE VORTEX | IMMERSION AND PERIPHERY

Virtual spaces that morph and flicker in the space between buildings, spilling out into Peninsula Square. The interaction between the virtual and the actual creates reflections and distortions of the physical world.

THE VIRTUAL AND THE ACTUAL THE VORTEX | INTERACTION AND DISTORTION

Using a combination of model views and still photography from Peninsula Square, the virtual spaces are superimposed into the existing built environment. The fragmented periphery of the space, as well as the distorted reflections in the convex walls produce interesting interactions between the virtual and the actual. Rather than image-led media, the virtual becomes colour coded spaces that morph and change through the spectrum as a reaction to the environment and the people within it.

THE VIRTUAL AND THE ACTUAL FRAGMENTS | COLOUR | MULTI-SENSORIAL

Collage showing the actual as viewed through the virtual. As you cross over the threshold out of the virtual space, it melds away, morphing the view back into the actual, or presenting a new virtual space in the near-distance.

THE VIRTUAL AND THE ACTUAL EXPANDED SENSORIUM THROUGH THE VIRTUAL CONTINUUM

Extract from “Cosmonaut Keep”, by Ken Macleod

The ambiance and experience of being inside the virtual spaces should be like nothing you can experience in the actual world, but instead a space of expanded sensorial experience—combining vision, sound, taste, touch and smell. The extract above describes the virtual continuum in Ken Macleod’s novel “Cosmonaut Keep”, which focuses on colour and complexity to create a space in which distance, perspective and shapes are ‘impossible to judge’. Due to its unique and often personal perceptive qualities, colour is an important device of the virtual continuum. It evokes moods and associations, attracts attention or dismissal, permits or absorbs light and affects all the senses separately and together.

EXPANDED SENSORIUM THROUGH THE VIRTUAL COLOUR PERCEPTION | MULTI-SENSORIAL SPACES

“Graced with Light”, by Anne Patterson

Colour has been used in art installations and architecture alike to create immersive experiences, or interruptions, whether subtle or radical, in the environment. The images above show works by James Turrell, Alejandro Mu単oz Miranda, Tomas Saraceno, Dennis Calvert, Mark Garry, Floto and Warner, Sean McGinness, Dave Gormon, and Anish Kapoor.

EXPANDED SENSORIUM THROUGH THE VIRTUAL IMMERSIVE, TRANSFORMATIVE, TRANSIENT SPACES

"When Liszt first began as Kapellmeister in Weimar (1842), it astonished the orchestra that he said: 'O please, gentlemen, a little bluer, if you please! This tone type requires it!' Or: 'That is a deep violet, please, depend on it! Not so rose!' First the orchestra believed Liszt just joked; more later they got accustomed to the fact that the great musician seemed to see colors there, where there were only tones." — Anonymous, as quoted in Friedrich Mahling, p. 230. (Translation by Sean A. Day.)

The virtual continuum as a multi-sensorial, colour-rich world, gives rise to the concept of synaesthesia. According to the Oxford Dictionary, Synaesthesia is: “The production of a sense impression relating to one sense or part of the body by stimulation of another sense or part of the body” It is a neurological condition, experienced by roughly 4% of adults as an integral part of everyday life. It is known to have a wide range of sense-combinations that include the senses as well as more conceptual notions and often include perception of colour. Some of the most common examples are: grapheme-colour (coloured words and numbers); sound-colour/chromesthesia (shapes and colours from sounds); number-form (mental map of numbers evoked); spatial-sequence (sequence of numbers seen as points in space); mirror-touch (experiencing tactile sensations

when observed on someone else); lexical-gustatory (tastes experienced upon hearing certain words). Synaesthesia is interesting for this project because it is the immaterial or ‘virtual’ representation of one sense experience by stimulation of another. For example chromesthesia allows the experience of colour to be evoked through stimulation of different sounds. The colours ‘seen’ by synaesthetes have been reported to be ‘in the minds eye’ but some have reported them to be ‘projected into space’. This ‘projector synaesthesia’ is a similar concept to the project so far. However the idea of using one sensory experience to directly create another is also an important concept. By filling the virtual spaces with specific sensory information, can they perhaps contain or emit information using a different sensory channel?

EXPANDED SENSORIUM THROUGH THE VIRTUAL THE CONCEPT OF SYNAESTHESIA

Wassily Kandinsky and Edvard Munch were reported to be synaesthetes, as well as Clare Morrall whose recent book “Astonishing Splashes of Colour” is characterised by her synaesthesia. Famous musicians such as Liszt, Mozart, and more recently Duke Ellington and Stevie Wonder are also reported to have chromesthesia.

Beginning to construct the visuals for the second film of the virtual realm on the Greenwich Peninsula. The film will combine footage and photographs from the peninsula with renders from the model. The initial scenes explore how the virtual spaces will begin to appear in the actual, distorting or enhancing the view, and inviting you into the space. The scenes then move into the virtual spaces, conveying the synaesthetic, immersive and multi-sensory, transformative, environments.

EXPANDED SENSORIUM THROUGH THE VIRTUAL VIRTUAL SYNAESTHESIA, GREENWICH PENINSULA

Beginnings of the final film chronogram. The film moves from the everyday on the peninsula, to glimpses of the virtual spaces, to interior views of the spaces themselves.

EXPANDED SENSORIUM THROUGH THE VIRTUAL VIRTUAL SYNAESTHESIA, GREENWICH PENINSULA

Chronogram for second film—’a virtual synaesthesia’,

EXPANDED SENSORIUM THROUGH THE VIRTUAL VIRTUAL SYNAESTHESIA, GREENWICH PENINSULA