Journal

J Multimodal User Interfaces DOI 10.1007/s12193-011-0074-4

O R I G I N A L PA P E R

Art at the edges of materiality Ina Conradi

Received: 14 April 2011 / Accepted: 2 November 2011 © OpenInterface Association 2011

Abstract This paper discusses digital artwork that through the act of creation investigates and integrates emotive subjective abstract imagery in digital and traditional forms.

The areas covered are the art of experimental nonnarrative animation, 3D stereo and large-scale algorithmicbased painting and prototyping for interactive and digitally fabricated art installations and performances. The paper’s intention is to interpret a complex body of digitally generated artwork that attempts to integrate animation, fine art painting and scientific inquiry through artist practice as pursued through the process of academic research. The outcome of is collaborative works done at the Nanyang Technological University, School of Art Design and Media.

The overall aim is in better understanding of the fine art process and mutual influences that various and often opposed disciplines have in collaborative projects where emotion-based interaction is important.

Keywords Abstract · Art · Experimental · Painting · Pictorial · 3D stereo · Animation · Installation

1 Introduction In the art work under discussion, animation and digital painting methodologies investigate tangible edges, borders and interfaces of technology, materiality and space. The traditional concept of painted canvas is replaced by proposed immersive transformational media experiences (Fig. 1).

Fig. 1 Envisioning creating new pictorial space using 3D stereoscopy, experimental animation and traditional painting methodology

Following thematic progression from the last few years until recent all the art works presented in this paper have in common digital component with the very same goal to explore alternative methods of artistic expression, using available and emerging synthetic image creation technologies and their evocative possibilities for vivid, real and stimulating pictorial spaces.

The works continue to pose “the questions of material and immaterial, concrete and digital states” in art practice, and are focusing on how to translate virtual properties of digital media into tangible art outcome. This new art resides at the edges of virtual, digital, electronic materiality emphasizing psychological, invisible and poetical experiences [1].

1.1 Origins and inspirations: about abstract art and painting

I. Conradi ( ) School of Art Design and Media, Nanyang Technological University, Singapore, Singapore e-mail: inaconradi@ntu.edu.sg url: www.inaconradi.com

Regarding his creative work, the poet and art critic, Baudelaire stated, “I want to illuminate things with my mind and to project their reflection upon other minds, the sensation

J Multimodal User Interfaces

Fig. 3 Viewer immersing oneself into animated projected painting

Fig. 2 Jackson Pollock painting expressing motion and energy of his ‘inner world’

of spiritual and physical bliss, of isolation, of the contemplation of something infinitely great and infinitely beautiful, of an intensity of light which rejoices the eyes and the soul until they swoon, to the sensation of space reaching to the furthest conceivable limits” [2].

Abstract art remains misunderstood by the majority of the viewing public. Most people, in fact consider it meaningless.

Yet around 1910, when groups of artists moved away from representational art towards abstraction, preferring evocative symbolic colors to natural color, signs to perceived reality, ideas to direct observation, there was never an outright dismissal of meaning. Instead, artists made an effort to draw upon deeper and more varied levels of meaning, the most pervasive of which was that of spiritual [3].

The early American painting style of late 1940s and 1950s, have no uniform stylistic traits on the surface. What brought them together is the same feverish energy and preference for the monumental scale and highly personal expression. They did not represent their emotions, but enacted them before the canvas [4], such that for some artists, the process of painting became the very subject of the painting [5].

Of particular inspiration are the drip paintings by Jackson Pollock (Fig. 2). Pollock created his very own unique style of painting by dripping, pouring, splattering, and flinging paint onto a huge canvas on the ground. This method enabled him to create very dynamic paintings from which the viewer can almost feel his action and force when looking at them. When the viewer encounters his large scale highly textured painting; they feel almost engulfed by it. Despite all the frenzy in his work, Pollock described that when he is painting, he has a general notion as to what he is doing. He

can control the flow of paint. “There is no accident, just as there is no beginning and no end.”

Today the same concept of ‘being and becoming part of the painting’ can be experienced through existing technology on an immersive level. New emotive and tangible contemplative space-awareness is created though phenomenal nature of the space formed between the observer and the picture plane, between the viewer and the viewed, using multiple disciplines and interactive scenarios (Fig. 3).

1.2 Experimental animation The evolution of abstract painting is simultaneous with that of moving pictures. In many ways, developments in film and digital technologies provided new approaches and opportunities to achieve the goals of the 20th Century Abstract Painters. The essence of which is not for the viewer to understand it, but to experience it.

Russian painter Wassily Kandinsky believed that colors and lines can be used like music to create moods without a subject. He discussed the problem of de-materializing the picture plane, urging artists to transform it into a sensation of indefinable space so that the spectator might experience the extension of “the dimension of time” [6].

With the use of video and computers, it becomes possible to directly link abstract forms and colors with their own musical orchestration [7]. Technology has provided opportunities to imagine entirely new approaches to non-objective forms and to liberate the imagination of artist. While many of the Abstract Expressionists and Action painters painted large canvases in order to encompass the viewer, film and animation projection makes it possible to engulf the viewer in cinematic fields.

Abstract painting and the moving image are very much related and the conceptual approach used in painting when applied to computer animation creates new imaginative aesthetics out of non-objective moving forms. Unlike still paintings, moving images have the added dimension of time. The painted composition is constantly changing, thus requiring additional planning and consideration on how the image will

J Multimodal User Interfaces

Fig. 4 Digital planning and visualization for installation featuring moving animated paintings done as a large scale simultaneous projections. Titled Emotion Study, installation consists of separate screens each corresponding to a particular emotion sequence from the film

Drawing on the ideas of the merging of digital painting, avant-garde film-making and sound cultures, the work under discussion aims to craft immersive, interactive and 3D animated installations using digital images, seeking innovative convergence of art and technology to transform spaces into novel experiences. Developed works aimed to be presented in a way that echoes the forms of traditional painting: transformation of a 3-dimensional space (‘canvas’) via the projected film (‘brush strokes and colors’). The result is a unique “scape” within which the viewer would be actively reconsidering preconceived idea of two dimensional ‘painting’. The very somewhat participatory, sensorial, individualized engagement that involves viewer is goal for the arts created. Works explored the following oversized image creation using algorithmic paint strokes, high resolution computer rendering techniques, (Fig. 7), digital prototyping and printing; imagery integrating animation and 3D stereo.

2.1 Internal External installation (Post Museum, Singapore 2008, 1000 sqm)

Fig. 5 ‘Emotion Study’ Large scale animated projection on the side of the National Gallery of Modern Art Museum Building, Inaugural Opening titled ‘Corporeal Kaleidoscope’, Attakkalari India Biennial 2011, Bangalore, India, 2011

evolve from start until end. At any point in time, the paused frame should be able to stand on its own as a complete and resolved painted composition [8] (Fig. 4).

With the further introduction of other disciplines into fine arts, painting can create new kind of pictorial space through use of large scale projection spectacle, incorporating architectural elements as a backdrop canvas. Recently screened at the Attakkalari India Biennial, ‘Emotion Study’ film was projected on the side of the National Gallery of Modern Art Museum Building as a large format artistic electric canvas (Fig. 5).

2 About the art work “The works will be structures but not sculptures. They would be painted but not paintings. They would be decorative but not decorations” [9].

The following compositions were done in 3D environment of Autodesk Maya using custom-made algorithm brushes.

(Fig. 6). Auto Desk Maya is a software commonly used in Feature Animation for modeling and animating as well as creating visual effects. The algorithm custom brush is used to “paint” in 3D space and was rearranged to create an aesthetic composition. During the working process, low resolution renders have to be made periodically to observe the progress and changes made to the brushes as it cannot be fully previewed real time within Maya’s viewport. The strokes appear in the forms of lines, which serve as are reference of its position and trajectory as opposed to an accurate representation of the final product. Due to software limitations, images generated with Maya paint effects is incompatible with Mental Ray renderer, and hence cannot benefit from lighting solutions such as Global Illumination & Final gathering.

The series of works began as a virtual planning and digital pre-visualization for public art proposals for site specific locations in Singapore (Fig. 7). Pre-visualization of artwork in both internal and external spaces is explored through the construction of 3-dimensional sets and photo space manipulation. Realistic pre-visualizations are achieved with 3D package Maya and integrated renderer Mental Ray, utilizing advanced lighting solutions and materials to achieve close to life renders. Accurate Spatial reconstruction of the selected locale is essential in planning and visualization of the artwork in space. The space/artwork relationship has to pass both aesthetic and practical considerations.

Based on these visualizations the first actual art installation was a site work titled Internal External and it was exhibited in the Post-Museum Gallery, Singapore (Fig. 8). A normal method of rendering was not possible due to extreme

J Multimodal User Interfaces Fig. 6 Autonomous and independent expressions of algorithm had enabled development of complex dynamic patterns. (a) algorithm preset brush, titled funMesh, changed the look (b) with altering attributes such as creation, segment length, width and texture and growth and behaviour, displacement and forces

Fig. 7 Public art proposal for Esplanade Exterior public sculpture ‘Natural Systems: Elixir of Redness’ tempered glass, exposed U-channel with concealed ground mounting 14 panels 1100 × 227 × 200 cm high resolution renders 160 cm × 227 cm, at 300 dpi solution for each panel would equate to 18899 × 26780 pixels, file size 1.41 GB

size of each composition, would exceed the limitations of Maya. Thus a batch render script is created in notepad to instruct Maya to slice up the render into multiple smaller tiles; these individual tiles would then be composited with Adobe Photoshop in post production.

‘Internal External’ is a multilayered environment combining video projection, 3D computer animation, and space modifiable backlit digital painting. Through an integrated design and fabrication practice, aim was in trying to create experimental, built environments to enhance and celebrate the potential for social interaction through sensation and physical engagement. Enhanced by light and augmented through complex patterns the exhibition space was reconstructed again in Japan into an intricate spatial environment consisting of tall backlit compositions one can travel through and explore (Fig. 9).

In these two art gallery installations, the goal of the work was to express an enthusiasm for emotive animation and tangible fabrication processes as they relate to the built installation both physically and poetically. For the exhibit purposes, the animation is in most cases followed with the actually fabricated digital painting and print prototype. The aim was to merge generated physical output based on the animated dynamics and moving images. Depending on the available resources, the outcomes are ranging from commercial and

Fig. 8 Internal External installation Post-Museum, Singapore, 2008.

This is an example of materializing virtual painting into print on a tangible substrate. Installation consists of tall backlit compositions, 14 panels each 100 × 227 cm, digitally printed on UV VUTEk ® Q-S2000 back-lit polycarbonate using AVS technologies. Size of installation is 100 sg m

custom-coated 2D fabricated surfaces and or 3D sculptures.

In that manner, the works explore properties, limitations, and site-specific scenarios associated with an image process.

While developing methods to extend the intertwined boundaries of image aesthetics, animation and digital code tactile quality, an immersive experience remains main component.

Using dreams as a conceptual motif, and adopting simulated natural phenomena such as flow and growth, the work explored organic and natural design leading the viewer thorough the aesthetic impression of a dream- like state sensibilities, feelings and experiences that go beyond language.

Inundated with light, color, and motion, paintings are visually translating invisible realities, the ephemeral and everchanging appearances of nature in flux. The overlapping and mirrored layers of complex imagery are engaging in the repetition of marking the void: experiential, spiritual and infinite.

J Multimodal User Interfaces

Fig. 9 Art installation Kyoto International Textile Center, Gallery Ex, Natural Systems: Elixir of Gold, Digitally generated still images on UV VUTEk ® Q-S2000 back-lit polycarbonate AVS technologies, 300 × 300 × 227 cm, Kyoto, Japan, 2009

2.2 On ‘Emotion Study’, experimental animation (running time 9.27 min, HD DVD, color, June 2010) The idea of fundamental study on perceptions of emotion provoking images originated within Co-Space research into cinematic storytelling with the real-time and rendered imagery. At the time, Emotion Study Survey was to assist in planning visual and behavioral representation of animated form. Production of digitally generated stills, photographs, and paintings depicting various emotional states proceeded. The study addressed the audience’s emotional perception towards abstract images in order to understand viewer’s reaction and emotion when they are exposed to non-representational images. It is based on Rosenberg and Ekman’s studies [10], and empirical tests designed to establish a coherence between expressive and experiential systems in emotion, in which they found that there was a high degree of temporal linkage and categorical agreement between facial expressions and self-report and an even stronger agreement for more intense emotional events. Thus, emotion theorists have utilized these results as solid evidence to introduce the self-report and cue-review to the subsequent emotion tests.

In another related study, the self-report method was used to collect the viewer’s data. Forty six out of one hundred participants fully scored all images [11]. Within these six basic emotions, the category of Happiness gains the highest consistency between normal viewers and creators, while the category of Disgust follows as the second highest constant emotion. It is inferred that the audience is able to discriminate between two polarized emotions even when they are exposed to purely abstract images without any textual instruction. Researchers also run a correlation test to disclose connections among different emotions, which (Fig. 10) shows a strong correlation between Fear and Disgust as well as Fear

Fig. 10 A circumplex model of affect. A circumplex model of affect illustrates of relations of feeling-related concepts that are places in a circular order and are contrasted as: pleasure-displeasure and arousal-sleepiness [12]

and Anger, while another pair of emotions having a correlated connection are Happiness and Surprise [12].

The result indicates that similar emotions within either category of positive or negative have a significant correlation. That is, a viewer is very likely to feel frightened when exposed to disgust images; likewise, the viewer may feel happy when something surprises him/her in a positive way.

Although the sample size was targeting a pilot result, the gender differences still can be seen as being among positive and negative categories.

While male viewers are more aware of the pair of Happiness and Surprise, female viewers are more sensitive about Disgust and Fear [13].

Over a hundred still images pertaining to the six different emotion categories: Happiness, Sadness, Surprise, Anger, Fear, and Disgust plus neutral were used putting the results achieved into the animations created.

For image generation research the diversity of stylistic methods of abstract painting, painterly automatism with intuitive authoring of imaginary forms, action painting, and color field painting with a highly articulated, psychologically based use of color. Kandinsky’s theories about the relationships between colors and emotions were also applied [14]. The resulting still frames rendered in Maya were then further layered to achieve more expressive abstract imagery reminiscent of Pollock’s ‘all-over’ drip paintings and works from the French Art Informel movement that employed a form of unconscious calligraphy to trace out the emotions the artist wishes to express (Fig. 11).

From these still images, experimental short sequences were developed and pieced together into a painted animated setting. A unique style of experimental animation is developed that aims to provide the viewer with a new aesthetic experience that is immersive and imaginative.

J Multimodal User Interfaces Fig. 11 Still images depicting the following emotions: Fear (on the left), Happiness (on the right)

3 Conclusion

Fig. 12 Part 1 Gloom from Emotion. Still frames from the ‘Gloom’, Animated paint slowly flowing down, from the top of the screen, while its details are constantly moving, then flowing down and dissolving away

The experimentation used Next Limit Real Flow, a fluid dynamics simulation graphics technology that gives the appearance of streaks of light and paint (Fig. 12). Some limitations of this method was also discovered, such as it could better mimic the appearance of paint when darker colors are used and the polygonal mesh must have more complexity to render a convincing form. The methodology developed in this project work around these limitations to create animation clips of a lush quality that may be put together into an abstract composition meant for large-scale projection.

The emotion ‘sadness’ was chosen to be the starting theme for exploration because it is associated with dark heavy colors and slow movements which can bring out the beauty in the evolving fluid forms. Rather than simply animating the splashing of fluid or ‘paint’ that is achievable to a limited extent with live-action movie photography, the focus was on the creation of a main sculptural form in the animation that would depict the targeted affective state [8].

These animated sequences are further developed to elicit the affectation of Gloom, Torment, Fear, Loneliness, and Lust (Fig. 12).

The challenge for the artist today is in presenting an artwork compatible to painting that is non-representational that defines itself within a physical space, and in doing so challenges the concept of pictorial space in twentieth century.

“An effective painting of today should present its space in such a way as to include both viewer and maker each with its own space intact. It is not that this experience should be literal it is simply that the sense of space projected by the painting should seem expansive enough to include the viewing and the creation of that space. Concerning contemporary painting today, the artist should strive to encourage a response to the totality of pictorial space: the space within and outside of the action of making the painting. The act of looking at a painting should automatically expand the sense of that painting’s space, both literally and imaginatively. In other words, the spatial experience of the painting should not seem to end at the framed edges or contained in box that is the picture plane” [15].

Further networking with interactive media, science and wide range of disciplines, painting can reassess its tools to be reinvented again. Study and art works described in this paper could be used both in emotion induction experiments and in emotion visualization and expression in multimodal interfaces. For example, the author has a collaboration project where the user’s emotions such as fear, frustrated, sad, happy, pleasant, and satisfied could be recognized from EEG in real time [13], and interpreted with art animations depicting the corresponding emotional states.

Acknowledgements – School of Art Design and Media, Nanyang Technological University, Singapore.

– Academic Research Fund Tier 1, Reference Number RG105/10, Funding by Ministry of Education, Singapore, 2007–2010.

– National Research Foundation, Singapore, Project Title: ‘Cinematics and Narratives: creating stories within real-time visual toolsets’, Asst. Prof. Mark Chavez, 2008–2010.

– Undergraduate Research Experience Program, URECA, Nanyang Technological University, Singapore.

J Multimodal User Interfaces

References 1. Gramelsberger G (2010) Heavy Matter: a laboratory study view.

In: Cologne AOMA (ed) Heavy Matter ISEA2010 Ruhr. Academy of Media Arts Cologne, Dortmund, p 29 2. Baudelaire C, Mayne J (1995) The painter of modern life and other essays. Phaidon, London 3. Tuchman M et al (1986) The spiritual in art: abstract painting 1890–1985. Abbeville Press, New York 4. Honour H, Fleming J (2005) A world history of art. Laurence King Publishing, London 5. Brown J, Frankenthaler H (1998) After mountains and sea: Frankenthaler 1956–1959. Solomon R. Guggenheim Foundation, Bilbao 6. Long RCW (1890) Expressionism, abstraction, and the search for utopia in Germany 7. Clearwater B (2009) Abstract cinema and technology. Available from: http://www.mocanomi.org/archive/images/abstractcinema_ onlinecatalogue.pdf

8. Conradi I, Khoo YH (2010) Reassessment of painting: fine art painting and experimental computer animation ADM09036. In: Proceedings of NTU URECA 9. Harrison C (1977) On color and abstract art: David Batchelor’s recent work. In: Sculpture: contemporary form and theory. Art & design, vol 55, p 39 10. Rosenberg EL, Ekman P (1994) Coherence between expressive and experiential systems in emotion. Cogn Emot 8(3):201–229 11. Mark C (2010) Access survey. Available from: http://www. visual-analysis.com/survey.html 12. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39(6):1161 13. Liu Y, Sourina O, Nguyen MK (2010) Real-time EEG- based human emotion recognition and visualization. In: 2010 int conf on cyberworlds, Singapore, pp 262–269 14. Kandinsky W (1977) Concerning the spiritual in art. Dover, New York 15. Stella F (1986) Working space. Harvard University Press, Cambridge