LMC 8803 - Research Methods Mini Literature Review We Used to be Friends: Disparity between branching performative technologies Anna Weisling October 28, 2015 Word Count: 3,074 
The term “multimedia performance” rests upon the assumption that all parties (“multi”) producing content (“media”) function collaboratively (“performance”), in a way which both enables individual expression and produces a product greater than the sum of its parts.1 Though this is certainly the case in many modern multimedia performances, it is patently false in others, and the reasons for such an imbalance (and the casual acceptance of such) is worth exploring. The increased computational affordances of the modern laptop has brought the oftvisited, ne’er-resolved topic of the audiovisual contract back into vogue (Ryan, n.d.). Today’s multimedia performance functions in a “real-time” format that is freed from the shackles of latency, input controls that span everything from the keyboard and mouse to infrared cameras to electrocardiograms. With horsepower capable of executing audiovisual renderings, streaming Netflix, and calculating your taxes all at once without breaking a digital sweat, there is little we can dream and not attain, at least as far as an artistic process is concerned. While the production and analysis process required to realize technology-enhanced or -enabled performances has become increasingly digitized, a surprising physicality has emerged through wearable technologies, sensors, and bespoke interfaces, and that physicality is being exploited in the realms of theatre, dance, and music—but not visual performance2 . The relationship between sound and image seems to be re-examined in phases, from color organ performances in the 1800s (Peacock, 1988) to experimental animation techniques in the early 1900s (Callear, 2008), to the iTunes visualizer and Video Jockeys in the 90s (Dannenberg, 2005). It seems inescapable, the link between visuals and music—color organ, music video, iTunes visualizer—indeed, the parallels have been explored by musicians, psychologists, performers, and academics alike. The ubiquity of the modern computer, however, has brought a new phase into consideration, one which is situated in a decidedly digital realm. But let’s step back for a moment. In 1991 animator/composer John Whitney Sr. identified some of the consequences that emerging digitization could have on aural and visual performances, and took great interest in the capabilities of the modern computer in producing new audiovisual relationships. His proposition that the smallest quantifiable units of sound and image could be broken down into the pixel and the sine wave (p. 597) suggested a kind of access to the heart of something new, a relationship that could not have been exploited through analog means. In fact, his only likening to traditional practices was that of compositional harmony and dissonance, explicitly stating that audio-visual works “belong elsewhere,” as enabled through the future “perfection of real-time audio-graphics computer instrumentation,” which would not replace or supplant classical music but rather “offer a development in modern times possibly as fruitful as the Classical epoch” (p. 598). For Whitney, and many others, the computer provided a new performance modality by imposing physicality on the concept of time, which could be stored, accessed, and manipulated in the computer’s memory. It remediated3 the basic principles of sound and image, allowing for the intimate manipulation of both elements in tandem. There is, here, a melding of two media that is so complete as to be an entirely new art form4—in fact, he states quite confidently that
To be clear, this paper assumes a “multimedia performance” involves either collaborative dynamics, with more than one participant/ performer, or a performance consisting of single individual creating or utilizing more than one medium. In all circumstances the use of computer technology and real-time creation or manipulation of that media is assumed. 1
2
A performance in which visual content, separate from the performer’s body, is generated or manipulated in real-time.
3
Refashioning earlier forms of media, as defined by Dr. Jay Bolter.
Not to be mistaken for Michel Chion’s concepts of audiovison, which I would argue maintains an imbalance of priority and privilege between the two elements. 4
2
“electronic music and electronic color-in-action combine to make an inseparable whole that is much greater than its parts” (p. 599). What developed in the decade to follow certainly reinforces many of Whitney’s thoughts, but it would be a disservice to claim that the new field of intertwined audiovisuals he envisioned had come to fruition. In the early 2000’s the landscape of new media performance certainly reflected the use of digital tools as instrumentation to control media in performance, however one side of the audiovisual partnership was clearly bounding forward as the other lagged behind.5 In most of these early cases, technology in audiovisual works was primarily used to perform music and impose (and enforce) the structural relationships between sound-to-image6 through mapping. In 2009 Ilias Bergstrom and R. Beau Lotto published the article, “Harnessing the Inactive Knowledge of Musicians to Allow the Real-Time Performance of Correlated Music and Computer Graphics.” In one fell swoop, this title alone managed to completely strip visuals of any performative autonomy, and close the proverbial door on any prospects of a meaningful and expressive performance by a visual performer. Instead, “using the musical instrument(s) itself as the primary source of control…by mapping…a performer will be able to more readily transfer his/her inactive knowledge of the instrument to creating visual music” (Bergstrom &Lotto, p. 92). More broadly, though, the title represents an ideology which was prominent at the time and which remains in place today: “We can make visuals now, but it’s not easy, therefore we are better served by implementing them as an accompaniment/support to the music, which remains our focus.” As a handful of makers explored the computational prowess of sound-to-image and (to a small extent) image-to-sound mapping in the 2000s, another field was clearly emerging, one which, by focusing on a single practice (music), launched the concept of technology-aided musical performance into a field of its own, while either ignoring or absorbing the concept of live visual performance. Bergstrom’s and Lotto’s concept of audiovisual performance solidified over time: Audio was king, and visuals were extra jewels on the crown, if requested. As we know it now, the NIME (new interfaces for musical expression) movement is secure in its legitimacy as an academic field. That’s not to say that debates don’t rage on regarding what “music” really is, or why digital tools can never emulate the expressivity and creative potential of traditional instruments (Wessel, 2002), but rather that those debates are happening in academic departments whose sole focus is creative computing, or conferences dedicated to HCI in music. That firm footing, however, was not always so assured—in its infancy NIME faced its share of doubters, and was, for many years, overwhelmingly practice-based and insular (Gurevich, 2014). What we see as an eventual byproduct of the maturation of this new field is an impressive number of journal articles spanning the fields of music, psychology, science, HCI, and art, including, importantly, a focus on the tangibility of the performance object. Marcelo Wanderley’s and Nicola Orio’s paper entitled “Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI” reflects not only the cross-disciplinarity of new instrument design (an article on how HCI enables musical expression, published in the New Performance Interfaces edition of the Computer Music Journal) but also the priority and interest given to physical devices as a way to make computer-mediated media meaningful. The article is saturated with terms like “gesture” and “expression” and draws unabashedly from the hard sciences (Wanderley & Orio, 2002).
It is important to note here that the computational power required to produce or manipulate aural material is orders of magnitudes less than to produce visual content—it is only natural for early progress to be made in music technology before it extends to video. 5
6
I say sound-to-image, not sound-and-image here, and wish to call attention to that distinction.
3
This burgeoning field of new musical instrument design seems to have had an impact on the conceptualization of the sound-image relationship (again), and in a potentially meaningful way (again). By 2005 several authors were writing about audiovisual performance not in terms of strict and/or irrelevant mappings, but rather performance capabilities. As electroacoustic composer and improvisor Thomas Ciufo stated in 2002 (in no uncertain terms), “real-time image and sound processing systems enable artists to affect sound an image generation, playback, and processing based on expressive performance attributes.” His conceptualization of an “intermodal form of expression” not only emphasized the structural relationship between audio and visuals, it also promoted the integration of these principles to create a larger work and spoke to the aural and visual in terms that were almost confusingly comparable. Although he formulated his idea of potential audiovisual capacities within the frame of parameter mapping, he also acknowledged the concepts of perceptual agency, that, “the important issue is not the equality between the media forms…but how the different media interacts.” It is particularly interesting to see this merging of mapping and interaction, as the former has seemed to undermine the latter, historically. “Equality,” for Ciufo, seems to involve recognizing and embracing the complementary nature of sound and image rather than attempting to pervert one sensory framework into another. Going a step further, digital artist and composer Brian Evan’s article “Foundations of a Visual Music” boldly stated that, “Visual music can be accompanied by sound but can also be silent” (p. 11). In a sea of visuals existing solely as accompaniment to musical performance, Evans’ claim was particularly bold in the same way as Ciufo’s, especially considering that both authors’ backgrounds are in music and composition—they were surely aware of the trends and capabilities within the audiovisual performance realm at the time. Unlike Ciufo, however, Evans seemed to completely ignore the minutiae of parameter mapping, focusing instead on expanding the applications of consonance and dissonance in a multimedia work. Though he did not shy away from relating visual parameters to musical ones, those constraints seem to have meant far less than the gestalt experience of sound and image working together, having been composed thoughtfully and with intent. Interestingly, in what is likely a byproduct of ever-increasing computational power 7, the terms “interaction” and “live performer” appear, almost casually, in the context of these audiovisual articles, in the same way they appeared in computer music journals a decade before. Roger Dannenberg, a professor of Computer Science, Art, and Music, seems to step up as champion for the video-underdog, boldly pointing out that, “visual events do not work the way sound events do,” and even attacking the popular iTunes Visualizer as superficial and gimmicky, providing animations that “offer only what is readily apparent in the music itself” (p. 28). In essence, he reinserts the dash into the audio-visual contract, at once validating them individually and reestablishing an equal partnership. It’s in this time that we see some call to validity and autonomy for visuals. Multimedia artists seem to wonder, finally, why the same technology that enables rich and complex musical performances is not being applied to the visual realm. As Dannenberg put it, “The unitgenerator, stream-processing approach of computer music language can be applied to video processing, and this is already possible…” (p. 30). His conclusion, in fact, points even more directly to this imbalance, as he posited that, “Interactive performance combining images and music is a natural application for computers. Because computer animation, video processing, and audio processing have all become
7
I’ll mention again, here, the reality of visual computational demands: impossible to equate with audio, even today.
4
possible in the digital domain, the computer allows for a high degree of interaction and control… Computers have gained enough speed to process video…with amazing flexibility” (p. 34). When Dannenberg asserted that, “images are not an interpretation or accompaniment to audio but rather an integral part of the music and the listener/viewer experience” (p. 25) he was revealing a sense of inequality that had been manifesting between digitally-created musical and visual performances. As simple a statement as, “Video processing opens up the possibility of images controlling sound” (p. 31) may seem, the role-reversal was in fact quite a controversial one, at least on a stage dominated by the musician. Yet, this is where we remain situated. Bergstrom and Lotto continue to reduce the history of visual expression to “systems…made to either accompany music with color or provide a form of visual music.” While not incorrect, this simplification seems to imply that the time for performing visuals is over, and as the article goes on to articulate, “despite the intense interest in visual music, and advances in its associated technologies, current tools/practices are limited in three ways: (i) highly constrained, limited mappings, (ii) limited controllability, and (iii) overly complicated processes for preparing new performances” (p. 92). There was a time, not so long ago, that the computer was considered a gateway to a new performative landscape, where the limitations of sound and image could be stripped away and our conceptions of an “instrument” could be reconsidered. As computational affordances increased, new interfaces for musical expression began to emerge, and by the time we could compute visual content in real-time, NIME was a field of its own. Where we now have musical instruments that harness movement, bio-signals, proximity, temperature, and any number of other input signals to produce expressive and exciting performances, we still find ourselves connecting the loudness box to the brightness box and calling it “live visuals.” Whether subscribing to Whitney’s ideas of “absolute correspondence” between musical and visual parameters, linking smallest quantifiable units, or combining them to reflect Evans’ larger concepts of form and structure, the fact remains that there simply is no way to perform such an objective. In order to enable meaningful collaboration and expression in visual performances, the visual must be given the same autonomy and design considerations for new instruments that music has. These articles each identify the role of visual in audiovisual performance as worthy of consideration and development. They tie the visual to the auditory at varying levels of constriction, from structurally bound to entirely autonomous. What they all stop short of, though, is the acknowledgement of the overwhelming disparity between interfaces for the production and manipulation of audio in live settings and the relative languishing of visual production tools. It’s not enough to push for complete subservience or total autonomy, or even finally giving legitimacy to the visual element of a performance. None if it means anything if there is no apparatus for the actual performance of the medium. There are those who try, from their seat at the computers, to produce such an experience*, stacking their click-and-drag control functionality against the robustly physical augmented guitar or brain-wave controlled haptic instrument, and new media artists seem, as a whole, to find this perfectly acceptable. However, we can not blind ourselves to the historical fact that the performability of both sound and image were once held at equal odds, and that the value of such a capability still lingers. It’s time to catch up to NIME, time for visuals to go through the growing pains of the innovative process. On the other side might just be a NIVE counterpart, populated by physical performance objects with the same expressive and gestural affordances as any other instrument.
5
Sources Bergstrom, I., & Lotto, R. B. (2009). Harnessing the Enactive Knowledge of Musicians to Allow the Real-Time Performance of Correlated Music and Computer Graphics. Leonardo, 42(1), 92– 93. http://doi.org/10.1162/leon.2009.42.1.92 Callear, S. (2008). Audiovisual Correspondence : An Overview. Chion, M. (1994). Audio-Vision: Sound on Screen. Columbia University Press. Ciufo, T. (2002). Real-Time Sound / Image Manipulation and Mapping in a Performance Setting. Dannenberg, R. B. (2005). Interactive Visual Music: A Personal Perspective. Computer Music Journal, 29(4), 25–35. http://doi.org/10.1016/S0041-1345(97)01103-2 Evans, B. (2005). Foundations of a Visual Music. Computer Music Journal, 29(4), 11–24. http:// doi.org/10.1162/014892605775179955 Gurevich, M. (2014). Methodology Soup : Diversity in NIME Research Practices, 5–7. Peacock, K. (1988). Instruments to perform color-music: Two centuries of technological experimentation. Leonardo, 21(4), 397–406. http://doi.org/10.2307/1578702 Ryan, J. (n.d.). As If ByMagic, 1–11. Retrieved from http://jr.home.xs4all.nl/MusicInstDesign.htm Wanderley, M. M., & Orio, N. (2002). Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI. Computer Music Journal, 26(3), 62–76. http://doi.org/ 10.1162/014892602320582981 Wessel, D., & Wright, M. (2002). Problems and Prospects for Intimate Musical Control of Computers. Computer Music Journal, 26(3), 11–14. http://doi.org/ 10.1162/014892602320582945 Whitney Sr, J. (1991). Fifty Years of Composing Computer Music and Graphics: How Time’s New Solid-State Tractability Has Changed Audio-Visual Perspectives. Leonardo, 24(5), 597– 599. Retrieved from http://www.jstor.org/stable/1575668
*Postscript: One might wonder if the very lack of development identified here holds meaning in and of itself, and rightly so. After all, if we continually revisit the idea of performative visuals, but only ever progress in the realm of performative sound, how much could we actually need or want such a change? The articles examined in this paper can speak only to the past—who will speak for the present? Who has the right, authority, or knowledge to do so? I have no formal training. I know, at any point in time, the absolute bare minimum needed to accomplish an artistic goal. It is with reluctance and modesty that I bring the topic of my own practice forward at all, and with even more so that I argue the following: that there is a call for this. There is a reason I began down the path of live visual performance, and that reason is that I was asked to, by one professor, one day. “Just make something and improvise with us.”
6
he said. I wasn’t a musician, I wasn't a performer, I wasn’t a filmmaker. I just wanted to be in a band. Don’t we all? I’ve found myself involved continually, in a visual capacity, with ensembles for the past 10 years. There exists no shortage of musicians, dancers, composers, and artists who embrace this performative element in their collaborative experiences, and in fact I’ve collaborated with musicians who have subsequently requested me to write a visual piece for them to accompany. I have submitted and been accepted to experimental music conferences and festivals (including NIME, three times), with pieces emphasizing visual instruments and live visual performance. There is no pride in this, in and of itself, I find myself confused and bewildered at every new opportunity. I can only logically trust that this reflects a larger desire, more meaningful than any small notions I have about my own personal contributions.
7