MUS7019 - Interaction Design Anna Weisling
DEVICE - A De-constructible Visual Instrument for Creative Expression Anna Weisling KEYWORDS Interaction design, visual instrument, audiovisuals, new visual interfaces
ABSTRACT
comparable historical context, and thus lacks a benchmark for expanding visual capabilities. The objective of this project was to create a tool for natural-feeling exploration and deconstruction, which would allow for a performance that is both capable of improvisation as well as repeatability.
This paper explores the research, development, construction, and implementation of a new visual instrument. Although new musical instruments are seeing constant innovation and progress, they are steeped in years of practice and pedagogy. Tools designed with the purpose of driving visual systems are rarer and present unique issues and capabilities. By analyzing basic interactions such as tilting and rotating an object and the purposeful use of gesture and manipulation, a tool for ‘playing’ visuals in a performative arena was produced, with capabilities evocative of musical expression. The capabilities of the resulting DEVICE (De-constructible Visual Instrument for Creative Expression) are discussed and explored in this paper.
• Interaction and feedback - When working with sound there is an instantaneous system of feedback from interaction to sonification. A plucked string produces an immediate auditory articulation of that gesture. A visual instrument, therefore, must provide the user with equal levels of feedback to allow for playability and development of form.
1. INTRODUCTION
The identification of equally universal units of change and articulation become as important as the interaction design itself. Despite the drastic differences between sound and visuals, there are undeniable similarities and connections, and the potential for visual instruments is undoubtedly promising. After all, “Music, visual, and numeric sequences all have phases, as do physical phenomena such as light waves” (Fry and Reas 2007, p293).
Within the field of new instrument design there is no shortage of tools for new musical performance. The methods of sonic expression include pitch, timbre, voicing, tempo, gesture, and more. However, when working within the realm of visuals these parameters become less clear. Some issues addressed in the process of instrument design include: • Smallest unit of change - While musical instruments utilize, at the most fundamental level, notes as the basic unit of articulation over time, visuals are composed of pixels of color that change over time. Even a low-quality image will contain several thousand pixels at any given moment, with three values defining its red, green, and blue levels, as well as a measurement of alpha or opacity. This renders the pixel an incredibly complex unit of measurement, which functions at a nearly impossibly fast rate of change. If one steps back further to individual frames, the rate of change remains incredibly fast, at an average of 30 frames-per-second. The challenge in finding a manageable control parameter has no ‘correct’ answer, and is rather the result of testing for usability and articulation. • Universal parameters - A matter of identifying variables that are shared by all visuals. By their nature visuals operate with inherent brightness, contrast, and saturation parameters, as well as criterion for rate of playback (including forward, backward, and speed) and placement on a physical plane. In most cases hue (the relative levels of red, green, and blue), blur, and opacity can also be manipulated. Content and generation of content, though controllable, are arguably not universal, as not every performance requires or functions with this level of control. • Playability instrument instruments possible on
- The usefulness or practicality of such an is brought into question. While new musical allow for extended technique not necessarily traditional instruments, a visual equivalent has no
• Polyphony - Textural and formal development of a performance often manifests itself in the layering of multiple pitches, building melodies and harmonies, and the sounding of multiple voices. Visually this can be linked to layered footage, separation of visual fields, or even timed events.
2. BACKGROUND Regarding systems of visual interaction, Myron Krueger stated that, “There was this very natural desire to identify with the image on the screen...[people] expected it to do things in the video world as much as it did in the physical world...it was as if evolution had prepared us for seeing ourselves on television screens combined with computer images” (Krueger, 1988). Designing an instrument that functions in real time using natural gestures proves difficult for the simple reason that the user is disconnected from the result of the interaction, which is (in this case, at least) being projected onto a wall. In his piece Videoplace, Krueger utilizes live video feeds of users, creating a direct connection to gesture and product. Furthermore, his system responds elegantly to natural gestures (Buxton, 2011) that are common to daily human interaction, such as pointing a finger or waving a hand. This kind of system accurately detects not only movement and shape, but also a sense of intentionality and allows a user to successfully interface with the system without special training or practice. The introduction of sensors to an instrument is no new practice, however the level of refinement is paramount to both the success of the tool as well as the frequency of its use. As stated by Atau Tanaka, “A musical instrument, as part of an engaging performance, becomes an expressive device in the hands of the performer. This gives an instrument a distinguishing characteristic when compared to a simple tool” (Tanaka, 2000, p.
389). An ‘instrument’ can be designed with as many buttons and lights as can fit on the available real estate, however that does not a successful tool for artistic expression make. An instrument must function on levels of macro- and micro-control and allow for both expression and virtuosity.1 The primary concern when designing a new visual instrument revealed itself to be a lack of familiarity. While new audio instruments have the benefit of a point of reference, a visual instrument is conceptually new, as the historical form of performative visuals is primarily physical expression such as dance or fireworks. As an example, the Metasax is a modified saxophone that carries with it all of the connotations and historical practice surrounding the saxophone itself, but with extended playability through the use of sensors (Burtner, 2004). There currently exists no comparable visual object to extend.2 However, just as groups and artists such as BioMuse (http:// www.musicsensorsemotion.com/tag/biomuse-trio/ or Marco Donnarumma (http://www.marcodonnarumma.com) are using new technology to sonify muscle movement, something that has never previously been considered as a traditional ‘instrument,’ it is possible to begin instrument design sans referent.
3. RESEARCH/DEVELOPMENT The first step in designing and testing a new visual instrument was to explore modes of interaction. Namely, why do we interact with objects? Normally this process would not necessarily be included in a paper of this nature, however because of the unique circumstances (a visual rather than musical instrument) it is an important area of discussion. A rough sketch of potential reasons behind why a user might engage with an object (see Figure 1) as well as an expansion regarding differences in motivation and end result3 provided useful potential parameters with which to begin and test design. • • • • • • • • •
Weigh Compress Extend Untangle Bend/Flex Smell Taste Depress/Elevate Protraction/Retraction
Figure 1 - List of reasons for interaction • Reveal - See what is inside, underlying, functional (suspicion-driven?) (ex- opening a locked box, scratching a lottery ticket). • Explore - Knowledge for the sake of knowledge, personal improvement (ex- studying a photograph, examining a leaf).
• Understand - Attempting to gain control though understanding it’s function (ex- opening electronics to see the wiring, dissecting an animal). • Destroy - Rendering something functional non-functional (ex- smashing a plate, popping a balloon). • Create - Manipulating an existing object into something new (ex- making clay into a pot). • Reward - Extracting something desirable from the object (ex- completing a puzzle). • Purpose - Using the object as a means to an unrelated end (ex- getting a key out of a drawer, writing with a pencil). • Entertainment - Interaction for the purpose of distraction (ex- playing a game, watching TV). • Pleasure - Deriving sensory stimuli from object (expetting a cat, running fingers through water). • Boredom - Interaction for no defined purpose (exbouncing a ball against a wall, twirling a pencil). These categories are admittedly vague, and lend themselves to overlapping and cancelation. Key emotions and motivations segregate what might be considered congruent concepts. For example, though Explore and Understand are arguably very similar, they can be separated by the delegation of a sense of control. Entertainment and Pleasure are separated by a physical motivation: Pleasure’s motivator is a physical stimulation, while Entertainment can occur with no such feedback. With the focus on Revealing, Destroying, and Understanding, an object can be engineered to draw these behaviors from the user. Perhaps more important in the design process was the how (Figure 2). • • • • • • • • • •
Pull apart Rotate Open Unscrew Listen Look Touch Shake Tap/Scratch Heft
Figure 2 - List of ways we interact with an object An overwhelming majority of these actions can be executed with any conceivable object, however, several of these actions could be linked to the instrument’s motivating behaviors. By selecting an existing piece of hardware (a piece of bathroom piping) capabilities such as pulling apart, unscrewing, and shaking (destructive behaviors), rotation, looking, and touching (understanding behaviors), and extending and opening (revealing behaviors) became purposeful means of interaction.
Through the instrument, the performer is able to affect all aspects of the music: from the micro-level of timbre to the event level of note articulation, to the macro level that reflects compositional structure. 1
Arguably, attaching sensors to the human body enables extended or enhanced dance, however for the purposes of this paper dance will not be equated to new visual performance. 2
Because this is a list of my own creation it is important to note that these designations are purely opinion-based, and only the product of brainstorming. 3
4. DESIGN Deconstructing an object can reveal more about its inner workings. The more we destroy something, the more intimately we know it. When it is ruined, the performer understand it better than when it was whole. The two states: Known and Unknown, and traversing from Unknown into Known allows for the cultivation of grander form within a piece. To activate, or ‘turn on’ the DEVICE a performer simply turns it upside down. To ‘turn off’ the instrument the action is repeated. As a modular object, component parts can independently control a visual frame that is split into three sections. The twisting of a lid or component part, the protraction/retraction of inset parts, and the screwing of one object into another make use of natural object interaction and can be mapped to the universal visual parameters of control. Tilting the object causes the visual displacement of subtracted frames in the playback (time) dimension, or ‘frame regression’ (Figure 3), while shaking it triggers a randomization of size and placement of the three visual fields (Figure 4). By shaking the instrument the user essentially sends the screen into a jumbled chaos, which then returns to normal after the shaking is halted.
Figure 3 - Object tilt
4. 1Mobility The natural gestures that the object encourages include rotation on both horizontal and vertical axes. While the horizontal turning is fairly uninhibited by a wired setup (whereby the wires come out of the bottom of the object itself), vertical rotation is rendered impossible beyond a couple dozen degrees. Size is also an important consideration. Because this instrument is manipulated entirely with the hands, it is large enough to handle easily, but small enough to facilitate holding for long periods of time. Furthermore, because it must be deconstructed, the size of the component parts lend themselves to being handled, and the method in which the object is taken apart needed considerable engineering as well.4 4. 3Instruction “Maybe the most difficult behavior to model is the actor capacity to recognize object features and to decide what actions are possible to perform with it.” (Kallmann 2001, p6) An obvious issue that arises when designing a new instrument is that of instruction for the performer. Traditional instruments, or systems designed as extensions of existing instruments, have the inherent advantage of shared cultural knowledge, where performers already have some idea of how the object is to be held, manipulated, or otherwise played. Furthermore, implementation of buttons, switches, or other common indicators of manipulation draw upon our familiarity with technology used today (keyboards, iPods, doorbells, light switches, etc.) and encourage ‘correct’ usage. As stated by Marcelo Kallmann in his 2001 dissertation on real-time object interaction, “A human-like behavior would recognize a given object with vision and touch, and then, based on past experiences and knowledge, the correct sequence of motions would be deduced and executed. Such approach is still too complex to be handled in a general case, and not suited for interactive systems where a real time execution is required” (Kallmann 2001, p6-7). The key to manipulation in the case of a new instrument is feedback. The design process should identify controls for interactivity, autonomy, and gesture (Badler 1977, p32). By linking this instrument to a visual feedback system, ‘correct’ behavior is be rewarded, and ‘incorrect’ behavior is acknowledged and future actions tailored accordingly. 4. 4Background Subtraction “The choice of realtime can have its cost in the narrowing of possibilities...for many composers realtime is simply a healthy constraint imposed on a nauseating infinity of possibilities.” (Ryan 1991, p4)
Figure 4 - Shaking causes randomization of visual fields Red, green, and blue values (hue) are manipulated through the inset tubing, allowing the performer to navigate through the color wheel by inserting and removing the pipe to various degrees. Turning the cap on the opposite end of the instrument navigates between live footage of the instrument itself and footage of its creation. The more times the cap is rotated, the more visible the inner workings of the instrument. 4
The liveness of the deconstruction is incredibly important, and so using the object itself to drive the visual system is essential. Because this involves a live feed, background subtraction becomes crucial. There are many ways to do this: • Chromakeying- Subtracting specific colors from the frame. Alternately, selecting only specific colors to pass through.
“Where considerations of stability and security predominate (as in holding a hammer or getting a jar lid unstuck) a power grasp is chosen. Power grasps are distinguished by large areas of contact between the grasped object and the surfaces of the fingers and palm and by little or no ability to impart motions with the fingers. Where considerations of sensitivity and dexterity predominate a precision grasp is chosen. In precision grasps, the object is held with the tips of the fingers and thumb” (Cutkosky 1989, p272).
• Math Ops- Using two video feeds, one with the object and one without the object, and then comparing the two and only passing specific selections. • Frame Changes- Setting a threshold for motion between pixels and passing only those that move above the threshold, excluding relatively motionless objects. • Motion Detection- Detecting moving objects and passing only that object. • CGI- Mapping the physical object itself to a computer framework and using the physical object to control the virtual one. • Camera Position- Using a camera mounted on a plane that captures only the object and not the user. Overhead, for example.
6.AESTHETICS
Because the hands and fingers are central to the deconstruction process, the primary concern is reduction or elimination of the face and body of the performer. Using frame changes coupled with thresholds for movement has been highly successful in eliminating movements of the body and face while preserving the object and hands.
• Whole frame - Visual field in tact, running in real-time.
5.CONSTRUCTION/IMPLEMENTATION
• Reduced frame - Resampling across the entire visual field, resulting in representative blocks of color in varying sizes. Can be coupled with any or all above states.
“The physicality of the performance interface helps give definition to the modeling process itself.” (Ryan 1991, p5) The DEVICE contains several sensors for the detection of deconstruction. Movement on the x-, y-, and z-axes is measured by means of an accelerometer, which is also capable of identifying shaking of the object. Perceptually it is natural and expected to equate directional fields with equal representation on a projected surface, therefore tilting of the object is mapped directly to the left and right areas of the frame. With the center section as an anchor, the flanking areas can be manipulated, essentially ‘scrubbing’ backward in time. This allows for extended performative gestures such as moving the instrument from left to right and delaying the frames so as to create a cascading effect. Once the left-right relationship has been established it can also be concluded that more forceful handling of the object itself will produce a resultant reaction from the system, in this case a randomization of the three sections in space. Furthering the control of time and space, the performer can now manipulate the positioning of the projected visuals in both temporal and spacial conditions, manipulating live and delayed visuals in tandem if desired. The inset tubing used to control hue is constructed with electroconductive ink running in parallel lines down the inside of the object. Coating the end of the tubing with copper tape allows circuits to be completed at different stages along the piping, resulting in the detection of changes in telescopic length. The performer is able to navigate through the color wheel at any desired time, and can adjust this parameter in congruence with any other variables. The cap of the DEVICE is also fitted with copper tape, which completes a dedicated circuit mapped to a pre-recorded video file. When bridged, this allows for the increase of speed of playback as well as an incremental introduction of the video to the live feed.
“The success of expressivity resides not just in the effectiveness of communication, but in the sense of agency that the system gives back to the performer.” (Tanaka 2010, p89) The corresponding Jitter patch created for the DEVICE sets an aesthetic scheme that supports and enhances the purpose of the instrument itself. The performer works within a visual theme reminiscent of design blueprints, with limited colors and blocklike shapes. He or she can navigate between this theme and its inverse (black-on-white to white-on-black or vice versa), allowing for exploration of time, space, and color in two distinct but connected aesthetics. The visual field is deconstruct-able to varying degrees: • Segregated frame - Visual field partitioned into three independent sections. • Deconstructed partitions - Visual field’s partitioned sections jumbled and randomized in space and, if desired, time.
With these four states the performer is able to deconstruct both the DEVICE and the resultant visuals to any desired degree, as well as introduce pre-recorded visuals and control color parameters. However, “In music system design, instrumentation extends from the mechanical design of sensors and controllers, through the electronics and software of interfaces and finally to the modeling of the higher level relations between performer and composition (Ryan 1991, p6). An instrument must be more that the sum of its sensors, it must be above all playable. In practiced settings the resulting performance with the DEVICE is satisfyingly gestural and responsive. When paired with an aural element, such as an electroacoustic performer, collaboration becomes very tangible and expressive5. The DEVICE, though in its infancy, takes several important steps toward what the author believes will soon be a fast-developing field of new instrument development. As visual artists continue to take their place on the stage next to musicians and performers their tools will be required to evolve beyond a VJ mixer or desk. An instrument for their art, something that can be held and manipulated in their hands, will lead to innovations in expression, gesture, and composition, and connect them to their art in a muchneeded way.
7. REFERENCES [1] Badler, N. (1997). Virtual Humans for Animation, Ergonomics, and Simulation. IEEE Workshop on Non-Rigid and Articulated Motion. - (0), p28-36. [2] Burtner, Matthew. (2004). Metasaxophone Systems: Matthew Burtner. Available: https://ccrma.stanford.edu/~mburtner/ metasax.html. Last accessed 1st May 2012. [3] Buxton, Bill. (2011). Gesture Based Interaction. Available: http://www.billbuxton.com/input14.Gesture.pdf. Last accessed 15th May 2012.
A piece entitled In Metal was created for the purpose of testing the object itself, and will see its premiere performance at the Sonic Arts Research Centre on May 25th, 2012. 5
[4] Cutkosky, M. (1989). On Grasp Choice, Grasp Models, and the Design of Hands for Manufacturing Tasks. IEEE Transactions on Robotics and Automation. (5), p269-279. [5] Fry, Ben; Reas, Casey (2007). Processing: A Programming Handbook for Visual Designers and Artists. Massachusetts: MIT Press. p293. [6] Kallmann, M. (2001). Object Interaction in Real-Time Virtual Environments. Unpublished doctoral dissertation, Swiss Federal Institute, Lausanne, Switzerland. [7] MediaArtTube (Producer). (2008, April 7) Videoplace ’88 [video]. Retrieved May 10, 2012 from http:// www.youtube.com/watch?v=dmmxVA5xhuo. [8] MintPass. (2009). Select. Available: http:// www.mintpass.com/select/select_detail.asp? Page=1&SrchItem=&SrchString=&SrchType=1&idx=52. Last accessed 12th Mar 2012. [9] Ryan, Joel (1991) 'Some remarks on musical instrument design at STEIM ', Contemporary Music Review, 6:1, 3 - 17. [10] Tanaka, Atau. (2000). Musical Performance Practice on Sensor-based Instruments. Trends in Gestural Control of Music. p389-406. [11] Tanaka, Atau. (2010). Mapping Out Instruments, Affordances, and Mobiles. Proceedings of the 2010 Conference on New Interfaces for Musical Expression (NIME 2010), Sydney, Austrailia. p88-93.