Stardust

STARDUST What I found there: The Looking Glass House

“She was up on the chimney-piece while she said this, though she hardly knew how she had got there.

And certainly the glass was beginning to melt away, just like bright silvery mist.�

“The amazing thing is that every atom in your body came from a star that exploded.

And, the atoms in your left hand probably came from a different star than your right hand. It really is the most poetic thing I know about physics: You are all stardust.”

–Lawrence M. Krauss

In my pilot projects I discovered that through play and participation we can create a collective experience that gives us meaning. The discovery in my research of presence as a quality in body-movement based interactions led me to venture beyond what we find on the screen to an embodied type of interaction. Inspired by Krauss’ quote and using Stardust as a metaphor for interconnectedness I came up for the idea for my final piece.

I created an interactive installation that allows participants to explore the physical boundaries between them and the system, as well as the boundaries between each other through a playful and explorative situation. They would experience themselves and their environment in a new way.

This means not having an outcome or an end result (like in a score-oriented game is winning or losing) but rather focus in the expression at the moment of interaction.This moment of interaction is what generates surprise and thus creates an enchanting experience.

209 stardust

FINAL PIECE

210 stardust

IT’S NOT A BUG, IT’S A FEATURE

While testing Lise Hansen’s Sync I knew that it could detect and track two bodies. While testing it out with my boyfriend I realized that it was only picking up his body.

I couldn’t understand why, and I wanted to be tracked as well. This bug in the system made me think how I could “trick it” so I could have the skeleton on me. By looking at our bodies on screen, we started to move in such a way that I could fool the computer vision software into thinking that I was the same person that was tracked in the first place. I managed to “steal” the skeleton. This made me realize that as computers become more pervasive in our lives, we must acknowledge that they are not fool-proof. By exploring our digital selves, on screen, we could come up with a creative way of modifying our movement in order to get the result we wanted on our reflection. This passing back and forth of the skeleton felt like playing a sort of digital tag: the playful game of passing something invisible so “you’re it.”

Tag is based on touch: you touch someone and that person “is it”, and then must in return pass “it” to someone else. I realized that Sync’s bug in that version of the software had become a feature for us: we could play a digital tag on this mode of Sync’s visualization.

stardust

211

stardust 212

stardust

213

214 stardust

Stardust could function on the same premise as this digital tag: passing stardust back and forth through touch. It also picks up the play element, as tag is a playful game that is not score-oriented but is rather focused on the fully absorbing activity of flow: passing it back and forth. My research on Hansen’s work had led me to this exploration of the material. And it was by exploring this material (by doing) that I got the digital tag idea that would be an important part of the Stardust project. I wondered what would happen if we did not have the screen at all, and had the “skeleton” or stardust on our real bodies. This way of experiencing embodied interaction seemed to play more on the merging of the physical and digital, and it would feel more like tag.

Going through my own material in my sketchbooks allowed me to discover ideas or connections between projects. This helped me come up with the conceptualization for Stardust.

I asked myself the following questions:

215

How can I create an enchanting experience through a playful situation?

What are the boundaries we have with each other in a physical space?

How are those reduced or stretched out when encountered with something unusual or magical?

How can I break away from the mirror metaphor represented on screens, and create an interactive system that allows people to experience themselves in a new way?

How can I work with content transfer through touch between participants and make it enchanting?

stardust

stardust 216

A participant would walk into a space and see stardust on an object. By inviting this person to touch that object the stardust would start spreading into this person’s real body.

The stardust would spread around, flow and glimmer into this person’s arms and spread to the whole body. The common reaction for other people observing is to try to be in the “area” where this is happening, or try to get it too. This would invite exploration to how to get it on them as well leading to the discovery of content transfer through touch.

Stardust would blend the physical and the digital through a mixed reality experience: one beyond the window or mirror metaphor the screen provides. It would play on the embodiment of the stardust, but could only be actualized by the aid of a system that I would had to create using open source tools (code, computer vision, and projection). This system is based on tag, allowing participants to explore the system and the boundaries between them in order to fully experience the installation. This means that Stardust would work only with body presence: if no one is there to explore it, it could not function. What I aimed to create as a result is an enchanting experience.

217 stardust

STARDUST PROCESS/CONCEPT

218 stardust

STARDUST PROCESS/SKETCHES

Stardust stemmed from an idea, rather than the technology behind it. Before going into technical details on building a system that would be able to put stardust on people’s bodies, I wanted to explore the concept itself, to see if it was worth following through. I used the method I have developed for myself in AR Hockey and started by sketching out the idea.

The idea for Stardust was to have an object with the stars on, and then, by touching this object, you would get the stardust on your body.

As my process developed, I adjusted this idea to having the content transfer of the stardust happen not through object-person, but person to person instead. The reason for this is because in this way I could explore the boundaries between each other in am ore immediate way, rather than being mediated through an object.

219 stardust

Early sketch of Stardust, showing how the system would work.

220

STARDUST PROCESS/PAPER PROTOTYPING

stardust

ATerwards, I role-played the concept out with classmates using paper prototypes. I gathered feedback from this part of the process and adjusted the concept accordingly.

stardust

221

stardust 222

223 stardust

STARDUST PROCESS/VISUAL RESEARCH

ATer exploring how Stardust felt like, I created an aesthetic palette that would give me a reference on the visual language I would use for the installation. This visual research is divided in the following categories: photo-realistic stars, dust and light, particle systems/point-cloud, handmade and luminescence.

224 stardust

PHOTO-REALISTIC STARS I looked at examples like the work from Semiconductor. This film used footage from NASA to create an atmosphere of exploration .

stardust

225

Darkened Cities, by photographer Thierry Coehen.

stardust 228

229 stardust

DUST AND LIGHT

230 stardust

PARTICLES/POINTCLOUD

Particle systems created by Josh Nimoy for the film Tron:Legacy. The research in this type of visualization led me to realize that for my project I wanted a visual language that didn’t look too digital. I wanted to appeal to a more hand-made approach that could convey enchantment rather than a futuristic scenario.

stardust

231

232 stardust

HANDMADE

Images from ‘A Little book of Coincidene in the Solar System’ by Woods Books. The image to the left made me ask myself: where does the dust end and the stars begin?

stardust

233

234 stardust

This project by Amy Friend is called Dare alla Luce, which translates ‘to give birth’. Literally it means to ‘bring to light.’ This is what I wanted to do with my project: bring it to light through a visualization that invited exploration.

Friend used a needle to make small holes in old photographs and re-shot them against light.

stardust

235

236 stardust

LUMINESCENCE

I researched bioluminescense and discovered how nature has its own ways to shine light as a defense mechanism or indicative of some biological process.

In this image I found interesting how what I tried to do with Stardust works as a negative (inversed) version of what we see here: instead of being surrrounded by light, in this case bioluminescense, you would have the light on you.

stardust

237

238 stardust Still, by Meghan Reynard. This installation used an old building as a setting, and using tracing paper and glow paint Reynard created this atmosphere of a moment that stands still.

stardust

239

240 stardust

STARDUST PROCESS/ARTIST REFERENCE

As a reference I researched projects that explored bodymovement based interaction, content transfer through touch, and the use of projection to break away from the screens. I looked into projects that dealt with presence and that used mirrors in an enchanting way.

stardust

241

242 stardust

PROJECTION MAPPING ON BODIES /A WOMAN THAT DANCED WITH LIGHT

As part of my research I came across a film called La Danse Serpentine. This film, created at the turn of the 19th century, was done by the Lumiére brothers and performed by the dancer Loïe Fuller, a performer that explored movement and used it as transformational material.

Fuller projected light on her body in order to create a dancing serpentine as she moved. The uniqueness of this relied in analyzing movement for itself (and not a pose, or a synchronized dance) and doing it with the aid of the technology of the time: the lighting and projection system she utilized. As Albright points out: “Refusing the binary of nature and artifice, Fuller was able to create magic in the midst of a conscious and deliberate use of technology.

She engaged the expressive possibilities of stage lighting by dancing with, rather than simply in the midst of, color and light.” (Albright 2007: 54). The uniqueness of the experience implied a technical system designed specifically for this type of performance. It was so unique that Fuller patented the system: the stage she used (in which she also employed mirrors) as well as the dress-movement technique. This is relevant because it indicates that embodiment with the aid of technology has been researched for quite some time, and that the merging of the digital with the real, the light flames with the body, had been experimented with before the screen was even invented.

La Danse Serpentine, Loïe Fuller, 1896.

Light Projected on Fuller’s body, creating arresting visuals in real-time for theatrical performances.

stardust

243

244 stardust

Fuller’s costume and stage design was so unique, she patented her designs.

stardust

245

246 stardust

PROJECTION MAPPING ON BODIES /KLAUS OBERMAIER

Obermeier uses light as material for his performances. In earlier projects, the performers had to memorize the video projected on them so that it appeared to have their silhouette. After, Obermeier used computer vision and created this in real-time.

247 stardust

PROJECTION MAPPING ON BODIES /BELL: CHASE NO FACE (MUSIC VIDEO)

This music video done by Zach Lieberman (cofounder of openFrameworks) was shot in real-time using OfxFaceTracker, an addon developed

by Kyle McDonald.

248 stardust

PROJECTION MAPPING ON BODIES /DELICATE BOUNDARIES BY CHRIS SUGRUE

This project by Chris Sugrue explores content transfer through touch, from our computer screens into our bodies. I found it beautiful and interesting because it explores the boundaries between the real and the virtual, the physical and the digital and served as an inspiration for my final piece.

stardust

249

“At last reality only exists as a projection of light. We are distances of light. All technology is late sun-cult.”

–Paul Virilio

252 stardust

SCREEN-BASED INTERACTIVE PROJECTS /GOLD DUST BY MEMO AKTEN

Gold, created in 2009, explores the idea of superstardom and the glamour of celebrity, as you appear as gold dust when you look at yourself on a screen.

This project served as a visual reference for my initial model and test because the aesthetics of it convey the gold dust you can’t quite grasp and is enchanting in this way.

Traces, by James Alliban is a screenbased interactive system that creates these streaks of light as you move. If no one moves, they slowly disappear.

The systems needs the participants to fully function .

253 stardust

SCREEN-BASED INTERACTIVE PROJECTS /TRACES BY JAMES ALLIBAN

254 stardust

OTHER PROJECTS /LASER_GRAPHS BY ACHIM MONHÉ

This photographic work by Achim Monhé was done photographing a laser beam going through dust. I ralized through this project the relationship between everyday dust, and the stardust I was referring to in my project: texture and light.

255 stardust

OTHER PROJECTS / YAYOI KUSAMA

I’ve been intrigued by infinity mirrors and the work of Kusama explored this theme. I incorporated the use of mirrors in Stardust, as sa way of selfdocumenting and I find interesting making small spaces feel infinite.

256 stardust

STARDUST PROCESS/TEXTURE & LIGHT EXPERIMENTS

As part of the aesthetic palette for this project, I made experiments that explored texture and light. I took photographs of glitter to analyze how the light afects the color when the glitter moves. This helped me visualize an option for how Stardust could look like. I made other experiments (both analog and digital) that dealt with the texture of the stardust described below..

stardust

257

258 stardust

By making images myself, I could get the qualities I was looking for as a visual reference: the way that dust and glitter behave like a swarm, or the way their color change with the light. This experiment led me to have visual material that would have been harder to find elsewhere.

stardust

259

stardust 260

stardust

261

stardust 262

stardust

263

264 stardust

RUTT-ETRA IZER

I explored digital techniques as well. I used the Rutt-Etra Izer application to extrude images, resulting in texture and point-clouds.

stardust

265

266

STARDUST PROCESS/COLLAGE

stardust

Using the visual references, I made collages of how they would look like on the body. This helped me choose a visual language that would represent enchantment in a better way.

267 stardust

ATer doing a quick survey and picturing the stardust look in the bigger context of the installation, these options communicate better the magical qualities I am looking for.

268 stardust

STARDUST PROCESS/MODEL MAKING

To visualize how the installation would work I used projection mapping on a model. When creating it I realized that I had used diferent types of tools in my process: depth camera, animations, watercolors, pencil, laser cutter etc. As mentioned in previous chapters, I was treating all tools as equal and in result I could get further in the project.

ATer visualizing Stardust in the model I created the system that would make the installation work through open source software, described in the next section.

My tool palette of the day: cuttingknife, pencil, feltpen, depth camera and black board.

269 stardust I needed to test the projection several times before I got the result I was looking for.

stardust 270

stardust

271

stardust 272

stardust

273

stardust 274

stardust

275

276 stardust

STARDUST PROCESS/OPENFRAMEWORKS

To create the interactive system behind Stardust I needed to work with a tool that would integrate the capture of body movement with the stardust animation and allow me to project it back on people. I had used Processing before but aTer talking with my tutors, the best option was to use OpenFrameworks (OF) instead: an open source toolkit that allows you to work with code in order to create interactive work. OpenFrameworks comes with some examples, but to truly explore its potential you can download addons, or libraries created by members of the OF community that do specific things: like track bodies, for instance. You build up from each other, adapting what you need to your own project, or creating your own addon, and giving it back to the community. The working ethos of sharing and building up from each other’s work was really helpful for me at this stage. To have started out from scratch would have meant years of research and study. But by using openFrameworks it was easier for me to build my own project.

277 stardust The environment I used to run openFrameworks is Xcode. I worked with the source code from libraries (addons) found on the openFrameworks website.

278 stardust

The addons I used to create the system for Stardust are OpenNI (that detects and tracks your body and places a skeleton with joints on it), OfxKinect (that uses Microsoft’s Kinect as depth camera), OfxCV (computer vision), OfxPortals (allows you to have a presentation screen in the form of an object: an FBO, and a control panel screen) and OfxBoids for the flocking animation that I used and modified in order to create the stardust.

ATer experimenting with openFrameworks I created the program that would make Stardust function. I took the visual references that I had researched and through this program I created, I used them to create a silhouette of the participant’s bodies, in real time. This allowed me to test out the system and to decide along with participants which type of particles or stardust worked best as a visual language. I used a screen for this test because it would be easier to judge which visual language worked best, and because I haven’t worked on the projection to the body part of the project yet. The use of the mirror metaphor through a screen would aid me to see people’s reaction and to test the real-time collages.

279 stardust

STARDUST PROCESS/FULL SCALE TESTING

280 stardust

PHOTO REALISTIC STARS

I observed people’s reactions when experiencing different stages of my project.

I discovered that their inhibitions went away and that they even behaved like children, exploring how they moved and making funny gestures. This led me to think they were truly being in the moment.

stardust

281

282 stardust

PARTICLE SYSTEMS/POINT CLOUDS

stardust

283

stardust 284

stardust

285

286 stardust

DUST AND LIGHT

287 stardust I tested myself how it felt like to experience Stardust at different stages of the process. As a designer, this gave me a richer insight into what the participants were going through while experiencing the interactive system.

stardust 288

stardust

289

290 stardust

At this point of the project, the idea was to have the stardust on your body, with no screens anywhere. But, as a participant, you do want to see how they look. I wanted to use real mirrors to do so. In this way, the main system would work with embodiment, and you had to experience stardust on you, yet the project would also include the perception of your body image, but not through a screen; it would be through a real mirror instead.

stardust

291

292 stardust

STARDUST PROCESS/A SHIFT IN POINT OF VIEW

When I moved from using a screen-based interaction to a system meant to use your body as both cause and efect I realized something significant in the project: the shiT of point of view. As I could see from my research, projecting light on bodies is not new. It is usually done on a performer, and us, as the audience, watch their bodies morph and become magical. Performers are trained to move gracefully, communicate emotion and ignore the blinding lights.

In most of these performances there is a stage and the point of view is from the audience (or camera) perspective.

However, when the audience is the performer the question arises: how to hide the magic trick (or the blinding lights)?

In Stardust the participants are performers yet to make the situation magical, they do not need to know how it works.

The point of view is shiTed from passive audience, to active participant, and in Stardust, the challenge was to hide the source of light.

In order to solve this problem, I tested diferent ways to use mirrors as a way to hide the projector. I used my model and tested a one-sided mirror or tinted glass (like the ones used in focus groups or interrogation rooms), reflective paper foil and angles of projection reflected on mirrors.

stardust

293

294

ONE-WAY MIRROR

stardust

As the light comes from the other side of the tinted material, you can see what is behind. Also, the light concentrates in the center, creating a hot spot that is not very magical. I needed to test the other options.

stardust

295

296

PAPER-FOIL MIRROR

stardust

The projection sends the light to the paper foil, but the paper foil does not bounce the light back to the participant (or my paper models of them).

stardust

297

298 stardust

REAL MIRROR

When I tested it, it worked. At this point I was satisfied with the setup, and had found a way for the participantperformer to not see the magic trick (or at least to not be too obvious) and thus buying into the surprise element of the stardust appearing out of nowhere into their body.

stardust

299

300 stardust

STARDUST PROCESS/FULL SCALE TESTING

I tested full-scale the alternatives that have worked out in the model. The first test was projecting directly on the person. To my surprise, the projector was not disturbing for the participant: she was too busy discovering these dust specks on her body. I figured that all this time I thought the projector would be at eye-level. By doing this test, I realized that the projector lying on the floor was just fine; it had an enough wide angle to display the stardust on people’s bodies. The second test was using a real mirror. It did project the stardust back into the body and was working as with the model.

PROJECTING DIRECTLY ON THE BODY

stardust

301

stardust 302

stardust 304

stardust

305

308 stardust

PROJECTING THROUGH A MIRROR

stardust

309

I used a mirror as part of the interactive system, reflecting light back at the participant. I realized that I needed to work at the right angles to create the effect I wanted, and to not shine light blinding people’s faces. When I managed to get the angle right, it worked.

stardust 312

–Jorge Luis Borges

313 stardust

“The crystal spies on us. If within the four walls of a bedroom a mirror stares, I’m no longer alone.”

314 stardust

FULL SCALE TESTING RESULTS

When I was documenting these experiments, I could see how the stardust looked like on others. From the participant’s perspective though, it looked and felt very different (I tried it for myself ): you saw only bits of the stardust on you, and did not get the full picture. A way of showing that would be to use mirrors around.

Based on the participant’s feedback and on my own experience I decided to place the mirrors at a certain angle, so that the projected stardust fell directly on people’s bodies but you could see your reflection in full scale in the mirror.

I decided to not use of the mirror to reflect the stardust back as it bounced too much light to the participant and I used it instead only as a way for participants to see how the stardust appeared on their bodies.

In order to make the final adjustments to the system, I created a flocking particle system, based on Craig Reynold’s flocking simulator. The way birds fly together, creating a murmuration, functions on rules like cohesion or separation. These are the same rules that I explored through soTware, to create the swarm quality of the Stardust.

MAKNG THE FINAL SYSTEM/TAG LOGIC

To solve the tag logic I first tried the following: The way tracking works is with blobs, a box detecting the outline of the body. When there are 2 people, there are two main blobs. Each blob has a center, or centroid, so a way to do it would be “when the 2 centroids suddenly become 1, it means that the blobs have merged; the people have come together enough (lets say touch) so that the program detects now only 1. That is when the point of contact happens; tag.” I got this input from Golan Levin at the Computer Vision Workshop I attended at Resonate Festival and made me realize the logic for the tag part of Stardust.

In the final implementation, instead of using blob tracking (tracking bodies using a bounding box), like I did on AR Hockey, I used a skeleton system, OpenNI.

315 stardust

MAKNG THE FINAL SYSTEM/FLOCKING

316 stardust

STARDUST/FINAL IMPLEMENTATION

stardust

317

stardust 322

This project is novel in that it plays on content transfer through touch not from a machine or object into a person (screen to body, for instance), but from a person to person (body-to-body) situation. I find relevant for my research to explore the relationships between each other through interactive systems as much as the relationships of the interactive system in itself, or the relationships between us and technology, and Stardust allowed me to do this.

Stardust allowed me to work with a shiT from point of view: from passive audience to active viewer/performer.

This represents a way of working with interaction design in which the system allows the participant to be a creative actor, and the outcome is diferent every time of experiencing the installation.

THE GLASS HOUSE/LIMITATIONS OF THE PROJECT

There were many challenges when creating this final project. When I decided to implement all I have learned in my AR Hockey project and the insights I gained from my research in this final piece I had very limited time. For the concept, research, development and implementation of Stardust I had around 10 weeks time. I realized that I got used to working in a new environment (openFrameworks) quite fast. However, my coding was not as concise and clear as it could have been. This resulted in taking more time for the implementation of Stardust than I thought. This can be experienced in the final installation as tracking being not 100 percent accurate, and the particle system (stardust) could have been more detailed and close to what I envisioned in my models and sketches.

Due to the fact that I used the Kinect (a gaming device) as part of the final system, there is a limited range for tracking the body. Whenever someone falls out of this range the computer vision software can lose the participant. I had to adapt the installation’s environment to compensate this limitations, by using a smaller room, placing mirrors closer to the participants and working with the equipment that my resources allowed. If placed in a different context, a gallery for instance, I could use a different depth camera to have a wider range, as well as adapting the system to an ideal situation.

323 stardust

THE GLASS HOUSE/REFLECTIONS ON STARDUST

324 stardust

Although this project breaks away from the screen metaphor and its bias from transparency, it was difficult to break from the mirror metaphor altogether that a screen can also convey. I learned that self documenting is almost innate in people, and that we always tend to look for our own image in situations like this. I learned to be pragmatic about it and place real mirrors in the installation, (I managed to stay away from the screen though). This made me realize that an enchanting experience in interaction design is not about breaking away from the mirror and the window metaphor, but allowing them to coexist: at times play on the window metaphor and at times on the mirror one. It is this space in between that I found more interesting to work with through Stardust.

In this project I explored the leap from a screen-based interaction to an interaction that happens on your actual body. The content (stardust) transfer through touch allowed me to blend the physical and the digital in a novel way: it is the digital stardust participants pass back and forth, but executed on their bodies.

This type of situation, similar to that of tag, explores the boundaries between us. This is something participants can discover as they experience the installation. Figuring out that it works in this way depends on each participant’s own personal physical boundaries and curiosity. I was interested in exploring this situation and the digital tag that Stardust embodies serves as an example Through Stardust, I’m adding to the field as well by creating an addon for this installation and releasing the source code online, for free to the openFrameworks community. I realize I have made a building block by creating a new tool and expanding the possibilities of this project as others can use it and build up on it n

325 stardust

THE GLASS HOUSE/CONCLUSIONS

“We come spinning out of nothingness, scattering stars like dust.”

–Jalal ad-Din Rumi