Yutong Wang | Interactive Art and Design Portfolio | Admitted to Harvard GSD, UC Berkeley, and UPenn

Yutong Wang

Born in 2001 in Wuhan, China, Yutong Wang (they/them) are a queer multimedia artist and designer based in Brooklyn, New York.

Yutong are currently pursuing a Bachelor of Fine Arts degree at NYU Tisch School of the Arts, where they double major in Interactive Media Arts and Art History.

Yutong are constantly exploring unconventional interactions with human body through their constantly-evolving works, which include immersive experiences, interactive installations, audiovisual performances, and creative coding projects.

Their works have been showcased at the NYU Interactive Telecommunications Program, theBlanc Gallery, Wonderville, and Elevator Shanghai.

Space Dementia

Interactive Immersive Experience

Visualizing the Homonymic Fictional Mental Disorder

Type Individual Project

Date December 2019

Instructors Sarah Rothberg + Mimi Onuoha

Keywords Spatial Interaction + Animation + Motion Tracking

Tools JavaScript ( p5.js ML5.js ) + Adobe After Effects + Adobe Audition + Adobe Photoshop

01 Project Description

This project is a site-specific interactive immersive experience visualizing Space Dementia, a hypothetical mental disorder that causes psychological instability to astronauts isolated from the human society in the outer space.

Consisting of an animation and an embedded interaction design, this project breaks the fourth wall in 2D visual art by tracking the motions of the audience and transforming these data into audio feedback. It brings interactivity into convential ways of viewing moving images.

03 Approach and Tools

Projection on wall

Animation

Visual Adobe Photoshop

Adobe After Effects

Zoom H6 recorder Microphone +

Audio

Interactive immersive experience

Interaction design

Connected to webcam

Ran by laptop

02 Key Concept 04 Project Flowchart

In Space Dementia, the audience is both a viewer and a player.

View passively receive information

Audience enters the space and project begins

Transform the role of the audience

Animation part I

Motion tracking

Adobe Audition p5.js

ML5.js

Audio feedback p5.js

Watch animation

Project ends and audience leaves the space

Animation part II

Animation part III

Audience’s motions are tracked by webcam

Interaction Design Participate actively generate data

Motion data transformed into audio feedback

See and hear instruction to interaction

Interactive part ends

Follow instruction and interact

Development Process I - Animation

Visual Part

Storyboard

I first sketched a storyboard for 9 major scenes in this 3-part animation:

Element Preparation

The elements used in the scenes were from royalty-free pictures found online. In order to use them, I removed their backgrounds and recolored them in Adobe Photoshop:

Animation

Cutting out elements in Adobe Photoshop, I then animated them in Adobe After Effects:

Animation Outcome

The final outcome is a dubbed digial collage animation in surreslitic style and it is 4 minutes and 29 seconds long.

Here are some stills:

Audio Part

Script

The script was written as the diary of an astronaut who was trapped inside his space craft and lost connection to the Earth. The 3 parts of the animation refers to 3 pieces of diary from 3 incoherent dates, during which the astronaut becomes more and more mentally unstable.

In the Part II of the animation, the astronaut mentions his condition of auditory hallucination and gives instructions that ask the audience to move their arms for motion tracking and spatial interaction:

“I also find this subtle harmony with the space, that I can hear the sound of the space coming through my ears into my mind. If you are listening, I want you to come closer. Come and stand in the center. Stretch your arms like you are about to embrace someone. Do you hear it? The sound of the universe? Stretch your arms so that you can hear different sounds, like you are playing an intangible harp — so mesmerizing. Now raise your arms high. Do you feel the change? The rising amplitude is waking everything in the world.”

Record, Edit, and AV Sync

The voiceover was recorded using Zoom H6 recorder and edited in Adobe Audition:

The audio part was synced to the visual part using Adobe Premier.

06 Development Process II - Interaction Design

This interactive part embedded in the Part II of the animation is for re-creating the auditory hallucination experienced by the astronaut. As the audience move their arms as the animation instructs, they will hear music that responds to their movements. I used p5.js and PoseNet in the ML5.js library, a pre-trained machine learning model for Real-time Human Pose Estimation.

Motion Tracking

Data Detection

PoseNet processes the image of human into a skeleton of 17 key points and tracks their coordinates. I chose to track the movements of the two wrists of the audience, the result of moving hands and arms, since astronauts move in a weightless environment with this gesture.

Data Process

PoseNet only generates x and y coordinates of tracked parts, so additional code is needed for calculating the distance between the left and right wrists.

the Distance Between Wrists

The distance between the two wrists controls the midi value of the audio: The y coordinate of the two wrists controls the amplitute of the audio:

Audio Feedback Testing

PoseNet part selection in p5.js

Calculate

07 Development Process III - Spatial Layout

This project is installed inside a room with three blank walls.

08 Project Outcome

This project was not set up as large as the ideal spatial layout has planned due to limitations in space. However, it was shown to and tested by 40 people in two in-class critiques. It received positive feedback, such as “It is a very creative and engaging way of storytelling.”

In-Class Screening

In-Class Interaction Testing

Breathing Mirror

Interactive Installation for Visualizing the Meditation Process

Selected for NYU ITP | IMA Winter Show 2022 and Wonderville Winter Show

Type Individual Project

Date October 2022

Instructor Chika Iijima

Keywords Interaction Design + Projection Mapping

Tools Touch Designer + MadMapper + MUSE 2

01 Project Description

Breathing Mirror is an interactive installation that substantializes a meditation process. The image of the player is connected to their brainwave through MUSE 2 and only a peaceful mind will return the artificially-processed, disturbing image into the true face of oneself.

02 Project Flowchart

03 Development Process I - Visual Effect

The visual effect was made in Touch Designer:

Brainwave Detection

The intensity of the visuals is controlled by the player’s brain activity, which is detected by MUSE 2, a brain sensing head band.

Person Detection

I programmed the installation to activate automatically when a player is detected by the webcam.

To have this effect, I included a person-detecting function using the Blob Track TOP module in Touch Designer. Visual effects decrease when brain is more relaxed

I used Mind Monitor, a mobile app for receiving the brainwave data from MUSE 2. This app generates an electroencephalography that splits the brain activity into 5 waves.

Among them, the Alpha wave represents the level of relaxation and meditation, which would be used by me in controlling the visuals in Touch Designer.

I used an OCS In CHOP module to receive data from Mind Monitor to Touch Designer. The value of the Alpha wave is then exported to TOPs that control the intensity of visual effects.

The more relaxed the brain, the higher the higher the Alpha wave value, the lower of the intensity of visual effects:

Front, Side, and Top

Development Process III - Prototype

Prototype Making

Before connecting the installation to MUSE 2, I created a prototype using corrugated cardboard for a test on the projection mapping of the visual effect.

Projection Mapping

And I projected the visuals onto the installation by connecting Touch Designer to MadMapper.

06 Development Process IV - Physical Setup

Fabrication

I used six pieces of plywood sheet boards and laser cutted them into a ellipse. Then I painted them in white for a better projection effect.

Prototype Testing

Setup

The plywood sheet boards are installed onto the wall piece by piece.

Here is the setup for Breathing Mirror in the NYU ITP | IMA Winter Show 2022:

Cutting Plan Laser Cutting Coloring

Breathing Mirror was exhibited in two shows:

NYU ITP | IMA Winter Show 2022

370 Jay Street, Brooklyn, NY 11201 December 18th - 19th, 2022

CIRCUIT

Motion-Tracking Video Sculpture

Selected for SCRIPTING, a juried group exhibition at TheBLANC Art Space

Type Duo Project

Collaborator Yulin Fu

Personal Roles Concept Developer + Visual Effect Artist

Date June 2022

Keywords Interactive Art + Motion Tracking

Tools Touch Designer + CRT TV

01 Project Description

The image of oneself subtly combines tangibility and invisibility. One can touch it and feel it but can never see it without the representation in other media. After the invention of electronic devices, images made up of electronic circuits have been displayed on flat screens everywhere for decades. But the image is still intangible, existing only as a form of energy.

To further emphasize and exaggerate how images distort and misrepresent the reality and the conceptions of the self, CIRCUIT captures the energy that enables the image and gives it a mesmerizing twist.

02 Project Flowchart

03 Development Process

The circuit-bending visual effect was made in Touch Designer:

“Ominous Television” Effect

Using Blob Track TOP module in Touch Designer, this effect can be trigerred by person detection and is automatically turned off if no person is in front of the camera. This interaction design was inspired by Ominous Television, a horror film trope of a TV suddenly becoming malfunctional. By imitating this effect, CIRCUIT symbolizes the invisible ubiquity of electronic devices in our life.

04 Testing

triggered When not triggered, the TV displays normal images captured by the webcam

triggered, the TV displays images with circuit-bending effects

05 Project Outcome

CIRCUIT was exhibited in a group exhibition:

SCRIPTING

theBLANC Art Space, New York City, NY 10016

June 25th - July 1st, 2022

“This video sculpture is a mirror alternating between reality and matrixesque screen distortions.”

— Art Critique by Morgen Mueller and Elvin Ou, Juries of SCRIPTING

Mouth Stitching

Audiovisual Live Performance

Selected for NYU ITP | IMA Spring Show 2022

Type Duo Project

Collaborator Yulin Fu

Personal Roles Concept Developer + Visual Artist + VJ

Date June 2022

Keywords Audiovisual Performance + Experimental Sound Design

Tools MAX / MSP + Ableton Live

01 Project Description

Mouth Stitching is an audiovisual live performance collaborated by Yutong Wang and Yulin Fu.

Using MAX/MSP and Ableton, the two artists created dynamic visuals responsive to the rhythmic audio part.

The audio of Mouth Stitching is influenced by Acid House music from the 80s Techno music. Yulin recreated the sound of the bass synthesizer Roland TB-303, which was released in 1981, adding a retro quality to the rhythmic Techno beats. Following a well-designed composition, the audio progresses with modulations of frequencies and shifts in layers of melodies that trigger synchronized changes in the visual part of the performance.

The visual style of this performance is a combination of figuration and abstraction. Using Jitter in MAX / MSP, Yutong processed pre-recorded videos of Yulin’s face into black and white silhouettes and generate white cross-hatching lines to map out these silhouettes. The alienated faces are deformed, rotated, or reversed in black and white along with the changes in the tempo and melodies of the audio counterpart. The aesthetic subtly mixes minimalistic colors with minimalist line art, and it is partially inspired by ad/ab Atom, an audiovisual installation by Japanese artist Ryoichi Kurokawa.

02 Visual Inspiration

03 Development Process I - Visual and Audio Design

The visual for this performance was made in MAX / MSP: The audio was made in Ableton Live:

The visual and audio were mapped and controlled by Novation Launch Control:

Ryoichi Kurokawa ad / ab Atom

Max Patch Max Patch in Presentation Mode

Visual Effect - Pixelated Visual Effect - Lines

Development Process II - Spatial Layout

Performance Space Floor Plan

Schematic

Elevation Plan

Performance Plan

05 Project Outcome

This 9-minute long performance was publically performed in two shows:

Designing Interfaces for Live Performance Live Final Show

370 Jay Street, Brooklyn, NY 11201

May 8th, 2022

NYU ITP | IMA Spring Show 2022

370

May 16th - 17th, 2022

Poster Design by Yutong Wang

ODYSSEY

Interactive Web-Based Data

Visualization of the Space Age

Type Individual Project

Date May 2021

Instructor Leon Eckert

Keywords Data Visualization + Interactive Website Design

Tools JavaScript ( d3.js ) + HTML + CSS

01 Project Description

As a member of Generation Z, I was raised up in a world with a long-existing obsession of space exploration. Seeing countless news on space technology and launches of space missions, I have always wondered the following questions: What is the reason behind this obsession? What is the origin of this obsession? How has it marked the development of mankind?

These questions drove me to research on the Space Age, an age of space explorations and technology starting from 1957. I have searched for data about significant events and space missions during the Space Age and created Odyssey, a web-based interactive data visualization.

02 Approach

Data Collection

The input data came from five datasets:

1. UCS Satellite Database

2. Chronology of Space Launches from Gunter's Space Page

3. Launch Sites from Gunter's Space Page

4. Chronology of Significant Events in Space from Wikipedia

5. GeoJSON Data for Countries in the World

Coding

The first two datasets were downloaded from the web as excel forms, which were converted by me into csv and json files for using in my coding process. The third and fourth datasets were scraped from web pages in json files by me using ParseHub, a web scraper. The last one was received from Prof. Leon Eckert as a class material.

This data visualization was coded in JavaScript with the use of d3.js library, HTML, and CSS:

JavaScript: Programmed Interactions

CSS: Designed Website Layout HTML: Coded Website Content

JavaScript: Programmed Interactions

Visual Inspirations

I wanted this project to have a clean, minimal, and informative design, so I used these following cover arts and website as both style and color references:

04 Design System

Based on my references, I created a design system: Color Typography - Gotham

Primary Colors

Project Outcome

The outcome is a eight-section interactive data visualization. These sections vertically aligned and users to need scroll down to view them all. In interactive sections, there are instructions for how to interact on the left. Here are six selected sections:

Force Graph of Orbital Launch Sites Bar Chart of Orbital Launches Per Year

Force Graph of Operational Satellites

Kitchen Genie

User Experience and Conversation

Design for a Cooking-Assisting Chatbot

Type Individual Project

Date November 2021

Instructors Regine Gilbert

Keywords User Experience Research + Conversation Design + Chatbot

Tools Figma + Engati

01 Project Description

Kitchen Genie is a cooking-assisting chatbot targeting college students who cook. I conducted user research for clarifying the pain points and needs of the target users and focused on conversation design for a chatbot in reponse to the research outcomes.

02 Inspiration

After living alone in my junior year, I started spending extra time on cooking, especially on looking for recipes online, which took a lot of my limited free time.

While talking to my friends and classmates, I found that similar problem exists among them and came up with the idea for creating a cooking-assisting chatbot.

03 Design References

Instacart Ideas & Guides

Curated recipes and meals on the website of Instacart

04 Project Content Overview

Alexa Shopping List Skill

Voice user interface for adding and sorting items in the Amazon shopping list

05 User Research

Qualitative Research: Interview

I interviewed four college students about their cooking habits and preferences for a cooking-assisting chatbot:

Frank

How often do you cook?

“Everyday.”

Do you need to look at recipes for cooking?

“I have a general recipe and I use the same ingredients. But sometimes I will look up recipes for substituting ingredients.”

How many people do you cook for?

“Usually one, but sometimes I cook for two people.”

Do you often cook new dishes that you have not made before?

“Recently it is quite frequently.”

How do you look for recipes?

“Ask a friend or a family member.”

Male 22

What tone do you expect from a cooking-assisting chatbot?

“Light-hearted and casual.”

What features do you expect in a cooking-assisting chatbot?

“Recipes and how to cook them. Recommendation system is not always accurate, so it is not on my priority list. But I am interested in how this function works.”

Kaitlyn Female 20

How often do you cook?

“I never cook at school because I am currently living in the dorm and it takes too much effort for cooking. But I cook on occasions like Thanksgiving or Christmas.”

Do you need to look at recipes for cooking?

“Yes.”

How many people do you cook for?

“Five people.”

Do you often cook new dishes that you have not made before?

“Yes.”

How do you look for recipes?

“Pinterest and blogs online.”

What tone do you expect from a cooking-assisting chatbot?

“Excited and cheerful”

What features do you expect in a cooking-assisting chatbot?

“I have not cooked for a long time, so I do not have many specific special answers. Maybe just recipes? And it would be cool if the chatbot can tell me when to add the next ingredient and when to turn off the heat.”

Liam Male 20

How often do you cook?

“Very little at school, but I cook once a week at home.”

Do you need to look at recipes for cooking?

“Yes, every time.”

How many people do you cook for?

“Three at home, one at school.”

Do you often cook new dishes that you have not made before?

“Half the time. I usually make the same cuisine but substitute with different ingredients.”

How do you look for recipes?

“Recipe websites and tutorials.”

Quantitative Research: Survey

I sent out surveys to 22 college students asking the same set of questions for more data references:

What tone do you expect from a cooking-assisting chatbot?

“Nicely and gently guiding the user to cook like a mom.”

What features do you expect in a cooking-assisting chatbot?

“Recipes and cooking instructions of course. Some cooking tips would be great, how to peel a tomato fast and something like that. Recommendation of dishes based on my personal preference sounds nice, because if I like a certain type of dishes, I would love to receive more similar food.”

Zheyu Male 21

How often do you cook?

“Five to six times a week.”

Do you need to look at recipes for cooking?

“Yes, but not every time.”

How many people do you cook for?

“Only for myself.”

Do you often cook new dishes that you have not made before?

“Sometimes, I usually one new dish per meal.”

How do you look for recipes?

“Youtube tutorials and TikTok.”

How old are you?

What tone do you expect from a cooking-assisting chatbot?

“Warm, soft, funny, and have bad jokes.”

What features do you expect in a cooking-assisting chatbot?

“Must-haves are food ingredients and cooking instructions. I do not have a strong preference for food so I don’t need recommendations that much. So are cooking tips. They are nice to have but not necessary for me.”

How often do you need to look at recipes for cooking?

Do you cook new dishes that you have not made before?

Research Insight

Based on the interview and surveys, I found four major challenges in their cooking experience:

06 User Persona

Dana Wong Junior Undergraduate Student

Age: 20

“Time management and health are important for me and my study.”

Non-unified measurements for ingredients in recipes

Lack of indication for calories

Education: New York University

Marital Status: Single Location: New York City

Dana is a junior in History at NYU. She recently moves to an apartment off-campus and she lives alone. Sometimes her friends might visit her in her home.

Unable to find ingredients

Time-consuming for watching cooking tutorials

And I drafted the feature list for my chatbot:

Sending Recipes

Giving Cooking Tips

Selecting Measurements

Estimating Time for Cooking

Calculating Calories

Helping with Ingredient Purchasing

And I set some characteristics of the chatbot:

Language English

Tone Casual, friendly, and patient

Response Concise

Use bullet points and highlights

Dana takes a full class load so she is very busy. She does not have a meal plan in dining halls, and she also does not like eating out because it is expensive.

07 Storyboard

Characteristics

Relationship with Technology

but would like to try

but seldomly sends posts

Pain Points

Does not have Alexa at home asking others

Goal

while having enough time for her study

Dana wants to eat egg fried rice

But she does not know how to cook

So she searches for recipes online

And she finds Kitchen Genie in the search results

Dana was intrigued by a cookingassistant chatbot

She gave it a try and followed the recipe

She made a first bowl of egg fried rice on her own

She is very motivated and wants to cook even more

10 Script Design 11 Prototype

The script was designed in Engati: