SM4601_group17_finalReport

SM4601 50683400 Kong king sin FYP Final Report 2008

City University of Hong Kong Department of Computer Science School of Creative Media BScCM Final Year Project 2007-2008 Final Report

Group 17 Hexic: Multimedia interactivity of choreography, sonic and graphic design

Student Name/EID

:

50683400

Student Name/EID

:

Student Name/EID

:

Kong king sin

Programme Code

:

BScCM – sm4601

Supervisor

:

Date

:

Prof IP, Ho Shing Horace Dr. WONG, Kam Wah Ms. Koala Yip 07/05/2007

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

1

SM4601 50683400 Kong king sin FYP Final Report 2008

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

2

SM4601 50683400 Kong king sin FYP Final Report 2008 Summary Acknowledgements Introduction 1.1 Motivation 1.2 Project goal 1.2.1 Aim 1.2.2_Minimum_requirements 1.2.3 Possible further enhancements 1.2.4 Project scope 1.3 Methodology 1.4 Project Schedule & Budgets 1.5 Creation of a prototype 2 Requirement Specifications 2.1 Software Requirement 2.2 Hardware Requirement 2.3 Functional requirement 2.4 Non-Functional requirement 3 Implementations 3.1 Performance 3.2 Motion Detection 3.3 Settings 3.4 Programming 3.6 Interactivity

4 Testing & Debugging

5 Problem Disclosure & Solving 6 Research & References

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

3

SM4601 50683400 Kong king sin FYP Final Report 2008

Summary This project is going to implement computer vision and motion graphic interacts with dancer. Seems programming and Stage performance are difficult to combine together. In previous year, artist experiment the possibility of combine program and performing art together. However, technical problems limited their creation. It also makes general public hard to understand the output of interactive performance.

This project presents the new generation of stage performance to professionals, art lover, and general public. They will enjoy such innovative performance, and thru to know more about New and Interactive media art.

Acknowledgements This project would not have been possible without the help, one way or another, of the following people.

Prof IP, Ho Shing Horace my project supervisor for his continuing support throughout the project’s life cycle.

Dr. WONG, Kam Wah, my project assessor for her feedback on my mid-project report and during the progress meeting of this project.

Ms. Koala Yip, for her valuable advice on stage performance.

My family and friends.

Introduction

1.1 Motivation Performing art is a traditional media. Interactive media is used as communicate human and computer.

This project is trying to investigate and improve the aesthetic value of programming on stage. Showing potential ability of interact with stage performer and director, It trys to integrate program into performance stage. In the traditional performance no matter dancing or drama, performer has to prepare. Also

1.2 Project goal In this section the aim, minimum requirements, possible further enhancements and scope of the project are given. This section acts mainly as a reference point. How the above were derived will become clearer in the oncoming sections of the report.

1.2.1 Aim The aim of this project was defined as being:

“To create and implement programming into stage performance, discovering the possibility of the Real-Time communication between performer, programmer and audience.”

1.2.2 Minimum requirements In order to realize this aim, the following minimum requirements were derived: • To direct live performance with dancer • To manage stage production suitable for live performance • To develop a software tools which will construct a virtual space by allowing dancer to interact • To carry out the communication between dancer and program • To report on the processes of developing, testing and evaluating the live performance.

1.2.3 Possible further enhancements Further enhancements to this project were focused around the live performance that would be created.

During the requirements analysis of the project, requirements were prioritized. Some of the requirements were deemed as essential in order for the project to succeed an others were deemed as non-essential.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

4

SM4601 50683400 Kong king sin FYP Final Report 2008 When the essential requirements were implemented, focus turned to the non-essential requirements and some of these were implemented. A list of which non-essential requirements were implemented and which were not will not be given here due to the number of non-essential requirements and because such a list at this stage would be out of context. However, in order to give an idea of the nature of any enhancements it can be said that possible further enhancements for this live performance fell into one of the following categories: a) They were oriented in providing interaction between audience, dancer, and software tools. b) They provided hardware item into the stage. For example, various lighting, moveable backdrop, which do software tools control.

1.2.4 Project scope The data for this project contains position and gesture form computer vision, motion graphic design, Sound design and choreography (movement design). The live performance management includes casting, show control, motion tracking hardware ,software research ,event planning and investigating the possibility of implemention program in lve peformance .

1.3 Methodology Choosing a suitable methodology for a software project of this type is imperative. This, the author Knew from the outset and this definitely became more apparent as the project progressed.

The main criterion when deciding upon development methodologies was a methodology that would be the priority of each part of the project scope. Due to the main idea of the project is the communication of dancer and computer. The first priority is research for motion detection. At the same time the project prepare the story broad and movement. Program and live performance are developed synchronal.

The project will use dancer action (speed, moving direction, angle of their four limbs, jump), performer position on the stage as the interaction mode.

E.g. if performer punch in the center of the stage, computer generate an explosion in center of the screen.

Furthermore the project wants to use the distance between two performers to interact with computer.

Response (output) Visual The project would like to use both abstract graphic such as ink or solid graphic such fish, 水袖 and addition shape of the dancer shadow.

Sound Music sample from difference song and loops For example, the program defines separated part of the stage. When dancer move to different part that will cause sound looping 4 circles.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

5

SM4601 50683400 Kong king sin FYP Final Report 2008

1.4 Project Schedule & Budgets 1.4.1 Schedule 2007 Dec Program development 2008 Jan Casting Confirm Storyboard Print ad Production 2008 Feb Choreography 2008 Mar Stage Management Studio testing 2008 Apr On Stage testing 2008 May Report / Presentation / Oral Defense Rehearsal On Show 2008 June Follow up work On stage Preparation 2008 July Exhibition Setting Exhibition Task Motion detect application development Motion detect device research Stage management Music composition

Duration (month)

Visual and sound element development Story broad

3

3 1 2 1

1

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

6

SM4601 50683400 Kong king sin FYP Final Report 2008 1.4.2 Budgets & Recourses Item IR Light (I cant found the ID form production center) VEM Ppanmin PANASONIC MULTI MEDIA PROJECTOR (PT-P1SDEA) WITH REMOTE X1, VGA CABLE X 1 PT-P1SDEA IR Pass filter MiniDV Computer Laptop/PC WIFI router Century Stand Projection Screen Costume

Quantity 6

3 3 3 3 1 8 1 2

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

7

SM4601 50683400 Kong king sin FYP Final Report 2008

1.5 Prototype 1.5.1 onstage Testing movement with generative graphic

capture :stage Technical testing1(movement)

capture :stage Technical testing2(movement)

capture :stage Technical testing3(movement)

capture :stage Technical testing4(without performance)

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

8

SM4601 50683400 Kong king sin FYP Final Report 2008

capture :stage Technical testing5(movement)

capture :stage Technical testing6(movement)

1.5.2 on stage setting

capture :stage Technical setting1 (front) this is first try on the closely environment with single capture set. This test program performance.

capture :stage Technical setting2 (side)

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

9

SM4601 50683400 Kong king sin FYP Final Report 2008 1.5.3 caption form generated graphic

screen capture :String1.0

screen capture :String2.0

screen capture :String2.1

screen capture :StringFly1.0

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

10

SM4601 50683400 Kong king sin FYP Final Report 2008

screen capture :StringFly1.0

screen capture :fish1.0

screen capture : fish1.1

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

11

SM4601 50683400 Kong king sin FYP Final Report 2008 1.5.4 Story Broad Hexic Actor:

a Girl A Boy

Introduction Program comes from Chinese philosophy, 1 comes 2, 2 comes 4, 4 comes 8 comes the world, that's what are the world's structure. Elements go where they come. Also disappear from it.

Scenes 1 Time morning Location forest: Girl weak up, dancing is her hobbit. She is happy when she is dancing. She dances on the mountain, forest and the lake; she dances with wind and rain. Her dancing makes birds sing and follow. However nobody can share her happiness. She feel sad because she always alone. She is alone.

Scenes 2 Time sunset Location River One-day girl pass a river, she found interesting. The shadow from the water can follow her movement, this moment girl is happy, she comes to the river/wall everyday. Someone can dance with her. But she discover her shadow only follow her. She feels sad again.

Scenes 3 * Time afternoon Location River Girl still goes to river/wall for her shadow; something special power makes her shadow become live.

Shadow has I own mind. Evolution, a kind of connectivity between shadow and girl, when times go by, shadow has his owns shape, a kind if boy's model. Girl is happy. They create their own thing on the river/ wall.

Scenes 4 finale Mid night Location River However girl are not satisfying enough. She wants boy become real and come to her would. Boy rejects.

Girl uses force. She breaks the wall/river. Pull boy out, boy becomes a human. He is not a spirit anymore. They can come together. When they stay longer. Boy disappear more. Boy only can exit in his world. He cannot go back to his world. Girl tries to stop it. But it is too late. Last boy only leave a bit of light. And disappear in girl's hands. Girl dies together.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

12

SM4601 50683400 Kong king sin FYP Final Report 2008 1.5.5 Costume

Sample custom

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

13

SM4601 50683400 Kong king sin FYP Final Report 2008

2 Requirement Specifications 2.1 Software Requirement 2.1.1 Development tools At the very beginning, Opens and InfoMus Lab’s EyesWeb, after researching the technique and theory will cost the project for long time on coding. Also balancing the project scope and aims, the project chooses the following development tools.

Software detection: Processing + Java language + JMyron Library Processing is an open source programming language and environment for people who want to program images, animation, and interactions. Students, artists, designers, researchers, and hobbyists for learning, prototyping, and production use it. It is created to teach fundamentals of computer programming within a visual context and to serve as a software sketchbook and professional production tool. Artists and designers develop processing as an alternative to proprietary software tools in the same domain. (http:// www.processing.org ,05/06/2008) Myron is the cross-platform, cross-language, open source, video capture and computer vision plug-in.

One core C++ object gets cross-compiled as a handful of high-level language "wrapper" libraries. The wrapper for Java and Processing is called JMyron. The wrapper for Macromedia Director is called WebCamXtra. The aim of the project is to keep computer vision free and easy for the new media education and arts community.(http://webcamxtra.sourceforge.net/ , 05/06/2008) Software Computer Graphic

OPENGL library

Computer vision technique

Frame differencing, Color detection, Edge detection

Sound analysis and MIDI Generate

jmetude, sonia sound library for processing

Core development environment

Processing 135Beta

Computer vision library

JMyron library

Music and loops composition

Apple Soundtrackpro

Texture design

Adobe Illustrator

Texture design

Adobe Photoshop

Draw Stage Set

Goggle Sketchup

Draw Gant chart

Excel

Technically speaking, the project process both computer vision , computer graphic ,sound analysis and sound generate. Due to this is one-man project, C++ and OpenCV development too will spend lot of time on program developing. Thus processing plus various java and processing library and increase project productivity. It is lightweight and open-source, it is available to cross OS platform.

2.2 Hardware Requirement Actually computer vision technique can handle most of the situation and environment. The problem or hardware is that the camera which capture the dancer in front of screen. Consequently camera also capture the projection to computer, feedback will arise. To prevent this situation, the project chooses ir technology to filter visible light. Helping Computer capture infrared vision.

Fire wire has length limited. However RCA cable can deliver video signal for long distance without signal drop. Before input video into computer, converting analog signal to digital do by fire wire adaptor.

Those hardware can decrease workload of computer calculation. Thus improve interactive Hardware Inferred light inferred pass filter RCA cable to fire wire adaptor Projector Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

14

SM4601 50683400 Kong king sin FYP Final Report 2008 Camera Speaker Project screen(infrared absorb surface)

2.3 Functional Requirement

ParticleSystem ArrayList particles; ParticleSystem(int num,Vector3D v) + run() + addParticle() + addParticle(Particle p) boolean dead() Vector3D public float x; public float y; public float z; Vector3D(float x_, float y_, float z_) Vector3D(float x_, float y_) Vector3D() +setX(float x_) +setY(float y_) + setZ(float z_) + setXY(float x_, float y_) + setXYZ(float x_, float y_, float z_) + setXYZ(Vector3D v) + float magnitude() + Vector3D copy() public Vector3D copy(Vector3D v) +add(Vector3D v) +sub(Vector3D v) +mult(float n) +div(float n) +normalize() +limit(float max) + float heading2D() + Vector3D add(Vector3D v1, Vector3D v2) + Vector3D sub(Vector3D v1, Vector3D v2) + Vector3D div(Vector3D v1, float n) + Vector3D mult(Vector3D v1, float n) + float distance (Vector3D v1, Vector3D v2) Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

15

SM4601 50683400 Kong king sin FYP Final Report 2008

2.4 Non-functional Requirement The team developing Hexic is fully aware that a stage performance purpose, as a result, one of our first Nonfunctional Requirements would be performance of main program. Instead of releasing software that is slow and buggy, the project is committed to making the release version of Hexic without any bugs that would interfere with the instance reaction of the software in any way. Also, the project had determined that the software must be high quality of graphic design, the project would design for many kind of motion graphic. Sound design is the concern from the start of Hexic’s development, the project also design for many kind of loops and mix many song for the show. Lastly, Hexic is based heavily on its show control, therefore the project planned to control no only software but also hardware.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

16

SM4601 50683400 Kong king sin FYP Final Report 2008

3 Implementations 3.1 Performance 3.1.1 work flow

Scenes

Sound/Song Generate 1.1sound 1.2Piano solo 1.3Teardrop

Duration

Start time Acting

0:01:00 0:01:00 0:00:40

0:01:00Weak up 0:02:00Miss lover 0:02:40Upset

2Violin

0:01:20

Trap in the 0:04:00box

Human 3(Calexico)

0:01:30

4Human

0:01:30

0:05:30In fantasy Back you reality and 0:07:00depress

Program

Ripple Myron Shadow String and Myron glob Box

Edge + string

Water fish

3.2 Motion Detection Although the project can pre-render or self create image. The project trys to connect program and live performance. JMyron does help the program capture the real time object. JMyron can output a sort of data from the real time image. Through the data program can get the dancer position (x,y,z),speed, and movement gesture each of the x-axis, y-axis, and z-axis . According these data can form more threedimensional of the stage.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

17

SM4601 50683400 Kong king sin FYP Final Report 2008

3.3 Setting ISO view

Top view

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

18

SM4601 50683400 Kong king sin FYP Final Report 2008 Stage design diagram

3.4 Programming The program capture the real-time movement form each camera. Computer gets the x-axis, y-axis, and zaxis image separately. Then calculate useful data from JMyron afterward sending the data through network library to main computer. Main computer use such data to generate graphic also it uses sonia library analysis input song/ music.

3.5 Interactivity While dancer perform on the stage, all of the movement can generate scenes related effect, it can be sound , computer graphic, collisions. Furthermore ,at specific scenes , dancer needs to fulfill some pre-set statement to trigger the event.

Same-time director can control the character inside the virtual world. Through these director can interact with dancer. Director can communicate with dancer through the program.

Also Dancers can interact with each other , in one-sconce dancers perform with others silhouette.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

19

SM4601 50683400 Kong king sin FYP Final Report 2008

4 Testing & Rehearsal 4.1 Testing In this section , aim to test the integrate with movement and computer performance.

The first prototype is very base techniques apply into the movement. The outcome in not very well. No scenario, only have simply motion tracking with some fly and bubble appear. After intrim report the computer software waes designed base on the story board.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

20

SM4601 50683400 Kong king sin FYP Final Report 2008 Choreographic This is the most important part of this project, What has this part done is directing the choreographer design the movement base on the story broad, After that, we discuss the possibility of the interaction, what new feature we can add into the dance. Should we use lifting? Should we dance outside the stage?

Which kind of movement is suitable for the show?

As a result we decide the dancer will dance with a virtual character that will combine with another dancer and computer effect.

Reference: http://www.vimeo.com/920818

4.2 Rehearsal Coming soon

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

21

SM4601 50683400 Kong king sin FYP Final Report 2008

5 Problem Disclosure & Solving 5.1 Venue The project aims to produce a live , real-time performance, the biggest consideration is space for performance. This tried to deal with this limitation at the very beginning. Venue priority was Osage Gellary,Fringe Club, Y-space dance Studio, CCDC Studio, and SCM TV studio. However first 4 venues are not available at ideal period. Last performance place was chosen to be SCM TvStudio.

TV Studio is not the idealess place for dance, include lighting , and floor sound system. However this is impossible to fight for dance space from other organization.

Also testing place is other difference from the project. Program development can be done in indoor laboratory. For the actual environment is totally different from desktop. Enough spaces can help the project do smoothly.

Solving To prevent waste time on fine tune program while rehearsal and before final presentation. The project demonstrates some of the situation of the stage environment. Part f the scenes have been executed several time.

5.2 Hardware When demonstrating the hard ware, we found that one infrared light is not enough to light up all the stage. In the final presentation, the project will use 4 or more infrared to assist to light the stage. This avoids capture image light flare.

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

22

SM4601 50683400 Kong king sin FYP Final Report 2008

6 Research & References 6.1 References 6.1.1 book references Laban Movement Studies and actor training [electronic resource] : An experiential and theoretical course for training actors in physical awareness and expressivity. Author: Casciero, Thomas.

Processing: a programming handbook for visual designers and artists / Casey Reas, Ben Fry. Author Reas, Casey. Publisher Cambridge, Mass. : MIT Press, c2007. ISBN 9780262182621 (hardcover : alk. paper) Works. 1998. L'oeuvre musicale [sound recording] / Pierre Schaeffer. Author Schaeffer, Pierre, 1910Publisher Paris [France] : INA-GRM ; Albany, NY : EMF ; [S.l.] : distribution, Musicdisc, p1998.

On sonic art / by Trevor Wishart.Author Wishart, Trevor. Publisher Amsterdam : Harwood Academic Publishers, c1996.Edition New and rev. ed. / edited by Simon Emmerson.

6.1.2 online reference Drop Spin Fade http://www.youtube.com/watch?v=XOWyAlgfMFY Shadow Monster

a string

http://www.worthersoriginal.com/viki/#page=shadowmonsters

http://tokage.cafe24.com/string.htm TENORI-ON http://www.global.yamaha.com/tenori-on/ http://www.youtube.com/watch?v=_SGwDhKTrwU Chris O’shea http://www.chrisoshea.org/projects/muon/video/

6.2 Research 漂移: Interactive media creative team (Shanghai and Singapore Arts Festivals)

6.3 Observation Osage Siren 藍胡子城堡 http://www.kungmusic.net/

Prepared by Kong king sin, Supervised by Prof IP, Ho Shing Horace, Dr. WONG, Kam Wah, Ms. Koala Yip

23