TsungHsun Ku Thesis Project MAS CAAD ETH Zurich

Mental Object in Perceptual Space Tsung-Hsun Ku

Master of Advanced Studies Thesis Project Prof. Ludger Hovestadt, Vera B端hlman

Chair for Computer Aided Architectural Design Department for Architecture ETH Z端rich

1

After silence, that which comes nearest to expressing the inexpressible is music.

-Aldous Huxley

2

|| -- Content -- ||

intro bridge music perception mechanism application code references

3

Mental Object in Perceptual Space// intro

|| -- Intro -- ||

4

Mental Object in Perceptual Space// intro

Maschine Mensch (Fritz Kahn, 1922)

People all have abilities to sense and feel this world, in the field of architecture, we sense the spaces with listening, sight, touch. Before these information from outer world entering to our mind, there’re 3 phases for people dealing with information in consecutive course, receptor, then brain, and the last, mind. We receive them by our receptors of senses, and process these inputs through our brains and consciousness, eventually, the perceptions are responded to our mind, the mental images within our perceptual space produced.

While we listening the music, it occurs to our mind with mental image within space as well, furthermore, it’s an inward accumulation of dynamic and cognitive reactions.

5

Mental Object in Perceptual Space// intro

However, there’s no approach to describe the essence of an object or entity precisely, in this case we can not expound the state or form of this perceptual mechanism at all, the totality of all arguments and articulations describing it is the object or entity itself.

Therefore, the information consist of what we received has to be formed as an objective entity which will be projected by our subjective perceptions derived from the recollections. All the actions on these process happened in a virtual field, an imaginary space, are the delineation of mental images itself in cognitive level.

As concept claimed by Deleuze, the plane of immanence, “there is a pure plane of immanence, univocality, composition, upon which everything is given, upon which unformed elements and materials dance that are distinguished from one another only by their speed and that enter into this or that individuated assemblage depending on their connections, their relations of movement.� What we call virtual is not something that lacks reality, but something that enters into a process of actualization by following the plane that gives it its own reality.

On the side of information from outer world, music, it happens in the present, remain objective, and unambiguous. The physical attributes of music are conveyed through medium, during the process, constitute an objective carrier loading consecutive audio inputs, which defines as an audio carrier.

6

Mental Object in Perceptual Space// intro

From the aspect of human perception, lyrics, timbres, rhythm, melody, tune, all of them provoke the corresponding objects and scenes in memories. These imaginations come from the accumulation of recollection. At the moment, the recollection breaks the chronological order, overlays themselves one another, become a composite of the past.

The composite from recollections is subjective, from inner side, from the past, at the meanwhile, the composite from music is objective from outer side, in the present. This composite of recollections from the past and music in the present reflect each other, construct the our own imaginary space in mind, a virtual field. In that field, this objective carrier intertwine with subjective composite of recollection, the perceptual images, yield an entity neither belongs to music nor human perception, which is no longer possess hierarchy but rather unique feature.

projection

objective entity

subjective images

music (information from outer world)

virtual field

7

Mental Object in Perceptual Space// intro

Perceptual images influence audio carrier, audio carrier respond to images, the entity is dynamic, existing within these two movements, but also crystallizing in every very moment. The difference between subjectivity and objectivity becomes obscure, the boundary is vanished, the functionalities are invalid, like the plane of immanence, like the process toward virtuality but not virtuality itself. It’s a union relying on topological relation, heterogeneous composites, flowing to the world consisted of sound, the body immersed in sound, the field fulfilled with the strength of sound.

8

Mental Object in Perceptual Space// bridge

|| -- Bridge -- ||

9

Mental Object in Perceptual Space// bridge

In this thesis, I attempt to explore the abstraction of perception and sound in virtual field, as well as the application of Self-Organizing Map in mechanism of human perception.

Following the procedure of entire process to get the entity representing the mechanism of human perception and sound in virtual space, an abstract object connected music and human perception.

To achieve this entity in virtual space, first step is to process the music input, by utilising Fast Fourier Transform, we can get the all amplitude and spectrum in frequency domain at every time interval. With extracting and filtering the data, we achieve an objective carrier in 3D space representing the ontology of music, the mutual representation of music in time and frequency domain.

After the objective carrier created, we search the images related to music as subjective inputs, which are based on the keyword from lyrics or any other related background descriptions. These image are projected onto objective carrier by algorithm, Self-Organizing Map, with the form of points cloud based on RGB value mapping in 3 axes. By SOM, we are able to overlay the essences of images onto the carrier in 3D space, representing the entity mixed objectivity

10

Mental Object in Perceptual Space// bridge

11

Mental Object in Perceptual Space// music

|| -- Music -- ||

12

Mental Object in Perceptual Space// music

Any piece of music has its own unique composition, including timbre, rhythm, speed, chord, amplitude, motive, multiple factors are arranged in intricate feature at every moment. Along with line of time, these features are connected inseparably, like infinite pearls on a string, a indiscernible entity. People can extract a segment with any length from this string, analyze it as a fragment of music, however, it’s not possible to count amount of pearls within. It’s stratified while we observe it with infinitely short distance, on the contrary, it’s smooth while we observe it with infinitely far distance.

Through new technological possibilities of recording sound and the synthetic creation of sound, Euclidian space led to a relative, multi-dimensional concept of space, as well as to artificial, composable musical spaces. The new conceptions of space fostered the notion of abstract space far beyond any physical existence, to the virtual, topological and topophonic acoustic spaces with which we operate today.

Music is a continuum moving in time domain, we can not fix it in visible Euclidean space. In digital era, music is interpreted as digital data set, it becomes the metadata which we can change it’s phenotype depending on the dimensions we want it located. As the music score is it’s representation in frequency domain, people can extract the data mathematically with equal time interval.

13

Mental Object in Perceptual Space// music

Therefore, I consider the music as the slices of continuum in time domain, convert it into frequency domain, as a static structure, a initial design grammar.

As interdisciplinary protagonist in architecture and music, Iannis Xenakis, performed his work, Metastasis in 1955, and the design proposal of Philip pavilion to Le Corbusier with same approach in 1958.

I adopted the main 3 factors in sound as values in 3 axes in Euclidean space by analogy. Through Fast Fourier Transform, the sound is extended on virtual plane at every temporal unit. Sound waves are transformed into sound points distributed at every layer of frequency domain, meanwhile, piling up along the third axis.

Metastasis (Xenakis, 1971)

14

Mental Object in Perceptual Space// music

This result is absolutely objective, featuring the synthesis of sound algorithmically, a transitional carrier from temporality to spatiality. As the analogy of music score in 3-Dimensions, it remains the realm out of time and space.

I attempted to find a balanced point showing the ontology of music between it’s mathematical structure and design grammar of 3-Dimensions. It has to be objective to interact with image data sets collected subjectively in next stage.

In this thesis, I confine the representation of the ontology of music in a cube with specific size. To feature the geometrical character from music and simplify the parameters, only the frequency band with the largest amplitude will be extracted.

time

Slices of music

15

Mental Object in Perceptual Space// music

time

amplitude

frequency

Audio Carrier performed in points cloud

16

Mental Object in Perceptual Space// perception

|| -- Perception -- ||

17

Mental Object in Perceptual Space// perception

Present, past and recolection

In each moment that we inhabit in the present there exists, a split between a present that passes, and a past which is preserved, and the present is a dynamic interpenetration of past and future. The aspect of our lived world that is here for us right now.

18

Mental Object in Perceptual Space// perception

19

Mental Object in Perceptual Space// perception

Things we experienced from outside world provoke the corresponding objects and scenes in memories. These imaginations come from the accumulation of recollection. At the moment, the recollection breaks the chronological order, overlays themselves one another, become a composite of the past.

20

Mental Object in Perceptual Space// perception

Music can be considered as an aggregation of creator’s thoughts and background, from the moment music is finished, it engraves irreplaceable point in the domain representing time as vertical axis and perception as horizontal axis. For searching subjective images as the accumulation of our perception to the music, keywords of the music are used in internet search engine, e.g. Google. Those keywords are chosen, filtered from the context of music, lyrics, author’s biography.

Through the mapping of those images, a subjective montage is acquired. The sentiment and subjectivity influences the proportion of each kind of images mapped in the montage, which exactly represents one’s perception to specific music through the selection of images.

21

Mental Object in Perceptual Space// perception

The subjective montage of images is converted to data set as an informational object, the ontology of image. Those images data set can be applied as vector inputs in Self-Organizing Map and projected onto audio carrier. And again, the values in 3 axes of Euclidean space is used as an analogy of RGB values of image pixels. The RGB value of every single pixel will be mapped into the coordinate of a point within the space. In this thesis, I confine all the image having the same amount of pixels to amplify the characteristic of essence of each image.

In this approach of featuring the character, image itself get rid of the pixels order, meanwhile, the cognitive meaning from images is no longer discerned but can be processed as feature vector for SOM in the virtual field. The subjective perception is preformed in geometrical appearance, the form, distribution represent and extend one’s consciousness into visible aggregation.

lyrics writer’s background song’s background

keywords

singer’s impression

22

search engine

images

Mental Object in Perceptual Space// perception

r

b

g

Subjective Image performed in points cloud

23

Mental Object in Perceptual Space// mechanism

|| -- Mechanism -- ||

24

Mental Object in Perceptual Space// mechanism

Self-Organizing Map (SOM) is a data driven algorithm with unsupervised learning, utilise the vector input from data as training data, teaching a set of sequential nodes which is an initial state. During the process of training, it yields the distribution of nodes reflecting the relationship within imported data.

The basic principle of Self-Organizing Map is the artificial neural network simulating the characteristics that cells are aggregated according to the similar functions and biological traits in human brain. All the data are reflected onto a node according to the similarity between feature vectors and nodes, the close the Euclidean distance between nodes, the higher the similarity.

As same principle in human brain, music as the objective carrier and initial set of data will be influenced by the image montage selected by personal recollections. To apply Self-Organising Map, all of them need to be presented as the form of points cloud. Through importing the image data in order, the distribution of points demonstrates how images in form of pixels aggregate on this audio carrier based on similarity, showing how human perception projects onto the information from outer world. More dense the distribution, performing more correlated in the traits of imported image, it implies that people who select images has more common inclinations.

25

Mental Object in Perceptual Space// mechanism

Audio Input

time

frequency

amplitude

Objective Carrier

Perceptual Image

r

g

b

Subjective Composite

26

Mental Object in Perceptual Space// mechanism

Subjective Images reflected onto Objective Carrier

27

Mental Object in Perceptual Space// mechanism

Conversely, it performs more diverse typologies of images composing the montage, more even in the proportion of each kind of images, it implies that people possesses wider biases in image selection.

Music data and image data are performing within this imaginary space, remaining the topology, communicating and learning. Both of them are connected by a new single entity, an assemblage consist of different typologies of data sets.

Eventually, the aforementioned distribution of points shows the moment that human recollection from the past influencing the present in ontology aspect.

Namely, the abstraction of the state of human cognition processing, which is beyond the limitations of the geometrical space, and into the experience of pure space perceived space and time.

Music

Object

28

Perception

Mental Object in Perceptual Space// application

|| -- Application -- ||

29

Mental Object in Perceptual Space// application

One's portrait image doesn't just conveys the visible appearance the but also the his own invisible impression and personality. For a famous singer, a pop star, beside his music works, our perception are deeply influenced by his strong personal impression as well.

In this thesis, I adopt 3 famous singer’s music as my attempt to perform how the human perception orchestrates with music yielding abstract objects based on the aforementioned methodology.

These musicians all made the history. Their profile pictures are becoming the icons, instead of simple images, expressing the thoughts behind their works and the styles of their performances, marking the irreplaceable notations along the time line.

At the present, we can perceive the impressions vividly through the singers' images. The memories in time domain and the voices in acoustic domain break the limitation and overlay on the images, which becoming the identity beyond time and space. Compared to other singers, their works can provoke more subjective perceptions and corresponding mental images in our minds.

30

Mental Object in Perceptual Space// application

Michael Jackson song/ Billie Jean

David Bowie

song/ Space Oddity

John Lennon song/ Imagine

31

Mental Object in Perceptual Space// application

Billie Jean She was more like a beauty queen from a movie scene I said don't mind, but what do you mean I am the one Who will dance on the floor in the round She said I am the one who will dance on the floor in the round She told me her name was Billie Jean, as she caused a scene Then every head turned with eyes that dreamed of being the one Who will dance on the floor in the round People always told me be careful of what you do And don't go around breaking young girls' hearts And mother always told me be careful of who you love And be careful of what you do cause the lie becomes the truth Billie Jean is not my lover She's just a girl who claims that I am the one But the kid is not my son She says I am the one, but the kid is not my son For forty days and forty nights The law was on her side But who can stand when she's in demand Her schemes and plans cause we danced on the floor in the round So take my strong advice, just remember to always think twice (do think twice.)

She told my baby we'd danced 'til three Then she looked at me She showed a photo of a baby crying His eyes looked like mine Go on dance on the floor in the round, baby People always told me be careful of what you do And don't go around breaking young girls' hearts She came and stood right by me Then the smell of sweet perfume This happened much too soon She called me to her room Billie Jean is not my lover She's just a girl who claims that I am the one But the kid is not my son Billie Jean is not my lover She's just a girl who claims that I am the one But the kid is not my son Billie Jean is not my lover Billie Jean is not my lover Billie Jean is not my lover

32

Mental Object in Perceptual Space// application

33

Mental Object in Perceptual Space// application

Subjective Points Images Cloud Collection of “BilleofJean” “Bille Jean”

34

Mental Object in Perceptual Space// application

Points Cloud Collection of “Bille Jean” performing the RGB values of subjective images

35

Mental Object in Perceptual Space// application

36

Mental Object in Perceptual Space// application

37

Mental Object in Perceptual Space// application

38

Mental Object in Perceptual Space// application

Space Oddity Ground Control to Major Tom Ground Control to Major Tom Take your protein pills and put your helmet on Ground Control to Major Tom Commencing countdown, engines on, Check ignition and may God's love be with you Ten, Nine, Eight, Seven, Six, Five, Four, Three, Two, One, Liftoff This is Ground Control to Major Tom You've really made the grade And the papers want to know whose shirts you wear Now it's time to leave the capsule if you dare This is Major Tom to Ground Control I'm stepping through the door And I'm floating in a most peculiar way And the stars look very different today For here Am I sitting in a tin can Far above the world Planet Earth is blue And there's nothing I can do Though I'm past one hundred thousand miles I'm feeling very still And I think my spaceship knows which way to go Tell my wife I love her very much she knows Ground Control to Major Tom Your circuit's dead, there's something wrong Can you hear me, Major Tom? Can you hear me, Major Tom? Can you hear me, Major Tom? Can you....

Here am I floating round my tin can Far above the Moon Planet Earth is blue And there's nothing I can do.

39

Mental Object in Perceptual Space// application

Points Cloud of “Space Oddity”

40

Mental Object in Perceptual Space// application

Subjective Images Collection of “Space Oddity”

41

Mental Object in Perceptual Space// application

Points Cloud Collection of “Space Oddity” performing the RGB values of subjective images

42

Mental Object in Perceptual Space// application

43

Mental Object in Perceptual Space// application

44

Mental Object in Perceptual Space// application

45

Mental Object in Perceptual Space// application

Imagine Imagine there's no heaven It's easy if you try No hell below us Above us only sky Imagine all the people Living for today...

Imagine there's no countries It isn't hard to do Nothing to kill or die for And no religion too Imagine all the people Living life in peace...

You may say I'm a dreamer But I'm not the only one I hope someday you'll join us And the world will be as one Imagine no possessions I wonder if you can No need for greed or hunger A brotherhood of man Imagine all the people Sharing all the world...

You may say I'm a dreamer But I'm not the only one I hope someday you'll join us And the world will live as one

46

Mental Object in Perceptual Space// application

Points Cloud of “Imagine”

47

Mental Object in Perceptual Space// application

Subjective Images Collection of “Imagine”

48

Mental Object in Perceptual Space// application

Points Cloud Collection of “Imagine” performing the RGB values of subjective images

49

Mental Object in Perceptual Space// application

50

Mental Object in Perceptual Space// application

51

Mental Object in Perceptual Space// application

52

Mental Object in Perceptual Space// code

|| -- Code -- ||

53

Mental Object in Perceptual Space// code

|| -- Main code -- ||

import ddf.minim.analysis.*; import ddf.minim.*;

////////////////////// //SOM variables

import peasy.*; import peasy.org.apache.commons.math.*; import peasy.org.apache.commons.math.geometry.*;

int bounding = 1000; int framerate = 60; int vecCount = 0; ////////////////////// //Music variables

import import import import import import import import

toxi.processing.*; toxi.geom.*; toxi.geom.mesh.*; toxi.geom.mesh.subdiv.*; toxi.volume.*; toxi.color.*; toxi.math.*; toxi.util.*;

float time = -bounding; Vec3D SCALE = new Vec3D(1,1,1).scaleSelf(bounding*2); ////////////////////// //Image variables int n = 64; int order = 0; int num = 0;

import processing.dxf.*; import controlP5.*; import java.lang.Math.*; import java.util.*; import processing.opengl.*;

////////////////////// //Light PVector l[];//lights stored as vectors float str;

import nervoussystem.obj.*; ////////////////////// //Class

/////////////////////// //Boolean

//---gui---------

Boolean stop = false; Boolean smooth = false; Boolean rotate = false; Boolean fill = false; Boolean input = false; Boolean turn =true; Boolean meshCreate = false; Boolean somOn = false; Boolean resetSom = false; Boolean backup = false; Boolean export = false; Boolean record = false; ////////////////////// //Timer int count = 0; int bCount = 0; float t1 = 0; int t = 0; /////////////////////

PeasyCam cam; ControlP5 controlP5; PMatrix3D currCameraMatrix; PGraphics3D g3; //---toxiclibs--WETriangleMesh mesh; ToxiclibsSupport gfx; //---minim------Minim minim; //---custum-----SOM somgo; MusicMesh music; ImageInput imginput;

void setup(){ ////////////////////// //List

ArrayList<Vec3D> ArrayList<Vec3D> ArrayList<Vec3D> ArrayList<Vec3D>

size(1280, 700, OPENGL); stroke(250); frameRate(framerate); smooth(); strokeWeight(1); initLight(); guiSetting();

vecList = new ArrayList<Vec3D>(); backupList = new ArrayList<Vec3D>(); vertBackup = new ArrayList<Vec3D>(); pointsList = new ArrayList<Vec3D>();

54

Mental Object in Perceptual Space// code

g3 = (PGraphics3D)g; gfx = new ToxiclibsSupport(this); cam = new PeasyCam(this, 0, 0, 0, 1300);

} bCount ++; } if(somOn && !music.song.isPlaying()){ backup = false; resetSom = false; bCount = 0; //music.song.pause(); somgo.somRun(); }

minim = new Minim(this); music = new MusicMesh(minim,gfx); imginput = new ImageInput(n); imginput.getVecList(order); somgo = new SOM(); }

if (meshCreate){ music.generateMesh(); if(export){ music.exportMesh(); } }

void draw(){ background(0); smooth(); //lights(); gui(); pushMatrix(); noFill(); stroke(150); strokeWeight(0.5); box(bounding*2); popMatrix();

if(!rotate){ //imginput.displayVec(); } if(frameCount%framerate == 0){ t++; } }

if (fill){ fill(200); } t1 += 0.005f; if (rotate){ camera(); camera (2800*cos(t1), 2800*sin(t1), 0, 0, 0, 0, 0, 0, -1); } info(); imginput.getVecList(order); music.musicRun(); if(backup && !music.song.isPlaying() && !somOn){ if (bCount < 1){ for(Vec3D p : pointsList){ backupList.add(p.copy()); } } bCount ++; } if (resetSom && !music.song.isPlaying() && !somOn){ if (bCount < 1){ pointsList.clear(); pointsList.addAll(backupList); backupList.clear(); vecCount = 0; //resetSom = false;

55

void initLight() {//by Marius Watz randomSeed(11); l=new PVector[7]; for(int i=0; i<l.length; i++) { str=random(TWO_PI); l[i]=new PVector(cos(str)*10,0.3,sin(str)*10); } str=random(120,180); }

Mental Object in Perceptual Space// code

|| -- Class of Self-Organizing Map -- ||

for (int i = 1; i < iteration+1; i++){

class SOM{

int int int int

float resolution = (float)((Math.log(iteration) Math.log(i))/Math.log(iteration));

fr = 80; iteration = 15; pow = 15; root = 2;

SOM(){

Vec3D dV = vec.sub(nearestSom); dV.normalize(); float scale = (float)resolution*6; dV.scaleSelf(scale);

}

Vec3D newSom = nearestSom.addSelf(dV);

void somRun(){

int index = getIndex(pointsList,nearestSom);

if (vecCount < vecList.size()){

pointsList.set(index,newSom);

Vec3D vec = vecList.get(vecCount);// if (!stop){ pushMatrix(); translate(vec.x,vec.y,vec.z); sphere(5); popMatrix(); pushMatrix(); translate(vec.x+bounding*2.5,vec.y,vec.z); sphere(5); popMatrix(); som(vec);

for (int j = 1; j <= pow; j++){ ArrayList<Vec3D> level = (ArrayList)levelList.get(j); float amp =pow(root , -j); for (Vec3D som : level){ Vec3D dV2 = vec.sub(som); dV2.normalize(); float scale2 = (float)amp*resolution*6; dV2.scaleSelf(scale2); Vec3D newSom2 = som.addSelf(dV2); //int index2 = pointsList.indexOf(som); int index2 = getIndex(pointsList,som); //println(" index2 is : " + index2); pointsList.set(index2,newSom2);

} } vecCount++; if (vecCount == vecList.size()-1) { print("Finished!!! the amount of vectors is " + vecList.size() + " and the amount of Vertices is " + pointsList.size()); vecCount =0; if(order < 50){ order+=1; } if(somOn && order == 50){

} } } } ArrayList getAllNeighbor(Vec3D target,int level){ int targetIndex = getIndex(pointsList,target);

somOn = false;

if(targetIndex < 0){ targetIndex = 0; }

} } }

ArrayList<Vec3D> allNeighborList = new ArrayList<Vec3D>();

void som(Vec3D vec){

ArrayList allSepNeighborList = new ArrayList();

Vec3D nearestSom = (Vec3D)getClosestPoint(pointsList,vec); ArrayList levelList =

allNeighborList.add(target); allSepNeighborList.add(target);

56

Mental Object in Perceptual Space// code

for (int i = 1; i <= level ; i++){ ArrayList<Vec3D> tempList = new ArrayList<Vec3D>(); int m = targetIndex - i; int n = targetIndex + i; if (m < 0){ int m2 = (pointsList.size()-1)+m; Vec3D a = (Vec3D)pointsList.get(m2);// tempList.add(a); } else { Vec3D a = (Vec3D)pointsList.get(m); // tempList.add(a); } if (n > (pointsList.size()-1)){ int n2 = n - pointsList.size()+1; if(n2 > (pointsList.size()-1)){ n2 = pointsList.size()-1; } Vec3D b = (Vec3D)pointsList.get(n2);// tempList.add(b); }else{ Vec3D b = (Vec3D)pointsList.get(n);// tempList.add(b); } allSepNeighborList.add(tempList); allNeighborList.addAll(tempList); } return allSepNeighborList; } Vec3D getClosestPoint(ArrayList<Vec3D> ptsList, Vec3D target){ ArrayList distanceList = new ArrayList(); for (Vec3D p : ptsList){ float d = p.distanceTo(target); distanceList.add((float)d); } int minIndex = distanceList.indexOf(Collections.min(distanceList)); Vec3D closePt = (Vec3D)ptsList.get(minIndex);// return closePt; } int getIndex(ArrayList<Vec3D> ptsList, Vec3D target){ HashMap<Vec3D,Integer> hmap = new HashMap<Vec3D,Integer>(); for (int i =0 ; i < ptsList.size();i++){ Vec3D p = (Vec3D)ptsList.get(i);// hmap.put(p,i); } if(ptsList.contains(target)){ int index = hmap.get(target); return index; }else{ return 0; } } }

57

Mental Object in Perceptual Space// code

|| -- Class of Music Processing -- ||

class MusicMesh { ToxiclibsSupport gfx; VolumetricSpace volume; IsoSurface surface; VolumetricBrush brush; WETriangleMesh mesh;

void musicRun(){ makeFFT(); if(frameCount%density ==0 && time < bounding && (amp < bounding && amp > -bounding) && (freq < bounding && freq > -bounding)){ if(song.isPlaying()&& t < 211){ pointsList.add(new Vec3D(amp,freq,time)); //backupList.add(new Vec3D(amp,freq,time)); time += step; } } musicPtsDisplay();

Minim minim; AudioPlayer song; AudioMetaData meta; FFT fftL; FFT fftR; BeatDetect beat; BeatListener bl;

}

ArrayList<Float> highValueList = new ArrayList<Float>(); //ArrayList pointsList = new ArrayList(); Vec3D SCALE = new Vec3D(1,1,1).scaleSelf(bounding*2); float float float float float float float int int int int int int

ISO_THRESHOLD = 0.3; brushDensity=1; step; theta = 0.001; freq; amp; maxValue;

void musicPtsDisplay(){ //backupList.clear(); if (record) { beginRecord("nervoussystem.obj.OBJExport", "filename.obj"); } for (int i =0; i < pointsList.size(); i++){ Vec3D p = (Vec3D)pointsList.get(i); strokeWeight(2); if(beat.isKick() && t < 211){ p.x = p.x*cos(theta) - p.y*sin(theta); p.y = p.x*sin(theta) + p.y*cos(theta); pointsList.set(i,new Vec3D(p.x,p.y,p.z)); } point(p.x,p.y,p.z);

DIM=100; brushSize = 60; cuePos = 0; density = 10; totalSec; count =0;

} if (record) { endRecord(); record = false; }

MusicMesh(Minim _minim,ToxiclibsSupport _gfx){ this.minim = _minim; this.gfx = _gfx; //this.song = _song; song = minim.loadFile("Imagine.mp3", 512); song.loop(); meta = song.getMetaData(); fftR = new FFT( song.bufferSize(), song.sampleRate() ); beat = new BeatDetect(song.bufferSize(), song.sampleRate()); bl = new BeatListener(beat, song); totalSec = ((int)(meta.length())/1000)-20; //step = (bounding*2*density) / (framerate*210); step = 1.7; getMaxAmp(); volume = new VolumetricSpaceArray(SCALE,DIM,DIM,DIM); surface = new HashIsoSurface(volume); brush = new RoundBrush(volume,brushSize); }

58

} void makeFFT(){ fftR.forward( song.mix ); int highest = 0; for (int i = 0; i < fftR.specSize()-1; i++){ float adjust = log(i+1)/log(2); float ampValue = log(fftR.getBand(i)*adjust+1)*200; float adjustH = log(highest+1)/log(2); float highestValue = log(fftR.getBand(highest)*adjustH+1)*200; if (ampValue > highestValue ){ highest = i; } } float adjustH = log(highest+1)/log(2); float highestAmp = log(fftR.getBand(highest)*adjustH+1)*200;

Mental Object in Perceptual Space// code

amp = map(highestAmp ,maxValue/2 , maxValue,-bounding+200, bounding-200); freq = map( highest , 0 , fftR.specSize()/6 ,-bounding+200, bounding-200); } void getMaxAmp(){ int bufferMillis = int((song.bufferSize()*1000) / song.sampleRate()); while(cuePos < meta.length()) { int highest = 0; song.cue(cuePos); fftR.forward( song.mix ); for (int i = 0; i < fftR.specSize()-1; i++){ float adjust = log(i+1)/log(2); float ampValue = log(fftR.getBand(i)*adjust+1)*200; float adjustH = log(highest+1)/log(2); float highestValue = log(fftR.getBand(highest)*adjustH+1)*200; if (ampValue > highestValue ){ highest = i; } } float adjustH = log(highest+1)/log(2); float highestAmp = log(fftR.getBand(highest)*adjustH+1)*200; highValueList.add(highestAmp); cuePos += (bufferMillis*2); } int index = highValueList.indexOf(Collections.max(highValueList)); maxValue = (float)highValueList.get(index); } void generateMesh(){ if(!smooth){ count = 0; mesh = new WETriangleMesh(); volume.closeSides(); surface.reset(); surface.computeSurfaceMesh(mesh,ISO_THRESHOLD); for (int i =0; i < pointsList.size()-1; i++){ Vec3D p = (Vec3D)pointsList.get(i); brush.drawAtAbsolutePos(p,brushDensity); } } gfx.mesh(mesh,true); if (smooth){ count ++; if (count < 40){ new LaplacianSmooth().filter(mesh, 2); } } } void exportMesh(){ TriangleMesh export=new TriangleMesh(); export.addMesh(mesh); export.saveAsSTL(sketchPath("som.stl")); } }

59

Mental Object in Perceptual Space// code

|| -- Class of Image Processing -- ||

|| -- Class of BeatListener -- ||

class ImageInput{ int n; ArrayList<ArrayList> allList = new ArrayList<ArrayList>(); int limit = bounding-150; ImageInput(int _n){ n = _n; PImage[] images = new PImage[n]; for ( int i = 0; i< images.length; i++ ){ images[i] = loadImage( i + ".jpg" ); } for(int i = 0; i < images.length; i++){ ArrayList<Vec3D> tempList = new ArrayList<Vec3D>(); for (int m = 0; m < images[i].width; m++) { for (int n = 0; n < images[i].height; n++ ) { int loc = m + n*images[i].width; float r = red(images[i].pixels[loc]); float g = green(images[i].pixels[loc]); float b = blue(images[i].pixels[loc]); float x = map(r,0,255,limit,-limit); float y = map(g,0,255,limit,-limit); float z = map(b,0,255,limit,-limit); float newX = adjustment(x); float newY = adjustment(y); float newZ = adjustment(z);

class BeatListener implements AudioListener { private BeatDetect beat; private AudioPlayer source;

tempList.add(new Vec3D(newX,newY,newZ)); } } allList.add(tempList); } } void getVecList(int order){ vecList = (ArrayList)allList.get(order); } void displayVec(){ pushMatrix(); translate(bounding*2.5,0,0); noFill(); stroke(150); strokeWeight(0.5); box(bounding*2); popMatrix(); for (Vec3D p : vecList){ float a = map(p.x,limit,-limit,0,255); float b = map(p.y,limit,-limit,0,255); float c = map(p.z,limit,-limit,0,255); stroke(a,b,c); strokeWeight(2); point(p.x + (bounding*2.5),p.y,p.z); } } float adjustment(float var){ float y = abs(var); float newVar = ((limit- y)/limit)*var*2 + var; return newVar; } }

60

BeatListener(BeatDetect beat, AudioPlayer source){ this.source = source; this.source.addListener(this); this.beat = beat; } void samples(float[] samps){ beat.detect(source.mix); } void samples(float[] sampsL, float[] sampsR){ beat.detect(source.mix); } }

Mental Object in Perceptual Space// code

|| -- Function of GUI -- ||

void info(){ cam.beginHUD(); textSize(13); text("Time of Processing : " + t , 25, 500 ); if(somOn && music.song.isPlaying()){ text("You have to pause music first", 25, 520 ); } if (meshCreate){ text("Mesh Mode", 25, 540 ); } else { text("Particle Mode", 25, 540 ); } if (somOn){ text("Self-Organizing Map Mode",25,560); }else{ if(music.song.isPlaying()){ text("Music is playing",25,560); }else{ text("Music pause",25,560); } } if (vecCount < vecList.size()){ text("Vector " + vecCount + " is proceeding ",25,600); }else{ text("Finished all of Vecotrs !!",25,600); } text("Vector Source : Image " + (order +1),25,620); text("The Name of Song : " + music.meta.title() ,25,640); text("Amount of SOM : " + pointsList.size(), 25, 660 ); text("Amount of Vectors : " + vecList.size(), 25 ,680); cam.endHUD(); } void guiSetting(){ controlP5 = new ControlP5(this); controlP5.addButton("Music_Pause", 0, 20, 20, 70, 30); controlP5.addButton("Music_Play", 0, 100, 20, 70, 30); controlP5.addButton("SOM_Run", 0, 20, 55, 150, 30); controlP5.addButton("SOM_Pause", 0, 20, 90, 150, 30); controlP5.addButton("Reset_SOM", 0, 20,125, 150, 20); controlP5.addButton("Backup_SOM", 0, 20, 150, 150, 20); controlP5.addButton("Create_Mesh", 0, 20, 175, 150, 30); controlP5.addButton("Smooth_Mesh", 0, 20, 210, 150, 20); controlP5.addButton("Rotate_Cam", 0, 20, 235, 150, 20); controlP5.addButton("Prev_Input", 0, 20, 260, 70, 30); controlP5.addButton("Next_Input", 0, 100 , 260, 70, 30); controlP5.setAutoDraw(false); } void keyPressed() { if ( key == 'e' ) { export = !export; } if (key == 'r') { record = true; } if (key == 'f') { fill = !fill; } }

61

void gui() { if (mouseX<200&&mouseY<200)cam.setActive(false); if (mouseX>200&&mouseY>200)cam.setActive(true); cam.beginHUD(); currCameraMatrix = new PMatrix3D(g3.camera); camera(); controlP5.draw(); g3.camera = currCameraMatrix; cam.endHUD(); } public void Backup_SOM() { backup = !backup; } public void Reset_SOM() { resetSom = !resetSom; } public void Rotate_Cam() { rotate = !rotate; } public void SOM_Run() { somOn = !somOn; } public void Music_Pause() { music.song.pause(); } public void Music_Play() { music.song.play(); } public void Create_Mesh() { meshCreate = !meshCreate; } public void Prev_Input() { order-=1; num = 0; somOn = false; vecCount = 0; if(order < 0){ order = n-1; } } public void Next_Input() { order+=1; num = 0; somOn = false; vecCount = 0; if(order > (n-1)){ order = 0; } } public void SOM_Pause() { stop = !stop; } public void Smooth_Mesh() { smooth = !smooth; }

Mental Object in Perceptual Space// references

|| -- References -- || Christoph Cox. (2003). How Do You Make Music a Body without Organs? Gilles Deleuze and Experimental Electronica.

Kirsty Beilharz. (2004). SPACE: Designing Sounds and Spaces: Interdisciplinary Rules & Proportions in Generative Stochastic Music and Architecture.

Matteo Melioli. (2005). Inhabiting Soundscape : Architecture of the unseen world.

62

Mental Object in Perceptual Space// references

63