HART, unexplored application for the heartbeat sound

unexplored applications for the heartbeat sound

school | Politecnico di Milano - Scuola Del Design master | Design&Engineering, Progetto e ingegnerizzazione del prodotto industriale supervisor | Barbara Del Curto co-supervisor | Sara Colombo co-rapporteur | Andrea Pavoni Belli candidate | Marco Conte academic year | 2016-2017

unexplored applications for the heartbeat sound

“Everything in the world has a spirit which is released by its sound.“ Oskar Fischinger

table of contents

HART:

unexplored applications for heartbeat sound abstract

11

introduction

13

1.Research

18

1.1 - a focus on musical instruments design 1.2 - what music are we playing?

1.3 - who are musicians today?

1.4 - sound and movement 1.5 - new sound sources

2.Focus design 2.1 - a focus on musical accompaniment 2.2 - sound by our body 2.3 - musical instruments for the body: market situation 2.4 - a general question: can we play music more instinctively?

3.Concept design 3.1 - a problem to solve: heartbeat sound amplification out of medical application

19 24 27 30 35

42 43 45 51 55

60 61

3.2 - the technology behind auscultation 3.3 - the possible heartbeat sound 3.4 - who could be interested in heartbeat sound?

3.5 - possible heartbeat interaction 3.6 - final brief: live heartbeat sound

4.Project development 4.1 - system requirements: heart sound system performances 4.2 - system technology: suitable solution for multiple users 4.3 - final concept: HART, the heartbeat sound controller 4.4 - system interaction: how do we get in touch with heart sound?

4.5 - technical drawings: HART pre-engineered version 4.6 - a working prototype, HART first system setting

conclusion 4.7 - future development: HART engineering and potential

5.References 5.1 - research 5.2 - focus design 5.3 - concept design 5.4 - project development 5.5 - image references 5.6 - special references

64 70 75 81 86

92 93 6-29 104 115 7-12 122 12-15 132 15-17 144 18-22 23-25

159 30-47 161 31-33 166 33-38 39-43 167 43-44 170 172 48-79 176 178 49-51 180 52-58 58-62 63-69 69-74

list of figures figure 0.0 - “Love Seeker” by MILLO

1.Research figure 1.0 - “Love Seeker” by MILLO figure 1.1 - Artiphon figure 1.2 - Dualo figure 1.3 - GePS interaction figure 1.4 - Statista report about revenue in the “Music Events” segment in 2016 figure 1.5 - Statista report about revenue in the “Digital Music” segment in 2016 figure 1.6 - Statista report about recorded music market revenue worldwide

from 2005 to 2015 figure 1.7 - XTH Sense performing figure 1.8 - XTH Sense bio-creativity inputs figure 1.9 - Leon Theremin figure 1.10 - Firewall visual/sound interaction figure 1.11 - Firewall fabric depth figure 1.12 - ROLY seabord figure 1.13 - Rudolf Laban figure 1.14 - Laban Theory Eight Efforts figure 1.15 - NrityaDHol appereance figure 1.16 - NrityaDHol setting figure 1.17 - Stimulus by Bart Hess figure 1.18 - Antonello Fresu looking for the right auscultation point figure 1.19 - “Offrimi il cuore” by Nero Project

18 20 20 21 25 25 26 28 28 30 31 31 32 33 33 33 33 34 35 35

2. Focus design figure 2.0 - “Breath” by MILLO figure 2.1 - percussions figure 2.2 - one-man-band figure 2.3 - “The human body” exhibition figure 2.4 - human body metro map infographics by Sam Loman

42 43 44 45 46

figure 2.5 - human body involved areas in most common musical instrument

playing figure 2.6 - dynamic and static acting for musicians figure 2.7 - main music genres tempo vs BPM activity figure 2.8 - interesting areas to focus on for a new musical instrument design figure 2.9 - real products on the market figure 2.10 - market analysis for personal artists experiments figure 2.11 - market analysis for concepts still under development, proto­types

and very young products figure 2.12 - design orientation figure 2.13 - more instinct in playing by Roli music setup

47 48 49 50 51 52-53 54 56 57

3. Concept design figure 3.0 - “Heart Slingshot” by MILLO figure 3.1 - “Offrimi il cuore” by Nero Project figure 3.2 - René Laennec portrait figure 3.3 - heartbeat sound amplification technologies figure 3.4 - what’s inside Angelsounds fetal Doppler by Jumper figure 3.5 - different points of heartbeat sound auscultation figure 3.6 - Antonello Fresu fixing the device on artist chest figure 3.7 - Paolo Fresu performing with his heartbeat sound figure 3.8 - target analysis figure 3.9 - Pi KeoHavong, dancer performing for “Offrimi il cuore” project figure 3.10 - example of a guitarist effects setup figure 3.11 - Touchè prototype figure 3.12 - lighiting project with Pulse Sensor and Arduino setup figure 3.13 - The Touch figure 3.14 - final brief

60 63 64 65-67 69 71 72 75 78-79 81 82 83 84 85 87

4. Project development figure 4.0 - “Love Runs the World” by MILLO figure 4.1 - wearable technology heart contact related moodboard figure 4.2 - shaping system of a compact device figure 4.3 - shaping system of a multiple components device figure 4.4 - system assembly options figure 4.5 - Andrea Pavoni Belli managing with acoustic departments

equipment at INRIM institute

92 94-95 96 96 97 105

figure 4.6 - analysis of different technological options at INRIM institute figure 4.7 - heartbeat audio EQ in order to reduce background noises figure 4.8 - heartbeat audio EQ in order to reduce background noises (20 band) figure 4.9 - example of a EQ pedal usually developed for bass figure 4.10 - HART lighting interaction system figure 4.11 - HART healthy aesthetics of interaction figure 4.12 - HART usability aesthetics of interaction figure 4.13 - HART system feedbacks and signal figure 4.14 - HART storyboard figure 4.15 - HART main app preview figure 4.16 - HART app functions preview figure 4.17 - HART system elements figure 4.18 - HART dynamic system figure 4.19 - HART orthographic projections figure 4.20 - HART axonometric projections figure 4.21 - HART sections figure 4.22 - HART exploded view figure 4.23 - HART renderings figure 4.24 - Unborn Heart™ fetal doppler device by Odosoft Ltd figure 4.25 - LED sound reactive system with TRINKET PRO figure 4.26 - desired lighiting effect figure 4.27 - NeoPixel circles figure 4.28 - PRO TRINKET 5V pinout figure 4.29 - HART prototype making of

105 106 114 114 124 125 126 127 128 130 131 132 133 134-135 136-137 138-139 140-141 142-143 145 146 146 146 147 152-156

All the unlisted pictures in this chapter, like sketches, schemes or renderings were all been created by the author. If they haven’t been numbered this decision came from the idea that they are enough clear to understand or in order to don’t complicate too much the layout and the general reading of this part of the work, which can be considered an entire image of a design process created by the author.

5. References figure 5.0 - “Kriebelstad” by MILLO

164

abstract How product design can generate concrete answers to the new contemporary trends in music production? Musical instruments are changing and this quite slow process is powered by digital revolution and general demand for ease of interaction and creation by the users; are there new possible sound sources provided by unexplored elements already present in music playing? Movement for example, could be transformed into sound expression, even if its generation would be involuntary.

Innovating in music today means looking for new sound sources.

The core output of the investigation is a question: “can we play music more instinctively?”.

In order to arrive to this question, a focus on rhythmic accompaniment has been highlighted as the most related theme to the concept of what could be more instinctive in music. A second step was analysing what part of the body could be involved in instinctive playing and what the market is already offering by this point of view. Heartbeat is basically unexplored in music production; the heartbeat is the most relevant involuntary sound produced by our body. We manage a lot with it in medical applications, but “standard” users don’t really deal with it, basically because they don’t have the way. Heartbeat could be a rhythmic live sound source for artistic performances; The target is wider; they are not just musician as predicted, they could be basically everyone interested in listening to his own heartbeat for different purpose: sport monitoring, relax-

ing activity and every possible artistic or expressive activity. Moreover, some researches demonstrate that listening to heartbeat could be a valuable emotional contact between people under a conversation and a strong psychological impact for ourselves.

How do we achieve the best heartbeat sound quality? How do we use the technology in the meanwhile we are performing different actions? What are the needs of the users who are interested in listening or amplification of their own heartbeat sound? The best available technology and the interaction with this completely new device are the core of a real product development. Moreover, a working prototype is considered essential for the further development of the project: such a technology for a totally new application needs to be tested and analysed more than others, before going to a proper engineered phase. In conclusion, future development and scenarios are explained, with the strong idea of a further and immediate development of the whole project called HART. keywords new applications product design interaction design wearable technology musical instruments new sound sources live performances heartbeat sound

Come può il product design generare risposte concrete alle tendenze contemporanee in atto nel mondo della produzione musicale?

Gli strumenti musicali stanno cambiando e questo processo abbastanza lento è animato dalla rivoluzione digitale in atto, oltre che da una generale domanda di maggiore semplicità di interazione e creazione da parte degli utenti; Ci sono nuove possibili sorgenti sonore fornite da elementi inesplorati, già presenti nelle performance musicali? Il movimento per esempio, può essere trasformato in espressione sonora, anche qualora la sua generazione fosse involontaria. Innovare nella musica oggi significa guardare a nuove sorgenti sonore; questa è essenzialmente la conclusione di un primo approccio alla questione.

Il risultato più rilevante della ricerca è una domanda: “possiamo suonare musica in modo più istintivo?”. Per giungere a questa domanda, un focus sull’accompagnamento ritmico e stato evidenziato come il tema maggiormente connesso al concetto di cosa possa essere più istintivo in musica. Un secondo passo è stato l’analisi di quale parte del corpo potesse essere coinvolta nel suonare istintivamente e cosa il mercato stia già offrendo in tal senso.

Il battito cardiaco è sostanzialmente inesplorato nella produzione musicale; il battito è il suono involontario più rilevante prodotto dal nostro corpo. Lo incontriamo frequentemente nell’ambito di applicazioni mediche, ma gli utenti “comuni” non hanno davvero a che fare con esso, principalmente perché non ne hanno

il modo. Il cuore può essere una sorgente sonora, una percussione ritmica per live performances; il target e più ampio del previsto, non si tratta solo di musicisti, ma sostanzialmente di chiunque sia interessato all’ascolto del suo battito per differenti scopi: monitoraggio sportivo, attività di rilassamento e qualunque possibile attività artistica ed espressiva. In aggiunta, alcune ricerche dimostrano che l’ascolto del battito può avere valore emozionale fra due persone in conversazione oltre che un forte impatto emotivo in noi stessi.

Come raggiungiamo la migliore qualità sonora del battito possibile? Come impieghiamo quella tecnologia mentre stiamo svolgendo altre attività? Quali sono i bisogni di quegli utenti interessati all’ascolto o all’amplificazione del loro battito cardiaco? La migliore tecnologia impiegabile e l’interazione con questo nuovo strumento sono gli elementi chiave di una reale proposta di prodotto. Un prototipo funzionante e considerato inoltre essenziale per lo sviluppo futuro del progetto: una tecnologia simile, pensata per una applicazione completamente nuova, necessita di essere testata e analizzata più di altre, prima di andare ad una fase di vera e propria ingegnerizzazione.

In conclusione uno scenario e degli sviluppi futuri sono pianificati, con l’idea solida di un ulteriore e immediato sviluppo della totalità del progetto chiamato HART.

introduction keywords nuove applicazioni product design interaction design tecnologia indossabile strumenti musicali nuove sorgenti sonore performance dal vivo suono del battito cardiaco

Concept design process can look really chaotic sometimes. How a research about musical instrument design can end up with a wearable device able to amplificate the real heartbeat sound is a story you should read carefully, accepting the strange journey. This project is the result and the concentration of an indescribable amount of different in­puts, before and during its development. Everything started with the idea of merging the two different backgrounds of a product designer and a musician.

The inspiration and the focus on the idea of unintentional music, started from the experience of playing and from the observation of what’s going on today in the music industry. Focusing on the rhythm, we wanted to go on the roots of music production, because it isn’t wrong to assert that every new trend in this field start explicitly or implicitly from a new rhythmic setup. The research process met other projects and studies interested about the topic; it has been widely influenced and powered by them with free agreement on this risk. A research isn’t a research if you know from the beginning who it is going to meet on its path and what would be the final result, especially if creativity is involved in the workflow. First of all, this work wish to be a link among different processes, the essential piece of a more complex puzzle played by different characters and institutions, the way how some intentions and experiments become realistic solutions. This work is a foundation of a wider view that wants really to become real for the near future. This project has been developed as a master de-

gree thesis for the Design Engineering course in Politecnico di Milano. Professor Sara Colombo supervised the research and the final concept formulation, later professor Barbara Del Curto followed the last design development process, bringing the project until its first official presentation in December 2017.

During the development a huge amount of knowledge about the topic has been provided by the psychologist and artist Antonello Fresu, who can be considered the inspirator and the first recipient of this work; with his 10 years old artistic project called “Offrimi il cuore”, Fresu has gifted what can be considered the engine of this design research. Last but not least, the acoustic department of INRIM, the national metrological institute settled in Torino, powered the development from a technical point of view; the department engineer and researcher, Andrea Pavoni Belli, declared his interest in supporting and following the project for a further development in 2018.

Un processo che porta alla definizione di un concept di progetto può apparire davvero caotico a volte. Come una ricetca riguardo al design degli strumenti musicali possa concludersi con un dispositivo indossabile in grado di amplificare il suono reale del battito cardiaco è una storia che si dovrebbe leggere molto attentamente, accettando la stranezza del percorso. Questo provrtto è il risultato e la concentrazione di un’indescrivibile quantità di input differenti, venuti prima e durante il suo sviluppo. Tutto è cominciato con l’idea di far

convergere due background differenti: quello del product designer e quello del mu­sicista.

L’ispirazione e la focalizzazione sul concetto di musica istintiva ha preso vita dall’esperienza del suonare e dall’osservazione di cosa sta accadendo nel mondo della musica oggi.

Concentrandosi sul ritmo, si è voluto andare alle profonde radici della produzione musicale, poiché non è sbagliato affermare che praticamente ogni nuova tendenza nel settore nasca proprio implicitamente o esplicitamente da un nuovo assetto ritmico. Il processo di ricerca ha incontrato altri pro­getti e realtà interessate all’argomento; è stato da questi ampiamente influenzato, con libera accettazione dei rischi in merito. Una ricerca non è tale se si conosce già cosa si incontrerà lungo il percorso e quale sarà il risultato finale, specialmente se la creatività è inclusa nel flusso di lavoro. Questo lavoro spera innanzi tutto di essere un collegamento fra differenti processi, il pezzo mancante di un puzzle costruito da più personaggi e istituzioni, il modo in cui alcune intenzioni ed esperimenti diventano soluzioni realistiche.

Questo lavoro è essenzialmente il fondamento di una visione più ampia che intende realmente attuarsi nel futuro prossimo. Si tratta di un progetto nato come tesi di laurea magistrale per il corso di Design Engineering presso il Politecnico di Milano. La professoressa Sara Colombo ha supervisionato principalmente la parte iniziale di ricerca e formulazione del concept definitivo; mentre la professoressa Barbara Del Curto, ha successivamente seguito la parte finale di sviluppo progettuale, por-

start tando il progetto alla sua prima presentazione officiale il 21 dicembre 2017.

Durante lo sviluppo del progetto, una grandissima quantità di conoscenza sull’ar­­gomento è stata trasmessa dallo psicologo e artista Antonello Fresu, che può essere considerato l’ispiratore e il primo destinatario di questo lavoro; con il suo progetto “Offrimi il cuore” che va avanti ormai da oltre 10 anni, ha donato quello che può essere considerato il motore di questa ricerca progettuale.

Infine, ma non per minore importanza, il dipartimento di acustica dell’INRIM, istituto metrologico nazionale situato in Torino, ha potenziato lo sviluppo da un punto di vista tecnico; l’ingegnere e ricercatore del dipartimento, Andrea Pavoni Belli, ha dichiarato il suo in­teresse a seguire i futuri sviluppi del progetto nel 2018.

1.Research

1.1 - a focus on musical instruments design

about interaction design applied to those products.

Digital revolution is surely also investing musical instrument market. What is inte­ resting and some how also strange is that in this specific case, digital is not at all replacing analogical in music. Digital is going in parallel, with not often a perfect communication with the “analogical mu­ sic”. It is roughly possible to categorise mu­­ sicians into three different behaviours: who is joining and going through the integration, who will remain loyal to the analogical for ever, who was just born with the digital and his interest about the analogical is just onto a superficial level.

Interaction design is «about shaping digital things for people use — and it is — the practice of designing interactive digital products, environments, systems, and services» (Wi­ kipedia 2016). From those quotes it is possible to notice how interaction design and digital are interconnected. In conclusion, if one of the goal of this project is to innovate in the world of music instruments, digital/analogical fight in music is an issue; discovering how to exploit positively this contrast it will be also an interaction design challenge.

Is the market of musical instruments arrived to its maximum level of de­ velopment? It is surely a very old market, with hundreds years of experience, history and tradition. Sometimes it is quite closed, sometimes really experimental, and generally full of variants and a certain level of confusion.

Does contemporary product design mind so much about this market? Maybe not: there are very interesting isolated case studies, but generally, product design for musical instruments is not really a leading design activity. This probably happens be­ cau­se musical instruments design, is in between, let’s say, of “two strong fires”: from one side we have few old fa­ mo­­ us companies dominating the market and proposing always the same successful products; from the other side, musicians are theirselves creative people, and they often like to invent their own instruments, without sharing their discoveries, because that is also one of the way to be unique. And when we talk about music production, uniqueness is just one of the key factor for success with audience.

Do they miss anything they are not able to find on the current market? The answer to this last question is probably one of the main goal of this research.

“[…] new design discipline, dedicated to creating imaginative and attractive solutions in a virtual world, where one could design behaviours, animations, and sounds as well as shapes”

A musical instrument may be considered one of the most interactive object on the market. It is reasonable to affirm that there aren’t design disciplines connected to music as much as interaction design. But an introduction is necessary before start talking

Moggridge, 2007 Is it possible to shape differently the idea of the digital sound? Can digital technology still improve interaction with 19

the instruments? What musicians think about the user experience they have with their instruments? How they interact with them? Is there a way or a reason to interact differently? Those are some fundamental questions a new musical instrument design research should mind about.

figure 1.1 - Artiphon

«IxD enables the building of a technology product by focusing on its visual interface and interaction, rather than the underlying functionality» (Techopedia 2017) Most of the instruments on the market are basically designed by a strictly functional approach, often with a kind of superficial attention for aesthetic results.

Most of the “good design” feeling about musical instruments is related to what kind of sound they provide, what sensations.

Material choice is also often just related to sound and at a second level, to other reasons.

User interaction is rarely the starting point, probably because a guitar is a guitar and a piano is a piano, and just very few designers or companies think it is still possible to invent new musical instruments with new playing interactions. By the way, it is clear that some user needs are changing; the world of music is changing and within this evolution, musical instruments should logically change too.

app on smartphone and tablet performing like controllers, infinite sounds possibilities. But what about the interface? It just looks like a strange fusion of different instrument in one single object. Are professional players looking for something like this? Artiphon looks almost like the swiss knife of the musical instruments, powered by all the contemporary digital technology we can implement into a new instrument design (fig. 1.1). It is surely a good project to introduce new people into the world of music, with an open userfriendly interface; but is gonna be hard to see something like this inside a recording studio or an important stage. figure 1.2 - Dualo

«Strum a guitar, bow a violin, tap a piano, loop a beat – on a single instrument. An intuitive way to create music and play any sound» (Artiphon 2015). This is the claim for a project called Artiphon, apperead on Kickstarter in 2015. Is this the correct way to approach new musical instrument design?

This object is surely full of interesting technologies, powerful digital integration, 20

Platforms like Kickstarter often offer a huge level of freedom in product design, so they are always some good reference to discover some innovation trends (if we mind about the “two strong fires, this is the 3rd smaller one).

In a project like Dualo (fig. 1.2),it is possible to notice a slightly different approach compared to the Artiphon idea. Dualo offers a quite new interface to interact with, but the target is again really open; the sound is purely digital and it can be what ever.

This is not a problem, but it is something which really orients the project to some specific range of users. For istance, there is a great chance that a project like this would be more attractive for users who have never played music before instead of being a new relevant instrument for every kind of player.

figure 1.3 - GePS interaction

potential to keep growing and evolving in ways we can’t really predict right now.

As we can observe, musical in­strument design can really take many contradictory paths, and it is always hard to judge objectively what’s right or wrong in the purpose, what represents the future, what is just a temporary trend.

It is maybe better to take a step back, far from the new technological releases. According to the Laban Theory, which will be explained better later, let’s try to figure out an example: “A musician is playing his guitar”. Let’s simplify massively the situation assuming that he’s not moving anything else except his fingers. So, if he is right-handed, it is possible to describe two different kind of activities for his hands: the right hand fingers are flicking the strings close to the sound hole. The left hand fingers are pressing the strings on the neck. For flicking in Laban Theory we could say direction is direct, weight is light, speed is quick, flow is free. Meanwhile for pressing: direction is direct, weight is heavy, speed is sustained, flow is bound.

Now is relevant to focus on a question: was the analysis of those fundamental movements the starting point for the guitar shape design? After percussions, a lute is maybe the second eldest musical

GePS is a performance system for musical appli­ cations. Hard- and software-design is open. The pro­ject is about controlling audio with gestures and movement. Containing a few sensors and radio, the glove will transmit the gesture-data to your laptop where the datastream is analyzed and used to create sound and music. The software system uses the full scale of the sensor data, without limiting it to MIDI. The main goal is to transform the complexity of gestures into music without restricting your movement for the sake of simple parametermapping. (www.geps.synack.ch 2017)

A totally different approach and intent in a new instrument design and technology which doesn’t really allows to undestand its full potential; the project software is opensource and this is maybe its strongest aspect to consider for an effective musical instrument design. The description talks about complexity of gestures, con­trol­ ling audio by freedom of movement. GePS was developed in 2015 and its nature of an opensource project would be maybe its 21

trace form is always connected with inner happening such as feelings, reflections, determinations of the will, and other emotional impulses.”

instrument. The concept of the guitar is so old and deeply-rooted in human history that is almost impossible in this context to answer scientifically to this question. But as a product designer, and a guitar player, I feel the right to say that the answer is almost “no”.

Some ergonomical studies are surely man­ datory into the design of a guitar, but ergo­ nomy minds about movement often just until a certain level. First issue the design of a guitar is focused on, is the kind of wood to use, for example. Ergonomy in terms of sizes of the body, lenght of the neck, shape of those elements are other relevant topics. But at the end, the funny thing, is that the movement of those fingers will still be the most important element to create an amazing performance; in fact, if you have talent, shape and kind of wood of the guitar will not influence very much the positive result of the performance.

So the question is why the design of many classical instruments doesn’t focus first its attention on that movement? Why many musiciants, at their starting steps with the learning process, are forced to learn a specific, sometimes not natural, posture? Just to start playing violin requires at least 3 months in order to learn the proper playing position and other instruments are not very different. Of course a professional player feels himself really confortable in the playing posture of its favourite instruments, but reaching that level is quite challenging for most of the people; they stop to learn playing sometimes just because they don’t feel com­ fortable with the posture. The question is what would happen if they will left free to choose their own playing position for ever and the instrument will allow them to do it.

Rudolf Laban, Choreutics, 1966 And basically there is nothing stronger than emotions driving a musician playing his instrument. But if we prevent that musician of his freedom of movement we somehow limit his possibility to express feelings by playing. In this way of thinking it is possible to affirm that all the musical instruments should be somehow re-invented or updated to a better freedom of movement for the player.

“the body can provide a direct route to the emotions [...]. In every physical action, unless it is purely mechanical, there is concealed some inner action, some feeling.”

Constantine Stanislavski, 1961 Onto a more macro level Laban Movement Theory looks at the categories in terms of phrasing and themes of opposites (v.

Laban, 1966): –– Mobility/Stability –– Inner/Outer –– Function/Expression –– Exertion/Recuperation This is even a step further in the consi­ deration of movement in playing. Let’s take again the guitarist example: his left hand fingers show many mobility/stability phrasings during the pressing and the sliding to an other

“A definite movement with a definite

22

in most of the musical instruments, that “something” is often in mute mode.

Let’s go back to the movement of the drummer foot which is probably the easiest example in order to figure out this new concept of the “upbeat mute mode”. As soon as the pedal is hitten a sound get out almost immediately from the bass drum; then the foot come back to his starting point, in order to be again ready for an other hit. During that movement, many musicians would say silence is quite fundamental. Well, not always! And not in every music genre. Let’s imagine to put a strange different percussion on that moving foot, the drummer will feel that also his upbeat time has a value, he could be inspired to focus more also on that and maybe move his foot less mechanically, with more freedom and creativity.

part of the neck; exertion/recuperation are strictly related to the mobility and stability of their movement. Right hand fingers alternate continuosly function and expression; an inner/ outer contrast can be seen in every couple of movement related to the rhythm.

Let’s consider an other similar topic: the foot movement of a drummer which is purely mechanical; basically he just needs to control the force and the rhythm he pushes the pedal with. Then the pedal transmits that sort of punching into the bass drum, which create that strong regular sound which is able to let thousands of people dancing and jumping. But what about the movement of the foot? First, none will see it; it is hidden by everything on the stage, especially the bass drum itself. And what if that foot actually would do something more than just hitting a pedal? what about the heel movement? what if the sole would just shift a bit on the pedal surface? what if the different levels of pressure on the drummer sit would be able to generate variations in sound?

By this kind of questions it is probably already possible to design a new drum set, just trying to exploit better all the “mute movements” of a drummer.

In music, the downbeat is the first beat of the bar. Meanwhile the upbeat is the last beat in the previous bar which immediately precedes, and hence anticipates, the downbeat. Both terms correspond to the direction taken by the hand of a conductor (Wikipedia). If we go out from the strictly rhythmic meaning of those concepts, the upbeat roughly represents the movement when the hand (or feet) of the player are not playing anything. But upbeat is related to a movement; what if that movement has actually something really interesting to say and to add on the sound creation flow? For the moment,

“The sharing of musical expression is likely to be based on empathic corporeal engagement with motion which is contained in sonic patterns as well as with synchronization to the movements of the other participants.

Likewise, disambiguation of musical structures through body movement is fully embodied activity and has an impact on how rhythmical structures are perceived.”

Mark Leman, 2007 An Embodied Approach to Music Semantics Have you ever seen a professional singer not moving on the stage? They just can’t stay stationary! They move continuously, in many 23

1.2 - what music are we playing?

crazy personal ways. What if we can give a sound to all the specific, unique movements of a singer? What if that sound is going to be designed to be an accompaniment of that singing?

Maybe just studying the “mute movement” of different musicians could be the key tool to drive relevant innovations in musical instruments design.

In the last 10 years the world of music has faced a quite big revolution: the recorded music started to be more accessible. Let’s just think about services like Spotify, Google Play, Apple music, SoundCloud, Youtube.

The intensive smaterialization and digi­ta­ lisation of music is it almost close to arrive to its end. Just 50 years ago we were wait­ ing for months in order to buy an expensive one song album contained into a 45s, and not everyone was having his own record player at home. Nowadays we don’t even know all the ways we have to listen music by smartphone, tablet, computer, tv, radio, mp3 player and who knows how many different devices already on the market. In addition, many ways we can listen music are often free, or very cheap. The result is that music lovers are crazy, random listeners, and because of that, they want more, and they want it unique, unrepeatable.

From the other side, albums sales are not anymore the way for musicians to earn money and survive. It is just an help, the way how they promote their art, let’s say their “business card”; recording became an act of saving a piece or a state of art in memory, the way to tell a story and share it everywhere.

“What a musical instrument allows to do in terms of sound production goes far beyond what our common sense would expect from any other ordinary object. It establishes an almost magical relation between gesture and sound and, in this way, the musical instrument is not only the medium for the musical idea, but also part of this idea.”

Fernando Lazzetta, Meaning in Musical Gesture, 2000 In conclusion, focus on interaction, deeper observation of movements in play­ ing, study on players instincts, higher knowledge in where, how and why the successfull instruments where invented, there are many ways, to innovate in instruments design, this investigation has the aim to lead to the most relevant one.

Many audiences are no longer satisfied with a passive role when it comes to music. They demand interactive concerts. They’re excited to post comments or photos on their favorite group’s blog. They want to vote for the next members of the YouTube Symphony Orchestra. They love to remash tracks from interesting groups. Heck, they’re ready to make the music themselves (even if it’s through a video game like Guitar Hero) (David Cutler 2011)

24

figure 1.4 - Statista report about revenue in the “Music Events” segment in 2016

figure 1.5 - Statista report about revenue in the “Digital Music” segment in 2016

In such a scenario live music performances are the real new medium between players and their fans. They are the way how they meet in reality, exchanging emotions, sharing a real experience, which has to be incredibile and not repeatable. If recordings are “business cards”, live tours are the real working life of many professional artists.

For this reason live music performances are growing in number, year by year. Music bands, especially the famous and very talented one, are getting used to earn much more money with their live performances than with all the rest. By this tendency, artists are asked to perform more and more; on a stage or in front of a camera, they need to create an experience, and they need to have the right tools to do it, plus, of course, their talent.

Playing live, playing everywhere are the needs pushing the design of new portable instruments, often powered by digital tech­ nology. It looks like the sound quality is not really the most important thing, because the winner is who is faster to play naturally, everywhere, everytime possible.

«Your body becomes a backing band...you tap out whatever music you fancy, and there’s no need to carry around heavy equipment.» (Fast Company 2014) This is a quote related to a product called Drumpants. A sort of new musical instruments transforming your body into a drum machine. Nowadays the music we play is also changing its traditional medium. What is easier to play? From one side we have a huge amount of massive 25

figure 1.6 - Statista report about recorded music market revenue worldwide from 2005 to 2015

language his using; brain, body and social activities are on the base of a good pitch like movement, stage presence, lyrics contents and dynamics are on the base of a good performance.

It is such a hard task to define where the music playing will go in the early future, but what we could be sure of, it’s that we can observe from many different points of view, a general movement pushing the whole industry for a higher level of freedom, sharing and ease of use.

elements, impossible to carry around without a car and with a higher possibility of damage or maintenance need and from the other we have wearable sensor fitting almost in your pocket, connected to your portable device and ready to play wherever you want. Sound quality and impact is probaly incomparable, user experience too. But those are two faces of the same reality; the music we are playing is exploiting is possibilities thanks surely to digital technology but also by the new kind of players are borning. Have you ever heard a beat box player? Just by his voice he is able to replace a drumset in some music genres.

In a world where the act of playing is a show by itself, sometimes thinking later about the sound, the more natural would be the playing action, the more effective will be the full message the artist is trying to send to his public. Musicians can’t play anymore just thinking about the perfection of their technique. It would be like if a good speaker just mind about his perfect knowledge of the

In addition we can’t really skip talking a little bit more about the exhausting fight between analogic and digital music. What was looking solved few years ago with the success of mp3 format and portable players like the iPod, is nowadays again under discussion with the return of the vinyl records. But if the idea of an again successful 30cm diameter disk is ridicolous compared to the enormous diffusion of a service like Spotify, the fight is 26

1.3 - who are musicians today?

really relevant on the side of the players with the musical instruments, recording and live performances. What many musician are used to have in their mind is that they feel a soul in the analogic they don’t feel into the digital; and this is not just by them, but of course also by their audience.

«The most ironic aspect of the debate about digital vs. analog recording is that nowadays a lot of music is a combination of the two.

For example, you might record a song onto analog tape, but mix and master it digitally, or release it on the Internet as an MP3» (Recording Connection 2017). The debate is not just related to the quality of sound; it is also a matter of budget and speed. The issue is always if it’s worth the trouble you meet in analogic for the kind of goal you are trying to achieve; the answer is not always black or white, and it can change player by player, listener by listener. What we could probably say is that the music we are playing is neither analogic, neither digital; it sounds how everyone is able and like to play and listen to it. How subjective and contradictory are discussions related to a language accent better than an other one in the same language family? we should probably just go to consider analogic and digital as different dialects of the same language. Hybrids talking both of them are already on the market, the right balance is always in the right user targeting.

Music can be made and played by virtually anybody, but only up to a certain level. Not anybody [...] can compose an emotionally rive­ting song like “Imagine”

by John Lennon; or a triumphant masterpiece like Beethoven’s 5th Symphony. To master an instrument, a person would have to dedicate a big portion of their life to the instrument to play music. Hence, the profession: Musician. (David Cutler 2010)

There is no doubt regarding to the easier possibility about playing music we all have now­ adays. Cheaper musical instruments, be­gin­ners version of them, youtube tutorials about everything, music apps, new digital in­struments easier to play. If you don’t have talent, you will not be famous, but you will play whenever, whatever you want, maybe just to feel good, just to express your feelings; music is not just for professionals, music should be for everybody. Just thousands of projects selling this idea and the result is that the position of the musician today looks always more blurry.

But this is happening everywhere, especially in creative professions, let’s think about photographers, designers, architects. More music is being produced, and surely more music will not even be known by someone, but this doesn’t really matter, because new consequences are coming from this process: when something becomes more accessible we start looking on it on a different perspective.

“We live in a modern world, and in contemporary music the central fact is contamination. [...]”

A study of the evolution of western popular music provides some interesting insights.

The statements are related to 3 different arguments: the pitch included details about harmony, melody, chords, and progressions

Ennio Morricone, 2007

27

(essentially how the notes were arranged and unfolded over the course of the song); the timbre related to the colour, texture, and quality of sounds used in the song, for instance the choice of instruments as well as recording techniques which affect this aspect of sound; and loudness, of course, is about the inherent volume of the music itself, before any adjustments by the listener.

to notice that when everything is louder, the dynamic range becomes much more restricted.

As in, the contrast between the really soft stuff, and the really loud stuff shrinks, so the overall emotional impact of the music is reduced. (The bulletproof musician 2017)

Lack of personality, this was a concept inside an interesting article titled “Are today’s artists more uniform and less musically adventurous than those of yesteryear?” by Noa Kageyama, a psyco­ logist studying and helping musicians being more confident with a stage. And if that lack is a problem, what ever should be designed for contemporary musicians should probably be “personality based”, or at least very strictly connected to the possibility to express freedom in playing, with the concept of a musician which is able to play himself instead of an instrument.

This is what maybe the XTH SenseTM team was thinking when it launched on kickstarter the “first biocreative instrument” (fig. 1.8). When you wear the XTH Sense an array of biosensors captures 7 types of signals from your body: 1) The sound of your muscles contracting when you move 2) The sound of the heart beating 3) The sound of the blood flowing 4) Temperature 5) Motion 6) Orientation 7) Rotation (XTH Sense 2017). «Our intelligent algorithms crunch the data from your body and extract the distinctive characteristics of your movement and inner bodily processes, like movement dynamics, muscular energy and temperature changes.

These are the expressive features of your body» (XTH Sense Kickstarter campain 2016).

In terms of pitch, the data suggested that the variety of pitch progressions used has shrunk over the years. In other words, musicians are becoming less inventive and adventurous in how they get from one note or chord to the next, and instead seem to be relying more and more on the same sequences and patterns that others have used successfully in the past.

A similar homogenization seemed to occur with timbre. Whether it’s due to an increasing reliance on the same instrumentation, or the utilization of the same limited toolbox of recording techniques, the palette of sound colors/texture/tone present in recordings has diminished as well.

Meanwhile, everything is getting louder. Which might not seem like a big deal, until you start figure 1.7 - XTH Sense performing

figure 1.8 - XTH Sense bio-creativity inputs

“The XTH Sense offers a unique op­ portunity for composers to chan­ ge the way we compose music, 28

playing with elements of chance and control, utilizing the infinite sound possibilities of the computer, and also tapping into our bodies as instruments themselves. I’m excited to incorporate the XTH Sense into my work to facilitate a more dynamic, interactive and spontaneous inte­ raction with audiences in live perfor­ mance.“

general diffused language instead of an art for a select few; this process just started and it is probably to early to figure out where it will go just in the next 5 years.

“Music is the universal language of mankind.”

Henry Wadsworth Longfellow (1807-1882)

Shara Nova, 2016

STEIM (the STudio for Electro-Instrumental Music) «seeks to support artists who are both the players and makers of their own expressive tools. STEIM stands behind a human-focused approach to technology. This technology has to be tailored to the individual.» (Steim website 2017) We are all different in our appereance; every human being is almost unique for the uniqueness and the variability of its DNA code. Can we imagine a music future where musician are all strongly different because they focus their music on their genetic uniqueness? Well, this already happens, because you can give the same song with the same guitar to a hundred of guitarist and none will play that song in the same way.

But what an activity like the STEIM one is talking about it is the possibility to take that difference and pushing it higher and higher, being powered by the instruments itself.

We are used to think about musical instruments as objects to learn; what if the instruments start becoming objects able to learn about us? We would probably all become new kind of music players, and music would probably start to be more a 29

old instrument received recently some more attention, but its impact in the world of music is still not very impressive, probably for the peculiarity of its sound. Have you ever seen a so-called thereminist playing on a stage? It is quite spectacular how the freedom of his hand rules the sound without touching anything, flowing in the air, modulating the sound with every little movement. The aesthetical impact is similar to the one of an orchestra leader, but the great difference, is that the movement literally creates and rules the sound. The theremin is also considered one of the first electronic musical instruments. And with this we arrive to a quite relevant consideration: if we look on a different and more free interaction with musical instruments, digital technology is often unavoidable.

This is also quite a “barrier to break”, because there are still many professional musicians avoiding or taking in very little consideration the importance of electronics in music.

From the start of the past century we jump to the closer 2012 with a project called Firewall.

The Michael Allison work is described by Aaron Sherwood by the following words: «is an interactive media installation. [...] A stretched sheet of spandex acts as a membrane interface sensitive to depth that people can push into and create fire-like visuals as well as expressively play music» (Sherwood 2012) and it is maybe one of the most interesting contemporary study about how we could interact differently with music.

Its intent was maybe largely visual and for artistic purpose, but the result — we can also see in the blog where it was published — was that many people are still interested to have a consumer version of the artistic installation.

We can take this as a short demonstration that consumers are looking exactly for those kind

figure 1.9 - Leon Theremin

1.4 - sound and movement Listening music always create a certain movement in human being; creating mu­ sic is basically about movement. Imagine when your foot just start without any control to hit the floor following the music rhythm, or when our hands start performing weird movements in the air like we are directing the music we are listening to. It is clear that almost everyone is able, already as a baby, to feel the music and represent it through movement. The interesting thing about this very specific human attitude is that everyone represents music with movement in his own specific way.

Who knows if Leon Theremin (fig. 1.9) was also minding about this when he invented the namesake instrument in the 1919. The the­re­ min was «the first analog synthesizer which also was able to continuously change the pitch. The right hand affects the pitch and left the volume.» (Wikipedia 2017) This quite 30

of inventions powered by our new technologies.

Moreover the Sherwood’s description explains in details the technology inside:

figure 1.10 - Firewall visual/sound interaction

The piece was made using Processing, Max/MSP, Arduino and a Kinect. The Kinect measures the average depth of the spandex from the frame it is mounted on. If the spandex is not being pressed into nothing happens. When someone presses into it the visuals react around where the person presses, and the music is triggered. An algorithm created with Max allows the music to speed up and slow down and get louder and softer, based on the depth. This provides a very expressive musical playing experience, even for people who have never played music before. A switch is built into the frame which toggles between two modes. (Sherwood 2012)

Firewall could be a new musical instruments which everyone can play without knowing any­thing about music. Users want to play naturally, without any limit and exclusive attitude. The challenge sometimes is to let the professional musicians be also interested about such an instrument.

From a project like that until today, many similar projects were developed often with similar technological approaches. Something to care about is the world of gaming: the Kinect sensor comes from Xbox console and it was invented to interact more naturally with video games, forgetting the traditional joystick interaction. But when we look on the amount of different application the Kinect is being used for, we should start thinking if video games have the power to drive a revolution of an entire sector like music production, where tracking the body movement and directly transforming it in sound is the last incredible Kinect feature. Are these applications changing the world of music production? Not really. Kinect, for example, is quite unprecise for professional use and is

figure 1.11 - Firewall fabric depth

also strongly connected to your position in the space, which sometimes can be annoying, especially in a music live performance. Ki­ nect music applications are great for temp­ orary interactive installation, music ga­mes, application not directly connected to pro­ fessional music production. ROLI Seaboard makes music in more di­ mensions. It’s a new way of thinking about keyboards. «The piano key­board has existed for several cen­ turies without any massive changes, but the designers at ROLI have just pushed it forward”» (The Verge 2015). This is really a new musical instruments, or better to say a re-design of it.

31

last decade and it is still the way how we can radically change our interactions with many objects.

figure 1.12 - ROLY seabord

We previously mentioned to Rudolf Laban; he was a movement theorist, a choreographer and a dancer. Laban (1966) is considered a pioneer of modern dance and he doesn’t actually have so much to share with musical instruments design, but his body movement studies can’t really miss from a movement and sound essay. The so-called LMA (Laban Movement Analysis) can be one of the most useful tool to figure out how a movement could be visualized and interpreted Laban theory categorized human movement into four component parts: Direction, Weight, Speed and Flow. Each of those parts has two typologies: –– Direction: direct or indirect –– Weight: heavy or light –– Speed: quick or sustained –– Flow: bound or free

On the soft silicone surface of the Seaboard RISE, you shape sound intuitively through touch. The RISE offers Five Dimensions of Touch - Strike, Press, Glide, Slide, and Lift - that open up an entirely new approach to expressing yourself on a keyboard. (ROLI website 2017)

“This can lead to the development of more intuitive ways for humans to interact with digital technology” The Wall Street Journal, 2016

Laban combined then these parts together to create the so-called “Eight Efforts”: Wring, Press, Flick, Dab, Glide, Float, Punch, Slash (fig. 1.14). For every effort, Laban identified then which component parts acts.

For example: For WRING: the Direction is Indirect, the Weight is Heavy, the Speed is Sustained and the Flow is Bound (Laban 1966).

So this is basically a scientific way to describe all the kind of complex movement, a body part could perform. If we think about a sensor which should be programmed to react differently to each of those movements, it suddenly becomes very useful to have a method to describe and somehow predict them.

Was the inspiration for a new product like ROLI Seaboard driven by a deep observation of the hands movement during a piano playing?

Yes or not, the naturalness of the gestures you need to play it are explicit, we could say an “hard” interface was transformed into a “soft” one, something comparable to the switch from buttons interface to touch display in smartphone design. Such a great innovation was probably the reason why ROLI received so much success and awards when it appeared on the market in 2013.

The product is about natural movement observation, and this is probably how the core of its innovation was driven. Movement observation can be considered the engine for many technological revolution in the 32

called “figure”. Dancing is figurative, and it often aims to translate sound in figures. But can figures become sound? If we mind about tip tap dancing this is not really a crazy idea: an entire musical/dancing performance just realized by a specific feet movement with specific soles made to create that peculiar sound you can hear after every step.

Something more contemporary? The concept of NrityaDHol.

figure 1.13 - Rudolf Laban

[It] highlights underlying op­por­tunity of designing percussion instruments producing music from Indian classical dances. It incorporates very basic sound to movement mapping. [...] NrityaDhol, by combining mo­ ve­ ment vocabulary of Ghunghroo and sound vocabulary of dhol, translate rhythmic foot tapping of dance into Dhol music. (NrityaDHol 2017)

The pro­totype was a bit invasive wearable

figure 1.14 - Laban Theory Eight Efforts

figure 1.15 - NrityaDHol appereance

figure 1.16- NrityaDHol setting

Dancing may be considered the most direct translation of music in movement.

The whole Laban theory was actually ini­ tially developed to help in dance learning.

Aesthetics in dancing is quite an important aspect, this is maybe why a position is often 33

device for the user, and strictly related to the specific indian dance, but the idea and the research around it, was great. NrityaDHol mix dancers and musicians in a positive way, it is a good step into the idea of an instrument able to transform movement in sound.

Those considerations are surely not rigid, but definetely recurring. Every musician moves, the creation of sound start always quite naturally from a movement, but some performers are more dynamic than others and this could be a further level of investigation.

There is no sound without movement, one of the big question is if movement could still find new ways to generate sound and why this could be interesting.

When an artist like Bart Hess shape so­ me weird fabrics and membrane able to or­ ganically move like human body parts after a sound input (fig 1.17), we also discover that sound can shape movement in many different ways. The 2016 artistic installation was called Stimulus, and it was shown in “Fringes of Beauty’’ exhibition at Textiel Museum in the Netherlands.

Dancing again, is the most simple example of the potential of sound in movement generation.

The two elements interact togheter in so many different and complicated ways that is almost impossible to mind about all of them. The human being is fascinated by this connection and it will probably never stop to imagine new interactions.

Nowadays we have many technologies to achieve personalyzed interactions between movement and sound. The re­ search doesn’t stop, but it continuosly meets barriers. There are still notable differences between an artistic installation and a new musical instrument, between music in gaming and professional music production.

Innovations about the topic are not always delivered to the right goals; an important amount of potential revolutions may be hidden in private musicians experiments. An effective innovation connects the right technology to the right need; is somewhere out there

Are musicians dancers? Maybe dance goes a little bit too far from the topic, but a classification of musicians based on the different movements they are able to perform, could be an additional step useful to figure better out the connection between sound and movement in music. Just by observation it is surely understandable the idea of separating musicians into “static” and “dynamic” one. In addition a musician can be static or dynamic for his own habit or for some constraints related to the kind of instrument he is used to play. Some examples? If you play an harp, you are definitely forced to be somehow static in your position, so you could be classified as a static musician. If your instrument is your voice, you’re almost free to move everywhere and in every way, during your performance; moving is also often the way to capture the audience attention, so a singer should be usually a dynamic musician. Playing a pia­no is often more static then playing a keyboard; classic guitar requires usually a more specific posture, meanwhile electric guitar, usually inspires a lot of dynamism. figure 1.17 - “Stimulus” by Bart Hess

34

When Antonello Fresu in Offrimi il cuore 2012 project ask to some musicians, visual artists and dancers to improvise using the­ irselves heartbeat as rhythmic base during a live performance, something like a new idea of creativity is being experienced. We are at Maxxi in Roma inside Nero Project in collaboration with cultural association Ti­me in Jazz: the heart becomes the unusual rhythmic accompaniment to play togheter with artists’ personal instrument or personal expression.

Improvisation and listening to ourself are the key points of the performance until reaching the armony with the heart beat, live picked up from an ultrasound echocardiograph. The exhibitions are not just a simple execution of auditory, corporeal or visual score but real dialectic conversations of the intimacy of every artist.

Heartbeat is just the most explicit unintentional sound produced by our body. Its rhythmic activity, very close to the rhythm every playing usually follows is the most interesting aspect. The heart rate is comparable to 4/4 time in music. If your heart beat doesn’t follow that metric, in medicine you have a desease called arrhythmia; a high percentage of music on the market (especially pop) follows a time like that, meanwhile for the beginners, playing in 4/4 time is just the most natural thing they can do. Is it a pure coincidence? Maybe not.

figure 1.18 - Antonello Fresu looking for the right auscultation point

figure 1.19 - “Offrimi il cuore” by Nero Project

a relevant new method to connect mo­ vement with sound?

1.5 - new sound sources An involuntary movement occurs when you move your body in an uncontrollable and unintended way. These movements can be anything from quick, jerking tics to longer tremors and seizures. You can experience these movements in almost any part of the body, including the neck, face, and limbs. (O’Connell 2016)

“Hearing a heartbeat has the same effect as looking each other in the eye“

If our body can involuntarily move and movement is always the unavoidable starting action for sound production, the idea of unintentional sound is immediately a possible scenario.

Joris Janssen, 2010 This is the remarkable conclusion of research at Eindhoven University of Technology (TU/e) 35

in partnership with Stanford University and Philips Research. The research focuses on improving human contact when using digital communication channels.

Let’s consider also our brain:

cians to use sensor-based computer instruments to perform music. Musicians can now use data from different sensors (that reflect cardiac sound.

Simultaneously, Brain-Computer Interface research has shown that cerebral patterns can be used as a source of control. Indeed, cerebral and conventional sensors can be used together with the aim of producing a ’body-music’ controlled according to the musician’s imagination and perception.(Burak et others 2005)

The researchers attached electrodes to the scalp of such subjects to determine if the brain’s electrical activity was somehow affected by the rhythm of the sound. The EEG recording detects the combined electrical activity of thousands of neurons working together in the cerebral cortex. [...] The EEG recordings showed that the waves of brain activity (alpha and beta waves) became synchronized around the auditory rhythm. That is, the ongoing oscillations of brain waves became phase shifted so that the peak of the wave always occurred at a precise point relative to the next beat in the drum rhythm. Rhythmic sound synchronizes brain waves.

(R. Douglas Fields 2012)

Nowadays biological sound is a reality in a quite considerable amount of scientific studies. Those researches didn’t turn in mass production items yet and there are not very impressive experimentation with famous artists all over the world. The reason could be related to some issues mostly connected to the technology we now have to perform such a new idea of sound production.

Tracking the body in terms of muscles contractions, brain activity or heartbeat is not a difficult task with the medical devices we nowadays have, what is hard is to translate the body inputs into sound outputs.

The challenge is related to the IT process to develope, but also to the specific sound to link on the body input. Meanwhile most of the existing musical instruments produce the sound phisically, mostly by the vibration of a certain object, in the case of the biological sound, the situation is unexpectedly less natural; with the exception of the well-known heartbeat sound, our body doesn’t produce sound directly. So we have to think about our body as a controller able to modify, often with a quite complex and uncontrolled process, a pre-selected sound on a certain range; we need an interface, we need a software where a specific sound created before is controlled by sensorial inputs. Unless we start to make easier, more understandable and direct

What about the other sounds our body is able to produce involutarily? What could happen if the metric of a live playing is ruled by something like the heart beat, which traditionally changes by emotions?

Maybe someone could say this is enough to plan a totally new musical genre.

“The last time you whistled a tune or hummed a song why did you choose that one? You may not consider yourself a musical person, but your little act of unintended music may be the key to unlocking within you a wealth of unsuspected creativity, a kind of creativity that goes way beyond music, too.”

Lane Arye, 2001

Advances in computer science and specifi­cally in Human-Computer Inter­action have enabled musi-

36

the trans­formation of body inputs into sound out­puts, we will still be far to think about our body as a musical instrument.

Musicians know their hands and their feet are very good “tools” to produce sound with an instrument, but they don’t consider other parts of their body similar tools or moreover the source inputs of a sound wave with a specific identity such as the vibration of a guitar string or a leather under tension.

“Music is not limited to the world of sound. There exists a music of the visual world“ Oskar Fischinger, 1951

37

sources

Dualo, the new musical instrument for all [online, website] https://www.kickstarter.com/proje­c ts­ ­­/63006320/dualo-the-new-musical-instrument-for-all 5 futuristic instruments changing how we make music [online, article] https://www.fastcodesign.com/3041135/5futuristic-instruments-changing-­how-wemake-music

1.Reasearch 1.1 - a focus on musical instruments design

Interaction design brief [online, article] https://www.interaction-design.org/literature/ book/the-encyclopedia-of-human-computer-interaction-2nd-ed/interaction-design-brief-intro

Ben Hayes When analogue, digital and acoustic sound meet [online, article] https://roli.com/stories/when-analogue-meets-digital

Interaction design brief [online, article] https://www.techopedia.com/def inition/27248/interaction-design-ixd

Sachs, Curt The history of musical instruments Courier Corporation, 2012 The 10 most curious and successful kickstarter instruments [online, article] https://www.gadgetdaily.xyz/the-10-most-curious-and-successful-kickstarter-instruments/

Rudolf Laban Choreutics Ed. Lisa Ullmann. Macdonald & Evans, 1966 Fernando Lazzetta Meaning in Musical Gesture. Trends in gestural control of music 2000

Introducing the artiphon instrument [online, website] https://www.kickstarter.com/projects/artiphon/introducing-the-artiphon-instrument-1

Mark Leman An Embodied Approach to Music Semantics Musicae Scientiae, 2010

Stanislavsky, Constantin.

Creating a Role Trans. Elizabeth Reynolds Hapgood. New York: Theatre Arts Books, 1961 38

Godøy, Rolf Inge, and Marc Leman Musical gestures: Sound, movement, and meaning Routledge, 2010

2015 music industry trends [online, article] http://thenexttech.startupitalia.eu/55­3 120160113-kurv-musica-chitarra-virtuale

C. Cadoz and M. M. Wanderley.

Trends in Gestural Control of Music Centre Pompidou Paris, IRCAM - Centre Georges Pompidou, 2000

Mogees [online, website] http://www.mogees.co.uk/ Kickstarter [online, website] http://www.kickstarter.com/

1.2 - what music are we playing?

Drumpants [online, website] http://www.drumpants.com/

Kusek, Dave, and Gerd Leonhard The future of music: Manifesto for the digital music revolution 2005

Analog digital what’s the difference [online, article] https://www.recordingconnection.com/reference-library/recording-education/analog-digital-whats-the-difference/

9 ways musicians actually make money today [online, article] http://www.rollingstone.com/music/ lists/9-ways-musicians-actually-make-money-today-20120828

Why this is a great time to be in music [online, article] http://www.savvymusician.com/blog/2011/01/ why-this-is-a-great-time-to-be-in-music/

The new economics of the music industry [online, article] http://www.rollingstone.com/music/news/ the-new-economics-of-the-music-industry-20111025

Andrea Doria La musica il suono e le frequenze: informazione coerente a 432 hertz [online, article] http://www.scienzaeconoscenza.it/blog/consapevolezza-spiritualita/musica-suono-frequenze-432-hertz

2015 music industry trends [online, article] http://www.lawyersrock.com/2015-music-industry-trends/ Statista [online, website] https://www.statista.com 39

1.3 - who are musicians today?

Nrityadhol [online, website] https://nrityadhol.wordpress.com

The future and role of music in modern society [online, article] http://www.savvymusician.com/blog/2010/04/ the-future-and-role-of-music-in-modern-society/

Aaron-sherwood [online, blog] http://aaron-sherwood.com/blog

Are todays artist more uniform and less musically adventurous yhan those of yester year? [online, article] http://www.bulletproofmusician.com/are-todays-artists-more-uniform-and-less-musically-adventurous-than-those-of-yesteryear/

What causes involuntary movements? 14 possible conditions [online, article] http://www.healthline.com/symptom/involuntary-movements

1.5 - new sound sources

Offrimi il cuore | Roma Jazz Festival 2012 [online, video] https://www.youtube.com/watch?v=CKa8TlaltD8&feature=youtu.be

Kickstarter [online, website] http://www.kickstarter.com/

Polotti, Rocchesso Sound to Sense, Sense to Sound, A State of the Art in Sound and Music Com­puting Pietro Polotti and Davide Rocchesso, editors IUAV, Dipartimento delle arti e del disegno industriale, 2008

Steim [online, website] http://steim.org/what-is-steim/ XTH [online, website] http://www.xth.io/

Trevor Pinch, Karin Bijsterveld The Oxford Handbook of Sound Studies Oxford University Press, 2011

1.4 - sound and movement

Lane Arye Unintentional Music: Releasing Your Deepest Creativity Hampton Roads Publishing, 2002

Theremin [online, website] https://nl.wikipedia.org/wiki/Theremin Seaboard-rise [online, website] https://roli.com/products/seaboard-rise 40

Arslan Burak, Andrew Brouse, Julien Castet, Remy Léhembre, Cédric Simon, Jehan-Julien Filatriau, Quentin Noirhomme [online, article] A real time music synthesis environment driven with biological signals ICASSP06 - IEEE International Conference on Acoustics, Speech, and Signal Processing, May 2006, Toulouse, France, 2006 A. Tanaka

Eindhoven University of Technology Personal touch: Hearing a heartbeat has the same effect as looking each other in the eye ScienceDaily [online, article] www.sciencedaily.com/releases/2011­­/04/1104 07093118.htm (accessed July 20, 2017).

Joris H. Janssen, Jeremy N. Bailenson, Wijnand A. IJsselsteijn, Joyce H.D.M. Westerink Intimate Heartbeats: Opportunities for Affective Communication Technology IEEE Transactions on Affective Computing 2010

Musical perfomance practice on sensor-based instruments in Trends in Gestural Control of Music, M. M. Wanderley and M. Battier, Eds., pp. 389–406.

IRCAM, 2000

Eduardo R. Miranda On Biophysical Music Guide to Unconventional Computing for Music.

Berlin: Springer, 2017.

Y. Nagashima Bio-sensing systems and bio-feedback sys- tems for interactive media arts in 2003 Conference on New Interfaces for Musical Expression (NIME03), Montreal, Canada, 2003, pp. 48–53

Computer Music Journal Biophysical Music: Sound and Video Anthology MIT Press, 2015 [online, article] http://www.mitpressjournals.org/doi/ pdf/10.1162/COMJ_a_00333

R. Douglas Fields The Power of Music: Mind Control by Rhythmic Sound [online, article] https://blogs.scientificamerican.com/guestblog/the-power-of-music-mind-control-byrhythmic-sound/ Mogees [online, website] https://www.mogees.co.uk/

41

2.Focus Design

2.1 - a focus on musical accompaniment

“The first thing that struck me about contem­ porary music in general had been that there was not much interest in rhythm.”

Guitars, pianos, winds, and let’s include also the voice, even if it wasn’t designed by man, but it is probably the best designed of all those melodic instruments able to allow musicians to express directly themselves.

They support armony, poly­ pho­ nic sounds, dy­na­mics. But then, this direct expression which could sometimes work alone, it often needs and it is usually paired with a so-called accompaniment. Accompaniment is often less considered, less impressive in the mind of listeners, but in reality is not at all the less important part of music. «Accompaniment is the musical part which provide the rhythmic and/or harmonic support for the melody or the main theme of a song. There are many different styles and types of accompaniment in different genres and styles of music» (Wikipedia 2017).

Elliott Carter, 2002

figure 2.1 - percussions

We could roughly divide musicians in two ca­tegories: “melodics” and “accompanist”; but what often happen is that few musicians really like to be just supporting musicians.

Accompaniment is often considered and per­ceived like the b-side of music playing, especially at medium or not professional levels. Into this horizon it is useful for the development of this research to figure out some recurrent music players scenarios in order to highlight where are the most relevant issues.

–– 1st scenario: a songwriter needs supporting musicians to play live in order to have much impact and presence on the stage. His options are four: a - he is very lucky to have some people not caring at all about playing in a project where they will be always behind the scenes. b - he needs to pay quite a lot of money for some shift players, everytime he has a live concert, with the problem of having people not

Accompaniment is also rhythm, and without rhythm there is often no music. Drums and percussions are on the root of music production in the history of Men. But basically every musical instruments can more or less, accompany harmonically or rhythmically or both of them in the same moment. What is not that obvious to think about is that accompaniment is also in hand shaking, in foot tapping, in jumping, in dancing; and this is the easiest, funniest aspect of it: everybody at least once in life has surely accompanied a song.

43

knowing or not understanding his music. c - he register the other instruments by himself or with the support of other musicians and then he play that in stereo during his exhibition. The performance impact lose a lot of power because the public basically don’t like to hear something and don’t see who is playing it. Moreover is really hard to play live with a fixed, not changable accompaniment. d - with loop stations or strange body percussions he tries to fulfill the need by himself, being a so-called one-man-band, with the issue of learning or being able to manage with so many things in the same moment during a live performance.

not caring at all about playing in a project where they will be always behind the scenes. b - they needs to pay quite a lot of money for some shift players, everytime they want to have that kind of live set, with the problem of having people not knowing or not understanding their music. c - they register the other instruments by themselves or with the support of other musicians and then they play that in stereo during the exhibition. The performance im­ pact lose a lot of power because the public basically don’t like to hear something and don’t see who is playing it. Moreover is really hard to play live with a fixed, not changable accompaniment. d - with loop stations, using different instru­ ment in the same moment or strange body percussions they try to fulfill the need by themselves with the issue of learning and bein able to manage with so many things in the same moment during a live performance.

figure 2.2 - one-man-band

–– 3rd scenario: more than two members band are close to a live performance date and one of the member, suddenly can’t be present to the event (one of the worst case is when that member is the drum player).

Their options are four: a - they pay quite a lot of money or they just risk with the first person they have in their mind for a last-minute shift player, with the problem of having someone not knowing or not understanding their music. b - they have the missing instruments recorded and then they play that in stereo during the exhibition. The performance impact lose a lot of power because the public basically don’t like to hear something and don’t see who is playing it. Moreover is really hard to play live with a fixed, not changable accompaniment.

–– 2nd scenario: sometimes a two musician band need some supporting musicians to play live in order to have much impact and presence on the stage. Their options are four: a - they are very lucky to have some people 44

be designed for, seems the sound. Let’s make an example: into the stringed family the generation of so many variants comes mostly in size generating higher or lower sound; then almost automatically smaller stringed instrument like the violin are more melodic with a soloist role, meanwhile bigger stringed instrument like a contrabass are more accompanist, they play a rhythmic role in the composition.

c - with loop stations, using different instrument in the same moment or strange body percussions they try to fulfill the need by themselves with the issue to learn and be able to manage last minute with so many new things in the same moment during a live performance. d - they are just forced to postpone the concert Accompaniment is really important and when it is suddenly missing on a music project where this is not planned from the beginning, this becomes immediately a serious problem.

The recurring problem is the need to fill a sound emptiness with a solution that doesn’t require specific and/or missing abilities. The users described above should be able to nullify that emptiness with the same natural way we start hitting the floor with our feet meanwhile we listen to some music; they need to be multitasking, without losing the focus on their main activity, playing their main instrument.

Except maybe the percussions family, it’s quite rare to start designing musical instruments by the role they will go to play into a music composition. But in those scenarios analyzed before a missing role was coming before a missing sound, so why not thinking more about the role in musical instrument design process? May this drive to a relevant innovation for the market?

2.2 - sound by our body

Loop station, that is a sort of mixer able to record some parts of a music performance and repeat them in loop, is at the moment one of the most influent innovation related to this issue. But playing with a loop station often means to play alone and to be very precise in rhythm because the recorded accompaniment will not accept mistake or unpredictable variations. Loop station is a sort of tool able to create so-called one man band digitally, but if this works pretty well inside a recording studio, the changeable situation during a live performance can make really hard the life of a “loop player”.

Market is full of an incredible amount of melodic and accompanist instruments; but the main starting point they start to

figure 2.3 - “The human body” exhibition

45

be very useful. Let’s take a human body, let’s consider its parts by systems and by limbs in the same moment assigning to them the most common “musical activities” related to most common musical instruments. Where more lines are meeting in the same point we have that part of the body generally quite busy in music playing, where less lines are converging that part could be a possible area to focus on (fig 2.4).

Playing music involves always a certain percentage of our body. There are surely more and less involving instruments, but what we can be sure about is that playing activity requires an action in terms of movement, so if we are playing a part of our body we will be surely busy with the best possible performance of this action. For example singing requires roughly a very demanding control of the full respiratory system, plus a perfect control of the tongue and the mouth; in addition a singer should usually be the attraction on the stage, so his full body is often in action. What about a drummer? The drum involves in the same moment hands and feet requiring a strong physical activity of the full body.

Basically, if we analyze all the movements of a musician during a performance, whatever instrument he is playing, there probably won’t be any part of the body characterized by inactivity. What make more sense is to say that some parts are more in action than others; so for example the both hands of a pianist are quite more in action than his feet sometimes pushing the pedals in order for example to make longer the sound of a note.

The conclusion of this study shows how the majority of the instruments are “hand-based”.

One other inportant aspect is a concentration at the face area where mouth and nose are figure 2.4- human body metro map infographics by Sam Loman

“By better understanding what music is and where it comes from, we may be able to better understand our motives, fears, desires, memories and even communication in the bro­ ad­est sense.”

Daniel J. Levitin, 2007

So what parts of the body are the less emploied by the average of the musicians?

In order to give the most satisfing answer to this complex question a visual scheme could 46

figure 2.5 - human body involved areas in most common musical instrument playing

G

F

E

D

C

B

A 1

2

3

47

MOST COMMON MUSICAL INSTRUMENTS voice piano classic guitar electric guitar bass guitar synthesizer violin double bass cello viola harp ukulele sitar banjo drums xylophon bongos djambè tambourine maracas drum accordion flute oboe clarinet saxophone trombone trumpet harmonica bagpipes tuba theremin hang 4

standard instruments, like drums and piano.

–– legs in general Musicians hands are often busy, but what many of them can do is dancing and generally moving during their performance. Their legs always move in a very specific way, which is often unique for every artist. If we observe for example the different positions into the articulation of a saxophonist knee during a performance, the perfect connection between sound and movement is incredible especially for some players. What if that movement could be translated in sound?

This is not a scientific difference, but just by observation it’s possible to affirm that some

very important for the woodwinds family instruments; they include of course all the respiratory system. Feet are quite less employed in playing, legs in general are not very inportant. Other areas of minor concentration are hip, shoulders, chest. On the scheme, we can notice a consistent amount of line converging to the brain: of course every man activity needs brain, but those lines where connected to the instruments requiring more concentration to be played; let’s make a comparison between playing a violin and playing maracas: to play violin onto an acceptable level you need to study at least 2 years, most of musician think they are not able to play it, the ability is almost a sort of done and a rarity in the world of music; in addition violinist, except very infrequent cases, are not able to do nothing else meanwhile they play violin, except dancing rarely.

In order to play maracas you just need a sense of rhythm, and some fantasy in shaking your hands; during maracas playing you can perform many different activities, like singing and play part of a drums with your feet; so in this sense, there are instrument requiring a relevant brain concentration, and in­ strument so simple that you can play more instinctively, with your brain busy with other activities.

figure 2.6- dynamic and static acting for musicians

MORE DYNAMIC

Below a list of the areas where it would be interesting to focus on for a new musical instrument design: –– feet There is almost none unable to move his feet following a music rhythm, even if he never tried to play music in his life. Spontaneous feet movement by rhythm jumps into an idea of more instinctive playing which could be a key factor of a new idea of accompaniment.

Feet nowadays are very important just in few

MORE STATIC

48

strange phenomenous: the infographics above is showing a comparison between heart beat human range and frequency in bpm related to the most typical music genres. Could such a similarity be just a coincidence? And moreover, why the most diffused music genres are placed exactly where the heart beat is on its standard health frequency? Classical music is consider the most important music genres and it just covers every heart bpm! The idea of unintentional sound was described before and with the heartbeat we have its possibility to make it possible.

–– the full body in movement Experimental apps connecting the movement of the full body are not new at all. When Microsoft Kinect sensor was born, the possi­ bility to use the full body as a sound controller jumped very soon in the mind of many app developer and musicians. The issue with the Kinect sensor is surely the need to stay on a certain position in front of a camera. If that could change, the body could surely be a very interesting sound controller, but this idea is closer to the dance and choreography.

–– eyes Expert musicians are able to play without watching what they do on their instruments.

There are thousand of studies about what our eyes can do, let’s imagine not so futuristic contact lenses for augmented reality, psyco­ logical studies on eye tracking, Google Glass technology. Our eyes movement could surely be a sound controller. The only negative aspect is that maybe nowadays, the world we live in is already enough “eye-based” and closing the eyes during a music performance is sometimes very helpful for some musicians.

musicians are more dynamic than others (fig.2.6). What make them all similar is that they are all able to move their legs sometimes or always during their playing position.

–– heart There is not a direct, mathematic coordination between music rhythm and heartbeat, but the­re are some connections. The sound of our heartbeat is the most significant and explicit sound our body is able to produce involuntarily. That sound could be amplified, mapped, mixed and alterated with digital sound effect producing unexpected results already performend in some experimental projects. In addition let’s have a look at a figure 2.7- main music genres tempo vs BPM activity

20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 210

MAIN GENRE TEMPO average BPM classical music 40/208 reggae 60/75 blues 60/90 rap 60/110 hip hop 80/110 disco 108/120 RnB 108/120 rock 108/120 alternative 108/120 pop 100/125 funk 110/130 electro 120/130 house 120/140 dubstep 130/150 techno130/155 drum’n’bass 150/180 harcore gabba 190/220

0 BPM 10 20 30 40 50 60 70 80 90 100 110

and male adult subjects

sadness calm normal activity agitation anger fear sport

120 130 140 150 160 170 180 190 200 210

220

220

230

230

240

240

250

250

260

260

270

270

280

280

290

290

300 BPM

HEART BEAT FREQUENCY average BPM in women feelings examples

0 BPM 10

On the next page a diagram is going to visually summarize where is going to be interesting to focus on.

300 BPM

49

figure 2.8 - interesting areas to focus on for a new musical instrument design

G

F

BODY PARTS SELECTION

E

D

C

B

A 1

2

3

50

4

2.3 - musical instruments for the body: market situation

about this tipology?

This is not supposed to be a business market analysis, but more an additional overview of what’s happening on the market regarding products partially or totally carrying out the performances highlighted until here. If the first chapter of research showed general contemporary trends, now it’s time to go more focused in what the market is really offering for the question above.

What’s a “body instrument”? Nothing re­ al­­ly well defined, the expression doesn’t exist into any scientific paper but it’s a way to define a certain category of product.

We could consider it as something easy to use, promoting instinctive or even un­ in­ten­tional playing, mostly focused on accompaniment task, contemporarily play­able with an other main instrument and last but not least, based on body movement. What the market is offering

The market situation could be subdivided in three different answers to the consumers: –– real products on the market (fig. 2.9)

figure 2.9 - real products on the market

Stomp Box Drum 2010

Farmer Foot Drum 2005

Various wearable foot percussions 51

figure 2.10 - market analysis for personal artists experiments

What you can really buy out there nowadays are real percussions sometimes wearable, sometimes just designed to take a natural movement of one of our body part and transforming it into a specific sound the user can’t usually modify. This is really an interesting situation to focus on: why just those instruments are really for sale today?

The answer is maybe related to the fact that if an instrument is secondary, semplicity and/ or low cost are really important. Customers of those products look for simple specific sounds they like to add to their main sound performance. In addition the already analyzed tendency to have a bad consideration of fully digital music instruments is surely an important aspect of this phenomenon. Behind those contemporary products there is quite a big traditional concept: the so-called “one man band“; a concept which guides this analysis to the next relevant market situation.

–– personal artists experiments (fig.2.10) It is possible to find on internet an incredible amount of pictures like the following one.

Many of them look also quite old: the de­ sign issue is not new at all; some old and contemporary musicians have been used to face the problem, and their creativity, often more powered by an artistic method instead of a designer one, have driven an incredible amount of crazy, brilliant or ingenious solu­ tions. The cultural memory of those solutions is quite hard to keep and use for product design processes. Musicians are also not very friendly in sharing what makes their performance and their sound unique. Except for rare cases, a superficial observation is basically the only possible activity regarding this phenomenon, which is anyway quite important to observe.

Very personal and questionable solutions can’t 52

be the starting point of an industrial design process but some details they have within themselves could be crucial.

Next trend is the most contemporary and often even futuristic: –– concepts still under development, proto­types and very young products (fig.2.11) Makers movement, crowdfunding campaigns, digital revolution, social media are shaping a totally new economy; the market is already overcrowded by a significant amount of pro­ jects looking like real products you can buy, but as soon as the client starts to go deeper into the purchasing operations, he discovers the product doesn’t exist, it’s a concept, a functioning prototype under test phase or a sort of open project the user is asked to develope further or build by himself. Not every client on the market is ready for this way of thinking, but this is not the main problem: a technology which looks already existing and functioning is often not real or is not feasible, or moreover is not really something a client would buy without trials. Revolutionary ideas are maybe into those contemporary projects; totally unuseful concept/prototypes, maybe just interesting for their creators are something we can meet into this confused market. In the field of music, sometimes it seems like what was in the past just “homemade” for the specific use of an artist pretends somehow to become an industrial product.

Open source projects could be very useful development tools; failed projects could be the demonstration of the mistake inside a certain concept, the craziest projects could be an indispensable source of inspiration.

In particular the contemporary trends regar­ ding this category shows a great focus on 53

figure 2.11 - market analysis for concepts still under development, proto­types and very young products

Onyx Ashanti Beatjazz Exo-voice Prosthesis instrument 2013

Joseph Malloch 2011-today

XTH SenseTM 2015-today

Joseph Malloch 2011-today

GePS - Gesture-based Performance System 2015-today

DrumPants 2013 - today 54

new interaction design, human bod ba­sed, built on the flexibility and user-adjustable power of digital sound, usually controlled by computer softwares.

Body extensions, wearable devices, responsive to movement of a musician, but sometimes also a dancer. This is the trend, but the impact on the market is uncertain. There are projects like Joseph Malloch incredible research with dancers, and Onyx Ashanti winner of an important music instrument design award; they are already about 5 years old, they look revolutionary but if they are still floating on a prototype phase after so much time they are maybe not very interesting or too much futuristic for their potential users.

There are products like GePS or even the more relevant XTH Sense: they have a strong impact on media attention; They are functioning prototype with very interesting concept and features and an idea of open source development. GePS looks like a really interesting idea not communicated very well, with a strong maker profile. XTH Sense looks like “the answer”, but the kickstarter campaign just failed and they are still waiting to find the way to develope their final industrial product. XTH Sense is by the way a very important source of inspiration; there is a lot to learn by its team activity and way to communicate the concept, but it’s also necessary to mind about the possible motivations of their contemporary failure in fundings. People are maybe very attracted and interested by the concept of bio-creativity the product shows off, but when this become the umpteenth bracelet, they probably start to be less excited.

DrumPants looks definitely like an already successful product; the basic idea behind is to play your body as a drum, probably inspired

by people tendency to do it automatically.

Drumpants transform an already natural activity into an almost infinite possibility of sound creativity. Additional application for VR and videogames are not less important.

At the moment Drumpants is sold out, as they declare on their website. What make it so successful? First of all, the product looks quite well designed from a user interface point of view; in addition Drumpants, already in the name itself, is designed like a metaphor of something the user already knows, the drums.

Drums is a kit of object? Drumpants it is too; you need foot and hands to play it? Drumpants use explicitly those symbols everywhere.

This simple design choice (the metaphor) is probably the most interesting aspect of the product, but of course, not really the most innovative.

2.4 - a general question: can we play music more in­stinctively?

This is the final general question; the chal­ lenge this research jumped on powered by the observation of contemporary market trends, studies related to musical instrument design and sound production.

“Can we play music more instinctively?” is also a provocation in the running world of automation; it’s apparently fully in symbiosis with the smart technology boom of the last decade, where smartness is not really a perfect synonymous of intelligence, but it’s maybe more iden­ tyfing the resilience and sometimes the automate respon­siveness of some tech­ nological systems.

55

Limits in the answer are quite wide: more instinctive is also about brain activity without concentration, is about passive brain operations. But we know such a few about our brain that what’s instinctive is really strongly connected with the amount of control every individual has of his own brain. In addition if we push the concept of an instinctive action till to an uncontrollable action, we are still somehow in the topic of instinct but in a different way: that is unintentional or involuntary. So the question “how we could play involuntarily” is an additional more com­pl­icated question connected to the main theme.

We are not talking about a disruptive question.

The general answer is already in se­veral pro­ ducts just appeared on the market in the last 5 years, but this doesn’t mean the research is ending; it’s prabably more reasonable to affirm that is just on its early beginning. Moreover the continuous run toward a more instinctive in­ter­action with technology in general is a goal investing many different markets and design studies. And if it could sound strange to connect the words “technology” and “musical instruments”, nowadays this is also a trend.

Maybe, as soon as we will be able to play more instinctively, there could be a new way to be creative. Music talent and satisfaction in playing, they surely consist also of that level of freedom; technology should suddenly be able to predict and sometimes anticipate this new interest about freedom in creativity.

mind about the relationship between those elements. figure 2.12- design orientation

USER involuntary natural

TECHNOLOGY

INSTINCT

INTERACTION

digital analogic

SOUND digital real In addition, music playing is about crea­ tivity and creativity is not just about mu­ sic! Many contemporary prototypes are cros­sing the relatively old separations among different arts. For example, when a new musical instrument is based on full body movement, suddenly it could be also an expressive technology for dancers or even an educative tool for people with disability; many studies affirm the positive impact of music learning in every individual, even not for professional purpose. Sound production can jump into theatre performances, artistic installation, movie making and so many other possible applications. Human being behave with instinct in so many different scenarios that the higher need of instinct could be

Five main “actors” are involved into the possible design process related to this general brief: user, instinct, in­teraction, technology, sound. The sche­me below is a “thinking tool” which could be helpful to 56

get a confirmation of the good value inside our final question: “can we play more instinctively?” And “Yes! We already started to do it” could be a fast but effective answer. By this way of thinking, we could successively ask “how far we can go with instinct in music?” which is probably the central question the next part of the research is really going to focus on.

spreaded in creativity as a general topic.

A quite good and simple example of what could really mean in practice a more instinctive approach in creativity is in a product design process like the one performed by the already analysed the ROLI company. In fact, when we observe a classic keyboard and a mixer full of buttons and potentiometer both transformed into a full touch setup sensor like the one showed in the picture below, we figure 2.13 - more instinct in playing by Roli music setup

57

sources

This Is Your Brain on Music [online, essay] http://www.ldv.ei.tum.de/fileadmin/w00bfa/www/Vorlesungen/Brain_Mind_Cognition/Your_Brain_On_Music_Essays_ Group_A_2011.pdf

2.Focus design

TEMPO-RADIO project [online, website] http://www.gemmaroper.com/TEMPO-RADIO

2.1 - a focus on musical accompaniment

Tempo [online, website] https://en.wikipedia.org/wiki/Tempo

Accompaniment [online, website] https://nl.wikipedia.org/wiki/Accompaniment

Tempo indications and beats per minute (BPM) reference for social dance genres [online, article] http://www.beatsperminuteonline.com/en/ home/bpm-beats-per-minute-reference-fordance-genres

2.2 - sound by our body Dave Kusek, Gerd Leonhard The future of music: Manifesto for the digital music revolution Ed. Susan Gedutis Lindsay. Boston: Berklee Press, 2005

Fradà, G., and G. Fradà Semeiotica medica nell’adulto e nell’anziano Piccin, IV edition, 2009 Polotti, Rocchesso Sound to Sense, Sense to Sound, A State of the Art in Sound and Music Com­puting Pietro Polotti and Davide Rocchesso, editors IUAV, Dipartimento delle arti e del disegno industriale, 2008

Haga, Egil Correspondences between music and body movement [online, thesis] 2008 Sam Loman illustration of human body as metro map [online, website] http://www.just-sam.com/

Ascoltare il battito del proprio cuore fa capire meglio le emozioni degli altri [online, article] http://www.ansa.it/canale_saluteebenessere/ notizie/medicina/2017/05/02/ascoltare-il-battito-del-proprio-cuore-fa-capire-meglio-le-emozioni-degli-altri_e88b0d3c-c80843a6-8e0c-31aa46d7830b.html

Daniel J. Levitin This Is Your Brain on Music 2007 58

Time [online, website] http://www2.siba.fi/muste1/index. php?id=102&la=en

GePS, Gesture-based Performance System [online, website] http://geps.synack.ch/

2.3 - musical instruments for the body: market situation

Joseph Malloch [online, website] http://josephmalloch.com/ Drumpants [online, website] http://www.drumpants.com/

Musical Instruments Market: Global Industry Analysis and Opportunity Assessment 2016-2026 [online, article] http://www.futuremarketinsights.com/reports/musical-instruments-market

Future Sounds Like This: 10 Odd Modern Musical Instruments [online, article] http://weburbanist.com/2009/11/17/the-future-sounds-like-this-10-magnificently-modern-musical-instruments/

Global Electronic Musical Instrument Market 2017-2021 [online, article] http://www.prnewswire.com/news-releases/ global-electronic-musical-instrument-market-2017-2021-300410269.html

Joseph Malloch, Ian Hattwick, and Marcelo M. Wanderley Instrumented Bodies: Prosthetic Instruments for Music and Dance In A Framework and Tools for Mapping of Digital Musical Instruments, Ph.D. thesis, Music Technology, McGill University, Montréal, 2013

Farmer footdrum [online, website] https://www.footdrums.com/

Emma Tucker Wearable pods let musicians compose and perform tracks as vibrations on your skin [online, article] https://www.dezeen.com/2017/06/30/ royal-college-art-graduate-marie-tricaud-touche-system-musicians-compose-perform-vibrations/

Node:0 [online, website] https://www.indiegogo.com/projects/node-0of-the-sonic-fractal-matrix-a-vision-of-anew-model-for-music#/ XTH Sense [online, website] h t t p s : / / w w w . k i c k s t a r t e r. c o m / p r o jects­/1814357959/xth-sensetm­-the-worldsfirst-biocreative-instrumen/updates 59

3.Concept Design

3.1 - a problem to solve: heartbeat sound amplifi­­ca­tion out of medical appli­ca­tion

the potential of a pure physical phenomenon.

What happen if we focus on a phenomenon, a physical one? what happen if we try to make it available for everybody? Following this approach, what this availability will be in the future is more in the hands of who is more interested about it, who is more creative, who is able to notice different potentials and drive multiple solutions. Who invented the first lute couldn’t predict the idea of the electric guitar, but both of them come from the same simple starting concept and across an history thousands of years old.

New interactions, new technologies, new musicians scenarios, everything could be a reasonable starting point for a design process, but the question is where the most successfull instruments available on the market took their real origin? what kind of starting point are more effective than others? The answer is really hard because music is such an ancient phenomenous in mankind history that the disco­very of how exactly a piano or a guitar where invented it’s really complicated and it would be quite out of the topic. Anyway, just by a deep observation, is possible to highlight a shared starting point: most of the families of instrument seems to be born from the application and the consecutive varying development of a physical phenomenon.

Let ‘s make some examples to be more clear: the vibration of a string produces a sound, there we go with all the string instruments family, a quite relevant group of objects sharing the idea of controlling in different ways the physical phenomenon of sound production by string vibration. Some other examples are easily understandable just by mentioning the compression of the air in a tube or the vibration of a leather under tension. Many failing design approaches in this market try to be from their beginning a complete, multiple solution; there is often no room for different variants starting from the same technology because often the technology itself, just developed for the new instrument, is not open enough campared to

There is a sound; an involuntary sound which is not very easy to amplify but we can hear whenever we want just putting our ear close to a human chest: it’s the heartbeat sound.

What if we compare it to the physical vibration of a string? Deep down is the contraction of a particular muscle.

Furthermore there is an interesting con­­nection in between heart bpm and different music rhythm by bpm; this was one of the inte­resting conclusion of the pre­ vious research related to the possibility to focus on the heart, as a human body part actually never really involved into any musical instrument playing activity, with the exception of some recorded tracks somewhere in mastering environments.

First of all, heart needs a specific introduction: it produces the most recognizable sound of our body. Two aspects of heart beat sound are very im­portant: –– it is the highest volume sound produced by our body –– it’s mostly involuntary Heart was told to be the “hidden center of being, the pulsating nucleus and the primordial instrument which articulates the rhythm of 61

name of Nero Project the multimedia project called Offrimi il cuore (2007-2017).

As it was already explained in chapter 1.5, the project, is about let artists performing with their own heartbeat sound creating intimate live exhibitions.

Fresu started is research about 10 years ago, facing soon a great challenge: how to amplify the heartbeat in a live perfomance with other musical instruments or in the middle of other body activities and of course with the best achievable sound quality.

He jumped immediately to the need of a specific technology contained inside a device which of course shouldn’t have been too much invasive for the players.

Listening to his experience, the challenge has been more than hard and it is still unsolved, because what could have been a solution for specific successfull exhibitions, it’s not really a final flexible and out of problems solution for every further project development.

Nowadays ma­ny artists are still showing their inte­rest in taking part to the Offrimi il cuore experience, many of them declai­ red the experience to be one of the best of their entire artistic career, one of them felt even into a trance state during the performance, which is surely a very interesting aspect of investigation into the idea of music creativity in heartbeat playing (Antonello Fresu, personal communication, June 24th, 2017).

Some artists left their own instruments during the performance, producing sounds through plastic bottle, table and cardboard box; “it’s like the meeting with something so primordial and deep would have generated the need of facing barehanded theirselves body, limiting whatever technical or lingustic mediation.

The same happened also to who emploied as

existence, the place of life and death, [...] the fundamental interlocutor in the meeting with ourselves” (Gianella Demuro 2012: 10).

Listening to our own heartbeat could be an incredible experience which can stimulate strong feeling and unexpected creativity.

Relaxing video on Youtube entirely based on pure heartbeat sound listening are very common and diffused, they say it calms, it focus the attention and it can also help a baby falling asleep.

This wide range of effects probably exists because heart­beat is actually the first so­­und we listen to by the time we start to exist in our mother womb. What are the other occasions we have to listen to it in our life if we are not doctors? Probably during an intimate hug with a friend of us, with our partner or with an intimate relative; also in these situations it looks like the presence of that sound is again linked with very strong feeling related to human social relationship.

After a strong sportive action, during relaxing activities like meditation, yoga or normal resting, after a big fear, when we are really angry; the heartbeat sound listening is just connected to strong feelings and immersive activities. Our mind is used to ear it in these situations, what would happen if we could listen to it, even with a higher volume, whenever we wish? How much do we know about our heart? “What would change in the artistic performance of a musician if he would be aloud to listen to what’s literally happening inside him in the meanwhile he’s performing?“ (Marco Senaldi 2012: 16).

During experiments and studies this work had the great opportunity to get in touch with Antonello Fresu, the psychoanalyst and the artist who developed under the 62

figure 3.1- “Offrimi il cuore” by Nero Project

instrument his own voice. They abandoned words in order to focus on fonemas and semplified vocal sounds able to talk more directly with heart and with themselves. [...] The availability of artists to play, while retaining each other’s recognizable and indi­ spensable personality, has shown that the heart can be truly an unusual and affectionate musical instrument to be complemented by the personal artist instrument. [...] This vital organ, far from being a mere electrified muscle, is a privileged passage toward our unknown inner reality; it is able to activate profound and unforgettable dialectical interlocutions with the most intimate part of ourselves” (Gianella Demuro 2012: 14).

doctors visits before birth; the device is not wearable, it’s heavy and it has some more technical issues we could highlight on a further development phase (Antonello Fresu, personal communication, June 24th, 2017).

So a gap has been founded on the market: there is nothing able to easily provide heartbeat sound listening to users different than pregnant women or hospi­ tal patients in general. So this is the problem to solve, the issue this work is going to focus on until every adequate solution: how is possible to live amplify the heartbeat sound with high sound fidelity and with a technolgy which doesn’t have to be too invasive for the user himself?

The problem is that Antonello Fresu is some­ how limited to develope the project onto a further level, because of some pure technical and technological limits. He’s continuosly asked to perform his quite unstable self-made technology for many creative or educative applications but he can’t goes really much further. He’s currently using an echo doppler device designed for pregnant women wishing their baby heartbeat listening out of the

The luck of being in contact with Nero Project is a great opportunity to have an immediate case study for direct experimentation and application. In addi­ tion, “Offrimi il cuore” is a 10 years old project and Antonello Fresu offered his availability for sharing all the experience he collected in the last 10 years; this can surely boost the project development and it provides a great possibility for a correct solution to the problem.

63

Laennec at the Necker-Enfants Malades Hospital in Paris. It consisted of a wooden tube and was monaural. Laennec invented the stethoscope becau­ se he was uncomfortable placing his ear on women’s chests to hear heart sounds. He observed that a rolled notebook, placed between the patient’s chest and his ear, could amplify heart sounds without requiring physical contact. Laennec’s device was similar to the common ear trumpet, a historical form of hearing aid; indeed, his invention was almost indistinguishable in structure and function from the trumpet, which was commonly called a “microphone”. (Wikipedia, 2017)

figure 3.2- René Laennec portrait

3.2 - the technology behind auscultation

For example, Offrimi il cuore project is currently based on medical equipment adapted to the live music perfomance requirements. This is quite an interesting situation if we consider that the most well-known and successful heart auscultation method was born exactly from a musical instrument concept.

How many technologies we have nowa­ days in order to listen to our heartbeat?

First of all, thanks to a huge amount of wearable devices just appeared in the last decade, today is really easy to get a heart rate detection in bpm (beat per minutes).

This technology is going to be integrated even in many new generation smartphone but there is actually nothing related to sound production or amplification; applications are mainly related to the sportive or healthy market. Moreover translating bpm detection into sound doesn’t have nothing to share with the real heartbeat sound listening: the achievable result with this technology would sound really artificial and totally impersonal. This contemporary trend could be just an interesting demonstration of a growing interest in heart activity and monitoring.

Rappaport and Sprague designed a new stethoscope in the 1940s, which became the standard by which other stethoscopes are measured, consisting of two sides, one of which is used for the respiratory system, the other for the cardiovascular system. [...] Several other minor refinements were made to stethoscopes until, in the early 1960s, David Littmann, a Harvard Medical School professor, created a new stethoscope that was lighter than previous models and had improved acoustics. In the late 1970s, 3M-Littmann introduced the tunable diaphragm: a very hard (G-10) glass-epoxy resin diaphragm member with an overmolded silicone flexible acoustic surround which permitted increased excursion of the diaphragm member in a Z-axis with respect to the plane of the sound collecting area. [...] In 1999, Richard Deslauriers pa­ tented the first external noise reducing stethoscope, the DRG Puretone. It featured two parallel lumens containing two steel coils which dissipated infiltrating noise as inaudible heat energy. [...] In 2015, Dr. Tarek Loubani announced an open-source 3D-printed stethoscope based on the 1960s-era Littmann Cardiology 3 stethoscope, which is out of patent. (Wikipedia, 2017)

The rest of the heartbeat auscultation technologies are related to medical app­li­ca­ tions. Just in 1816, the stethoscope was invented in France by René

64

figure 3.3 - heartbeat sound amplification technologies

Littman 3200 electronic stethoscope

Glia 3D printed stethoscope

classic stethoscope

Angel sounds echo doppler for domestic use

Echo doppler for domestic use

Steth IO by Suman Mulumudi 65

A SCOPE™ Noise immune stethoscope (a military patent)

3D printed opensource electronic stethoscope

common veterinary doppler

common doppler in hospital

compact veterinary doppler

cardiotocograph 66

to provide real good quality sounds, but not efficient in some situations and application, a Doppler stethoscope is an electronic device that measures the Doppler effect of ultrasound waves reflected from organs within the body. Motion is detected by the change in frequency, due to the Doppler effect, of the reflected waves. Hence the Doppler stethoscope is particularly suited to deal with moving objects such as a beating heart. It was recently demonstrated that continuous Doppler enables the auscultation of valvular movements and blood flow sounds that are undetected during cardiac examination with a stethoscope in adults.

The Doppler auscultation presented a sensitivity of 84% for the detection of aortic regurgitations while classic stethoscope auscultation presented a sensitivity of 58%. Moreover, Doppler auscultation was superior in the detection of impaired ventricular relaxation. Since the physics of Doppler auscultation and classic auscultation are different, it has been suggested that both methods could complement each other. (Wikipedia)

Thinklabs ONE digital stethoscope Few other 3D printed stethoscopes, sometimes just analogic, sometimes with home-made electronic amplification solutions are avai­ lable on internet usually as opensource pro­­jects. A 3D printed iPhone cover concept design transforming the famous smartphone into a sort of electronic portable stethoscope is not missing. The sound quality provided by those project has to be tested and with the exception of the Loubani project, it is never employed for professional applications; it’s a trend and a possibility of exploring the core of passive acoustic technologies in this field.

What surely this recent trend tells is again a growing interest in heartbeat sound listening.

Keeping in mind the history of the first heartbeat listening tool could be useful for the topic development. It’s a quite cheap technology which is interesting especially because it’s analogic, but of course it has some limits related to sound detection, because the bell is really sensitive to whatever acoustic stimulation, and this is not always a positive feature.

What is important to highlight is that echo Doppler technology is mostly useful for visual output and it was developed initially for this purpose in many different applications. Its auditive outputs are just recently exploited, with positive results in some specific ap­ plications where passive acoustic systems are not efficient.

«A military noise-immune Doppler based stethoscope has recently been developed for auscultation of patients in loud sound environments (up to 110 dB)» (Wikipedia 2017). There are many historical design cases where a military technology was the perfect starting point for a consumer innovation.

One embodiment of the stethoscope utilizes a passive acoustic system. Another embodiment uti­ lizes an active Doppler system. The passive and ac­ ti­ve systems can be combined in one stethoscope

Leaving the passive acoustic systems, able 67

Veterinary science also needs heartbeat listening. The devices usually employ echo Doppler technology comparable with the same human one but with a slightly lo­wer power.

The probe and the full technology in general are usally smaller and more portable.

enabling switching from a passive mode to an ac­ t­iv­e mode suitable for use in very high-noise en­ vironments. (patent US8265291 B2 2011)

A noise-free stethoscope able to perform in 110 dB could be really an interesting starting point because for example the main problem with stethoscopes described by An­ to­ nello Fresu in Offrimi il cuore project is exactly the low level of insulation of the standard stethoscopes. But unfotunately, the issue with military technology is always accesibility, the patent is by the way online available, so its analysis could anyway offer some suggestions in the hypothesis of a echo doppler/stethoscope hybrid.

The cardiotocograph is essentially an hospital equipment providing visual output in the well-known cardiogram.

It has often the possibility to provide also an auditive output by the traditional echo Doppler technology. Its sizes are usually qui­ te relevant because of many other features and technology inside unrelated to sound detection. In addition the sound quality is too much composed by environment unde­ sired sounds (Antonello Fresu, personal com­ munication, June 24th, 2017).

The foetoscope or Pinard orn is an other passive tool, not really popular, which is really similar to the original idea of Laennec stethoscope. Its use is mostly related to pregnant woman during the last weeks before the birth in order to understand the fetus position. The tool is not really common in hospitals where Doppler and phonendoscope are also useful for the same task. It’s a traditional tool still loved by some midwife, surely not really expert in phonendoscope or doppler auscultation.

Just a useful disambiguation is necessary: the terms “stethoscope” and “phonendoscope” are usually in the habit as synonyms, but the first is actually related to chest auscultation meanwhile the second include a general ascultation of all the body innards sounds.

Nowadays the stethoscope, is usually the most common word to call all those devices.

Sometimes in the stethoscope the bell is double side, but practically this distinction is vanishing in marketplace and the differences are minimal and sometimes fictional.

About 15 years ago, the Doppler tech­ nology was designed to be just a medical pretty expensive technology; nowadays the hi­gh-priced equipments for hospitals mar­­ket are still available, but the Doppler ste­­thoscope is becoming a quite cheap tech­­ nology which satisfies the need of a specific market: the pregnant wo­men wishing their babies heartbeat au­scultation from the 12th week (because la­ter, around the 20th, the auscultation can be executed also by a classic stethoscope, if the user knows how to use it). Nowadays, in a range of price from 30€ till 150€, we have a tens of similar devices, all designed to be hold and overpass above the mother womb; the operation is always helped by a gel or an oily substance in general which enounches the sound quality, this happens also in the more expensive technologies in the hospitals. Digital sound is transmitted to 68

intensity/volume controller (potenziometer) 2 jack audio output board on/off led light 2 ultrasound diffusors 9V battery case

figure 3.4 - What’s inside Angelsounds fetal Doppler by Jumper

shows the most interesting potential of development. The picture above shows an Angelsounds® device produced by Jumper, one of the most diffused product for fetal heartbeat domestic auscultation. As it possible to notice the device can be largely minimized; its shape and empty spaces are just the result of some ergonomic study for the idea of handling it. Moreover the motherboard could be surely better developed in order to increase the sound quality. Ultrasound diffusor is very small and it can be surely powered and inserted into a different material eliminating the gel need. The heavy and bulky 9V battery is surely a solution for the final low price of the product and it can be changed, minimized and improved. By an approximated valua­

headphones or speakers and it can usually be recorded thanks to desktop software interfaces or integrated device display systems. The probes have different shapes based on different ergonomic studies, the presence of a sort of handle is the standard. The general quality of the products, in terms of materials and technical solution are quite low: this probably because they are generally somehow a kind of disposable technology, so specifically targeted, for such a short period like few weeks in pregnancy. That’s why is not complicated finding private sellers publishing second-hand devices on websites like eBay. Watching inside this technology, after many different analysis and trials, 69

be done in a different way: by searching on publications and websites, just really few sources about heartbeat sound qua­ lity are available. Some sound aspects li­ ke insulation from noisy environments and re­cording features are the only requirements some advanced products are showing off. The stethoscope technology is such a simple and already stan­dard technology providing good sound quality, so the quality itself is not really an issue for medical application, just because an higher, more performative sound production is not really required. Most of the doctors are still prefering the analogic tools instead the advanced digital one. The problems start when that sound needs to be amplified and maybe be played contemporarily with other sounds like in the case of music live performances.

In order to approach the issue is generally worth to consider again two main ways to listen to heartbeat sound: the phonendoscope is the analogic and passive tech­nology. It can provide a really high quality sound in comparison to the quality of the tool itself. One of the most expensive phonendoscope on the market is the recent developed electronic Littman 3100, a slightly traditional stethoscope powered by an electronic system able to amplify the heartbeat sound, record it, and isolate some background noises.

The main problem of this technology —no matter if it is analogic or electronic up­ graded— is that during a live perfomance si­tuation, whatever instrument the play­er is supposed to play with his heartbeat in background, that sound will be immediately catched by the phonendoscope bell, disturbing the full amplification system; the phonendoscope is not able to send to

tion echo Doppler technology could fit into a box 70x70x15mm, but this is not considering the issue of catching the sound in a perfect and always subjective body position. Those sizes are demonstrated also by the military patent showed in para­ graph 3.2.

3.3 - the possible heart­ beat sound After the analysis of the technology behind hertbeat sound amplification, it’s time to transport it into the unconventional sound production topic, so the audio output be­ comes the central focus and every available technology has to be considered for its audio output quality. The heartbeat sound is of course really well-known by everyone but it can have an incredible amount of variations in amplification just by the employment of different technologies. The problem in making sound considerations about all the methods is related to the fact they weren’t always designed to provide amazing sound experience and recording fidelity. The te­ ch­­ nologies are medical and they were de­veloped in order to work perfectly for the doctor-patient situation with the goal of the most efficient desease detection. All the following considerations are the result of a collection of users experiences, talks with experts, websites product client feedbacks and the development experience of the case study Offrimi il cuore, which with 10 years of trials in perfectioning the sound quality issue, had a lot to share regarding devices and technologies already experimented in the project goal. This couldn’t 70

figure 3.5 - different points of heartbeat sound auscultation

designed for the application. In order to catch the sound from it, some kind of ergonomical and audio “hacks” are mandatory, starting from an essential microphone setup.

the amplifier just the pure heartbeat sound.

Even the about 400€ Littman 3100 —with its insulation technology filtering parasitic sounds— is able to delete the undesired capture of the soloist instrument sound. This expensive technology wasn’t even adaptable to be linked to a standard amplifier because of a risk in delay for hertbeat playing during a live event, and in this specific case delay in sound is surely unacceptable (Antonello Fresu, personal communication, June 24th, 2017). This problem is basically related just to the music target, because other targets not interested in other sound production close to the bell, could actually fit this technology; but with the general requirement of a adaptable technology, the further design should mind also about this issue.

In addition the phonendoscope is surely not

The reduction of background noises co­­­ming from skin rubbing is also an other problem related to the need of somehow fastening the stethoscope bell into the precise point for perfect heart auscultation; the point is restricted to a cer­tain chest area, but it is slightly different for everybody. One other unavoidable aspect is the possible temporary disappearance of the sound in relationship with breathing in and out; but this, let’s say, shouldn’t be considered such a problem: it’s probably more correct to say that it is a peculiar characteristic of the heartbeat sound.

71

“Heartbeat sounds are semi-perio­ dical signals that are generated by blood turbulence and the beating heart.”

relationship with the position of the auscultation device. Every technology has this limit because sound waves inside our body have to overpass many different layers in different directions and auscultation technologies are focused on relatively small body area. There is nothing detecting hertbeat sound from distance, a contact is required and this contact has to be selective.

Farshad Arvin, Shyamala Doraisamy and Ehsan Safar Khorasani, 2009 Moreover the judgement of the perfect point for amplification is basically quite subjective: when Fresu search the heart of the artist, he decide subjectively which is the best one, but some different selections or different experimentations by users are easily predictable.

Everytime Antonello Fresu prepare a per­ fomance of an artist he spends time in “searching his heart”. A similar operation is executed by every doctor, but faster because they surely know better the traditional point in relationship with people build and they look for deasease detection instead of sound quality. Every individual has a sort of best point where his heartbeat provide the best auscultation. This is surely also subjective, because for everyone some points could be more interesting than others. But for example, Antonello Fresu, like every future potential user, is asked to select a specific point where the device is going to be fixed. This could be a long process and it can be done just few minutes before the performance, so let’s shortly describe the Offrimi il cuore case study interactions in order to discover what does it mean today amplyfing heartbeat for live performances with market available technologies:

The ultrasound cardiograph or doppler stethoscope is an electronic active tech­ nology. The sound becomes digital and it loose surely some beautiful frequencies, but it also acquire more sensivity for other interesting frequencies and body sounds.

Moreover, the device gives a better sound quality with the help of a gel applied on the area of auscultation.

A need that can be quite annoying and almost impossible to manage with for some specific targets. Do you imagine a dancer with the need of applying a gel on her chest before a performance maybe 1 hour long, probably under a specific costume?

The technology has by the way a re­ levant advantage: the audio output is ea­­sily detectable by an amplifier and it usually employ the standard small jack transmission.

The scheme above shows the main four auscultation point for heartbeat sound. This is surely a key issue for the concept development: heartbeat sound changes in

1. Someone looks for the best personal heartbeat point of the performing artist about two hours before the performance; 2. Someone draws with a marker a reference sign on the artist body in order to remember where that sound achieve 72

the best detection (this step is necessary just because the artist can’t stay a lot of time with the device attached to its body, because it’s heavy and annoying to keep attached on skin);

figure 3.6 - Antonello Fresu fixing the device on artist chest

3. Ten minutes before the perfomance is going to start, the device is fixed to the artist body by an other person and the position is on the previous marker sign; 4. At the end of the performance the all equipment has to be removed by an other person.

Hopefully none will believe such a process could be acceptable for an interaction process with something supposed to be a musical instrument or whatever. The heartbeat sound specific and personal point of auscultation it’s a complicated issue the technology this work is going to develope has surely to mind about. A reasonable solution could be in the idea of a wider and flat surface of application able to capture multiple sound inputs and merge them into a complete and satisfing sound output. This somehow already happens in few echo Doppler devices characterized by the employment of a flat probe.

In addition a double jack audio output, like the echo doppler device currently on use has, could be really helpful exactly for this heartbeat sound body “research” (Antonello Fresu, personal communication, June 24th, 2017).

but the quality in sound compared to the difficulty in technology management is enough acceptable to keep this choice instead of some others already tried in the past.

Fresu experience declares the wish of the beautiful analogical sound provided by the stethoscope (he tried also the Littman 3100), with the consciusness and the experience of the impossibility of an acceptable level of insulation. The solution with a device called angelsounds has to be considered contemporary because the problems of audio quality and ergonomy are still quite relevant (Antonello Fresu, personal communication, June 24th, 2017).

Soud post-production technology and cul­ ture could surely help the achievement af a reasonable sound qua­lity with the echo Doppler technology; after all, Fresu never deeply tried to modify from inside the device he’s currently using.

One of the devices described in the chapter 3.2 (the affordable echo doppler for pregnant women) is the contemporary choice of the technology adopted in Offrimi il cuore project. The need of a gel and many ergonomical problems are still an issue,

Regarding the recording studio environment heartbeat sound has been detected and recorded in different ways so many times and with really positive results: Muse’s 73

“Follow me”, “Cosmic Love” by Florence + the Machine, “Smooth Criminal” by Michael Jackson, “Teardrop” by Massive Attack, “Running Up That Hill” cover by Placebo and “The Dark Side Of The Moon” album beginning

by Pink Floyd are just some example of a effective heartbeat sound recording in music field. The most difficult situation is the music live performance, where sound from every source could meet the amplifier and disturbing the all per­ formance. «Generally, the recorded heart sound consists of the background audio and other organs sounds. Therefore, pre­­processing of heartbeat sounds is an important task for de-noising of sam­ ples»(Farshad Arvin, Shyamala Doraisamy and Ehsan Safar Khorasani 2009).

A professional mixer would be able to correct many problems, but this is not always enough, especially in specific lower budjet situations.

An integrated filter, user-adjustable or automatic responsive to wrong inputs could be a markable solution into the project development process.

Some u­ sual­ ly digital filter called band pass filter, have the possibility to restrict a microphone input between certain ranges; this could be particularly interesting if we consider that the heartbeat sound travels basically on low frequencies (the frequency of heart sounds is low in range between 20 and 150 Hz), so such a device, accurately settled to capture just those frequences could be the most efficient solution, even in solving the feedback problem in passive acoustic technology. In addition, considering the hypothesis of a microphone integration, an even­tual deeper and further analysis of mi­crophones technology could be the key factor of an effective innovation. Some

piezoelectric microphones are for example designed specifically in order to work with low frequencies. A so called LBF (low bass filter) could stop the microphone detection of high frequencies typical for example in rubbing sounds. Every further more detailed similar option has to be considered in relationship with the final selected technology.

A totally different sound issue is the topic of sound output, or better to say “sound receiver”. If we consider the possible futuristic use of this technology, a strong distinction has to be highlighted: there will be –– pure heartbeat listening, so just the heart sound amplified by a speaker or a headphones set –– paired heartbeat listening, so the heart sound paired with other audio output like a playlist on spotify, a dancing soundtrack or a piano playing.

The pure heartbeat listening will never be an issue, the problem has is origin in the pairing activity: in all the situations where a slightly professional mixer is involved, a simple solution of bluetooth audio system or traditional jack audio output are functional.

In the case of a paired heartbeat sound in the same amplification device the issue is related to the mostly “monodirectional“ sound input technology of most of the devices, like headphones or bluetooth speakers, avai­lable on the market.

In this specific case the audio signal should be sent to a device capable of an integrated mixing of the audio input, this suddenly predict the need of a dedicated amplification devices for common users because the device itself can’t be charged of this task as audio signal supplier.

74

3.4 - who could be intere­ sted in heartbeat sound?

figure 3.7 - Paolo Fresu performing with his heartbeat sound

“There aren’t two hearts equal to each other, there aren’t two heart­ beats you can overlap, [...] we are our heart, heart is our center, secret and unheard.”

Marco Senaldi, 2012: 18 Who are the potential heartbeat players?

This work was initially inspired just by the world of music and musicians are still surely the primary target group but also one of the most complicated regarding issues already described in previous paragraph.

The musicians focus is also pushed by the greatest deal of attention in Offrimi il cuore project experience, which was mostly music related. Among 40 official artists, 35 were musicians; their interest in taking part to the expe­rience was often spontaneous with very specific and subjective results for everyone of them. Following a personal translation of a Marco Senaldi essay, three main musicians reaction to heartbeat sound playing paired with a primary musical instrument are distinguishable (Marco Senaldi 2012: 20):

and follow it, but he tries to change the rhythm in relationship with the musical exhibition”.

Paolo Fresu is an active Italian jazz trumpet and flugelhorn player, as well as a composer and arranger of music. He is able to perform a 45 minutes concert just based on his own heartbeat sound. Because is now used to it, he started also to play with his personal sounds, he improvises, he alterates it with effect with the same equipment someone could use in order to add effects to an electric guitar.

An other interesting musician reference is in Mario Brunello, an italian cellist, surely more classic than Paolo Fresu. He’s not a jazz man and he had the idea of playing Bach on his heartbeat sound; the idea was succesfull and an interesting thing happened during the performance: he temporarily became a jazz man improvising Bach, with his involuntary heartbeat sound in background.

During 10 years of performances and expe­ riences Antonello Fresu, Paolo’s brother, noticed that best performers are jazz men with a good skill in improvisation (Antonello Fresu,

1. “who follows the heartbeat sound like a sort of metronome, so he tries to built a musical composition on it”; 2. “who tries to follow the rhythm adding interpretation, considering it not just a background sound but the completing part of their music”; 3. “who tries really to play his own heart, then he doesn’t just listen 75

structure of his performance (Antonello Fresu, personal communication, June 24th, 2017).

The possibility of heartbeat rate control could sound almost like a crazy idea, es­ pecially asking to the majority of the people thinking to don’t have such a control of what happens inside their body; but this is just a consequence of something we don’t really know, especially in western society. It’s similar to what happens when someone starts to study singing: the teacher un­derstand what the beginner voice can do, he knows; teacher task is to push his student to reach a certain level he sometimes be­ lieve he’s not able to reach or control, then, just by training and technique, the beginner suddenly reaches that level and he’s of course surprised about something he never taught to be able to do with his own voice. Talking is something we start to do quite inconsciously when we are infants, then someone start do discover and go deeper in what sort of experience could be singing. Heartbeating is involuntary, from the beginning of our existance till our death; but what would happen if someone starts to knowing it better? A higher level of control is a hyphotesis already experienced.

personal communication, June 24th, 2017).

After all we are talking about a percussion which is quite difficult to follow, mostly for its variability. The quote below by Gershwin is particularly interesting if we consider that heartbeat is the first and the fundamental life signal.

“Life is a lot like jazz...it’s best when you improvise” George Gershwin (1898 - 1937) In addition Fresu noticed a great ability in percussionists: as soon as they were able to listen to their heartbeat they started to understand very well how to manage with it, and controlling feeling and inspiration/ expiration process, they were basically able to control their bpm (Antonello Fresu, personal communication, June 24th, 2017). So the heartbeat is surely an unintentional sound automatically produced by our circulatory system, but for some players, and maybe for all of them just by more training, it could start to be less involuntary and more controlled during a performance. After all, Offrimi il cuore approach was always to push for improvisation, in order to preserve the natural very expressive surprise of the artists in dueting with their own heartbeat; but some artists now confident with the experience after some trials already started to show a different behaviour with it: they control it, they alterate it by effect, therefore they put creativity on it.

An interesting story is related to a spanish mime, Juan Miguel, an artist used to manage with his body in 30 years of career; well, after few seconds, he was immediately able to perfectly control his own heartbeat, speeding or slowing his rate in coordination with the

Gender is an other topic. It’s not a coinci­ dence that in Offrimi il cuore project most of the participants are male: fixing the current selected technology to a woman breast is not just a challenge for the embarassing situation of the person who has to do it, but also for the harder task in finding the best point for heart auscultation. Moreover the device has to be tighten by a sort of artesenal bib Fresu created and in this case, the breast can’t 76

surely be strongly tighten (Antonello Fresu, personal communication, June 24th, 2017).

Since none wish this project to be sexist, female user targeting is an other relevant issue to solve.

the contemporary musicians performance orientation because of the already explained technical problems of the amplification instrument. But recently in Belgium, Fresu had the chance to test the Offrimi il cuore project to a particular target group: children.

The experiment, which took place in three different belgian cities, was based on two musicians able to live improvise with the children live hearbeat sound. Children were asked to offer their heartbeat sound for the performance in front of other children and it was quite interesting for example to listen the changements of their bpm before and after the music was starting. The activity was also quite funny and educative for them, and someone is already thinking about further educational activity for higher body knowledge in childhood (Antonello Fresu, personal communication, June 24th, 2017). Some scientific studies demonstrate the potential of heartbeat listening in order to get into a closer contact with people, stimulating socia­ lization and knowing more about our­ selves intimacy. What if your favourite band try to play something based on your heartbeat sound? What if a painter could feel more inspired in painting with his own heartbeat in background? What if sportive people get a really positive body reaction in performing workout in the meanwhile their listening to their own heartbeat?

What about an entire teather script based on heartbeat sound of the actors? What if a person used to meditation practices feels very effective the addition of the heartbeat live listening to his activity?

What if listening sometimes to our own or someone else heartbeat sound would be just an interesting experience? what if slowing or speeding artificially our

If musicians are the primary target group, they aren’t surely the only one. For example the Offrimi il cuore project was actually born with an other input: its creator was interested in the creation of a contemporary art installation where normal people were asked to sit in a dark room an listen to their own heartbeat.

When a visitor was going to sit on an armchair inside this dark room, his heartbeat was going to be amplified and the other spectators were able to listen to it. Moreover, when he was going to leave, his heartbeat would have been recorded and left to the speakers until a new visitor was going to come and a new cycle could start. This full artistic performance was having an additional goal in creating a sort of archive of recorded heartbeat sound by every different people (Antonello Fresu, personal communication, June 24th, 2017).

Let’s suppose your grand father died 10 years ago, have you ever imagined to have the possibility to listen to his heartbeat sound? Well, probably no, and probably this will never happen in reality, but this possibility it surely proposes a mix of very emotional and somehow also scary feelings which could be probably interesting for many people. A heartbeat library is a sort of idea in between life and death which is not extraneous to some fantasy or science fiction scenarios we already had the possibility to meet some predictions about in some novels or Hollywood movies. Fresu had to change later the initial idea of his work towards 77

those could be the most interested targets for heartbeat sound amplification (1st page) and/or listening (2nd page) figure 3.8 - target analysis

acrobat

actor

dancer

juggler 78

heratbeat sound amplification

musician

meditation

iperactive, stressed

traveller

sportive

sculptor 79

heratbeat sound listening

creative

heartbeat sound would actually relax or speed up our real heartbeat for calming applications or body preparation for sport­­ive performances?

“following the heart.(Barnaby D. Dunn 2010)

We could really understand how music can influence our personal condition of relax or agitation looking for example to the incredible amount of relaxing music everywhere or to the scientifically declared doping effect of music in human body. Surely the music able to relax our mind is on low bpm range meanwhile you can feel really energetic listening to techno music, that’s how people is able to spend hours inside a club in Berlin or performing well inside a gym. If we move listening to a higher level of interaction and control, slowing or speeding the heartbeat sound could achieve other very interesting and diffuse targets. The answer to this possibility could be just in prototyping and testing with real potential targets.

Sometimes we are scared about our heartbeat activity because we don’t really know it that much. Heart desease death is always on the top of the classifics related to worldwide death causes (Statista 2017). Making easier the heartbeat auscultation could also make the difference for people with heart desease or generally in prevention of those at risk.

“Listening to your heartbeat helps you read other people’s minds” New Scientist, 2017 The interest about heartbeat sound in other artistic disciplines is not uncommon in some recent projects. The idea of following the relationship between heartbeat bpm and music genres bpm is already described onto an article like “Music Playlist Recommendation Based on User Heartbeat and Music Preference” by H. Liu, J. Hu and M. Rauterberg (2009). The study related to how a human being could react in heartbeat listening are the center of many scientific articles and experiments like the following one:

In Offrimi il cuore project, among the 40 artists officially involved, there were two dancers, one mime, one illustrator and one sculptor. This is surely not enough to affirm the reality of other target applications, but it is surely a great start. Antonello Fresu recently also received a request by an acrobat working at Cirque du soleil interested in proposing a show with her heartbeat sound in background.

The task is even more complicated, because one of the most relevant problem in the contemporary technology which is being used is exactly the overstated movement of the performer. By the way all those kind of interests can be considered a demonstration: the general simple idea of giving to people the power to amplify their heartbeat so­ und seems to be an oppotunity full of po­tential, not just for musician or artists in general, but basically for every user somehow and for some reasons, not al­

[...] increasing interoception ability ei­ther helped or hindered adaptive intuitive decision making, depending on whether the anticipatory bodily signals generated favored advantageous or disadvantageous choices. These findings identify both the generation and the perception of bodily responses as pivotal sources of variability in emotion experience and intuition, and offer strong supporting evidence for bodily feedback theories, suggesting that cognitiveaffective processing does in significant part relate to

80

figure 3.9 - Pi KeoHavong, dancer performing for “Offrimi il cuore” project

ways so effective to predict, interested in heartbeat sound.

looks so unacceptable mainly because of the mostly handcrafted adaptation of a technology designed for a totally different category of application and user. How could an object designed for pregnant women share an acceptable interaction to a totally different target: artists (musicians, dancers, sculptors, mimes, etc..in the case of Offrimi il cuore project)?

“[...] visceral perception plays a role in the experience of the intensity of emotions.”

Heartbeat detection and the experience of emotions, 2000

More than a musical instrument inte­ raction this re­search is moving closer to the world of sound amplification.

In particular, the technology will surely include a microphone technology or the basic concept of it; hence from an interactive point of view it would be in­teresting to mind about the interaction we have with microphones. Microphones are probably one of the most plug&play technology, they don’t need pre-setting, they usually don’t have any controller, despite sometimes the volume or some different preset mode; it is possible to say they are more “macrosensors” than products. After you connect them into a audio system, they provide their specific standard

3.5 - possible heartbeat interaction How potential target should interact with this new technology? In paragraph 3.3 a quite arguable interaction was described with the support of the Offrimi il cuore case study, surely a fertile platform for interaction design development. The picture at the next page shows the bend the artist is usually asked to wear during his performance with his personal outfit above or without. Such an interaction 81

Musical equipment is the perfect example for this issue: musicians love to combine different tools, because by this fully personal combination of technology they become unique with their specific “sound set”. They hate limits in freedom of use, they love to customize sound technologies for different situation; the example of pedals effect for electric guitarist is one of the most effective one: every electric guitarist as always is one unique equipment of effect modifying the pure sound transmitted from the guitar to the mixer; products with those effect integrated are sold but they are never the favourite choice of experts. Professional products on the market regarding this field are like this, they are usually never trying to be all-in-one tools, because they need to be part of a more complicated evolutionary system.

A piezoelectric microphone it’s a specific typology of microphone well-known for the amplification of sound directly catched by body under vibration. In guitars, a piezo is the tool able to transform a not amplified guitar into an amplified one. How does the specific interaction work?

output, different for every sensor; then differentiation is mostly in quality and technology.

But what happens if the user wants a different audio output? he put an analogic or digital “filter” in between the microphone and the speaker. Everything becomes more clear with an example: if a singer wants a strange effect on his voice, he plug the microphone into an effect, then his alterated voice signal will be brought to the speaker. The singer doesn’t want a microphone with that effect integrated, because then, he should have a specific mi­crophone for every effect he would like to have on his voice. From an economical, but also an interactional point of view, it’s much more clever that the singer will have a good quality microphone able to perfectly amplify the natural sound of his voice; later he could always buy an effect he wants to use during a specific moment of the song, there he will activate it and the sound will be modified, not permanently. Of course on the market you can find microphones with specific effect integrated, sometimes even with switch on/off of the effect itself applied to the product. But can you imagine a microphone with ten different integrated effects? The cost would be unacceptable, and what if the client discover later he need eleven effects or just thre of them?

1. the guitarist plays his not-amplified guitar without any problem; 2. for a live concert in a big place, his guitar is supposed to be amplified, still allowing his ownmovement on the stage: he takes the piezo, he sticks it on the guitar body with the classic adhesive some standard models have from one side; 3. the piezo has usually a standard audio jack sound transmission, and it directly goes to the mixer where the guitar sound will be amplified for the audience with some mimxer editings in order to create

figure 3.10 - example of a guitarist effects setup

82

an acceptable sound for the audience; 4. after the performance the guitarist can choose for keeping the piezo on his guitar or removing it until a new performance. (this is also related to how good and flexible is designed the way the piezo is fixed to the guitar, models are different).

figure 3.11 - Touchè prototype

This all interaction could be a good starting point for the analysis of the final interaction with a “heartbeat sound amplifier”.

One of the most complicated issue is how to manage with the need of sticking something to skin without causing any irritation or other problems. This is also an issue hot in general in wearable technology; users look almost divided into who is ready to stick or put inside his body every kind of technology with a sort of enjoy atmosphere for new bionical trends and who still regrets too invasive technology and wish to be always easily able to keep away technology out from his body. The project called Touché and developed by Marie Tricaud as final project degree at Royal College of Art in London, is an impressive example of the first trend; this imminent choice will surely make happy someone and unsatisfied someone else: a relevant way to choose would be also to look at the most functional ed effective solution just in matter of sound quality. In addition sticking technology derectly to skin is still not really an easy task; “it’s far from stable for now but I have a few directions I want to explore. I know there’s quite a bit of people/ companies interested in the matter though because it would be really helpful to be able to stick wearables straight to the skin!” are the words of Marie Tricaud during a personal communication in July 16th, 2017.

On the picture below an image of Touchè. It’s

a recent project surely in the wave of this research. Touchè is about stimulating human skin with vibration in order to feel music with a not really involved sense until today: touch.

The musician or more probably the dj, controls the wearable pods with a specific mixer following music rhythm.

Moving to other different topics, a “heartbeat sound amplifier” not designed for medical environments, is a totally new opportunity which need to take care of orienting the user during the most critical phases of interaction. First of all the user would need to be secure that the technology is going to be in contact with is totally safe for his body; this because nowadays the fear of hidden problems into the new relationship between body and technology is always in the 83

are always sourranded by thousands of different stimuli, that’s why, for example, live musicians wear very frequently headphones in order to be isolated an listen just to what they need to listen to. The performer should know all the time if the amplifier is work­ ing or not, this could be easily executed for example by a pulsating led light; and what if that led light would pulsating following heartbeat rhythm? Similar application are developed even with arduino and a heart rate sensor which is what can easily rule the LED pulse and it is also very miniaturized and easy to integrate into the future device platform (an example of a system scheme by Adafruit project is showed in the picture below).

An overlook on the project The Touch shows the most contemporary level of development on this possible feature. «HB Ring is one of our first products and first ring in the world that lets you see and feel the real time heartbeat of your loved one directly on your ring» (The Touch 2017).

mind of the users; it’s enough to think about the recent studies related to the damage of smartphone radiations, too much in contact with certain parts of our body.

In matter of interaction, the user will surely need to understande easily how to position the device and how to turn it on. Basically it’s a matter of the proper level of affordance into the technology-heart joint; moreover it would be interesting to study the interaction for turn the device on: heartbeat sound is not like every other sound produced by musical instrument or voice. For example, when a microphone is connected to a mixer is usually immediately active, but the volume is down and none is singing or playing anything; in the case of heartbeat sound, the sound is always there and it can’t be surely stopped, or that would be a serious problem. In general, the user should be easily able to interact with listening or not listening, with the high or low volume of a sound which is always there.

A constant dialog is also important for the interaction process. Performers

Considering the workflow would be also an essential task: the performer will of-

figure 3.12 - lighiting project with Pulse Sensor and Arduino setup

84

already described by different points of view in paragraph 3.2 and 3.3. Echo Doppler technology usually works with the help of a gel applied on the area of auscultation.

Since this is mostly unacceptable for the totality of the potential users, this interaction process as to be radically changed. The gel has actually two functions: –– reducing the skin rubbing during best point of auscultation research (with related background noises) –– improving general sound quality (again related to accidental rubbing background noises related to micro movements during auscultation)

figure 3.13 - The Touch

ten need to turn on and off smoothly the sound amplification. This because we are talking about a particular rhythmic sound, always with the same timbre and the same identity. Exactly like it happen during a music performance powered by loop station, the musician loops simple sounds until they satisfy the desired sound richness; after and during this process, in order to create more dynamism in the composition, he stops and play later some parts. This would basically happen with the heartbeat sound: it hypothetically can’t stay playing all the time during the full performance because it would be almost like listening to always the same rhythmic scheme and sound by a drummer for 5 minutes. The drummer stops, change rhythm, go faster, slower, higher or lower in volume; this has to happen also to the amplified heartbeat sound. In addition, because of the uncommon device-body position, this interaction couldn’t just be solved by standard buttons: gestures would be the best way to rule heartbeat sound, if not yet in prototyping phase, surely in final product shaping and evo­lution.

Testing the Angelsounds® technology by Jum­ per a specific conclusion came up: when the probe is stable on a specific point of auscultation the presence or absence of the gel doesn’t really influence the quality of the final audio output. This means that the gel is mostly necessary to prevent skin rubbing and rubbing background noises in delicate subjects like pregnant woman. If we consider the background noise problem many audio technology could be designed in order to moving down or up the volume just when a certain sound is catched; in this case the heartbeat sound frequency. Moreover for the skin rubbing, the problem doesn’t exist in the moment the device is fixed on a certain point.

The only critycal interaction is basically on the first use: the user is asked to reasearch and select his favourite point of auscultation; this task could be per­ formed with the help of the traditional gel on the first interaction with the object, the so-called primary setup, or with the help of a smart material covering the probe temporarily or permanently. In the

Regarding the echo Doppler technology, an interaction aspect is strictly related to the employment of this technology. As it was 85

of definition, until realizing the final, always temporary, solution. Many users are already available for a test, a functional prototype, maybe not perfect in details and finishing, but remarkable in functions, will be the best strategy in order to understand if the concept is going on the right way; 3. taking advantage of the 10 years experience of Offrimi il cuore case stu­dy: as it was already explained, this work at­­tracted the attention of an international project. Its creator was available already from the conceptual phase, for sharing his knowledge and experience related to the topic. His stories, his mistakes and his advices have to be considered a precious source for the whole project development. At the end, the project could even not fit to every need this case study specifically has, but the possibility to test directly in such a related and valuable case study, is not just an option but an indispensable op­portunity.

case of the second, most innovative option, the specific requirements of this material have to be defined in details during the project development phase. The full interaction with the technology will be strongly influenced by this choice.

3.6 - final brief: live heart­ beat sound Considering all the considerations achieved until this moment, a final project brief could be described extensively in the following sta­ tement: “A device able to live amplify real and personal users heartbeat sound, which can be used both for live performances and for private listening” In addition, an objective has been established and some processes are planned in order to pursuit the goal: 1. analisys and improvement of some already existing technology for dif­ ferent applications. The gap founded on the market isn’t that much related to strictly technological challenges but in interaction and new targeting issue.

Nowadays we have almost all the necessary technology for he­art­beat auscultation and amplification, the problem is that basically, over the 95% of its development is in medical applications; 2. test of a functional prototype in or­ der to collect feedbacks for further de­velopments. This is nothing so innovative: when something doesn’t exist, we have to create it with different layers

The scheme on the following page is a summary of all the considerations included in this concept design chapter. It’s the final brief and it highlights the most important characteristics the future project development will have to mind about in matter of targeting and requirements.

86

brief

figure 3.14 - final brief

HOW

GOAL

A device able to amplify real and personal users heartbeat sound

analisys and improvement of some already existing technology for different applications test of a functional prototype in order to collect feedbacks for further developments taking advantage of the 10 years experience of “Offrimi il cuore� case study

most interested art forms: music, dance, acting, sculpture, circus, creative professions (considering the distinction in between live performances and training sessions) sportive people: heartbeat listening during workout as a ruling rhythm and a deeper body activity check people used to meditation or relaxing activities: heartbeat listening in order to focusing more and deeper into the activity children educational activities for a higher body knowledge and intimate experiences

TARGET art sport meditation education

curious people interested about the advantages of heartbeat listening patients with cardiac deseases, more able to check and understand their health level

WHO

focus on interesting comparison between music genres bpm and heartbeat bpm for potential shuffle playlist based on personal heart rate people heartbeat sound archive for multiple purpose (medical, cultural, psychological, social) experimental higher engagement for virtual communications

WHAT

REQUIREMENTS

wearable (in direct skin contact, allowing hands freedom and being portable for 3-4 hours) adaptable both for listening and amplification activities light weight (under 85g, battery included) small dimensions (limits 70x70x15mm, excluded a possible external wearable system) possible wireless audio transmission volume/ultrasound intensity controller 1 small jack audio output battery power source desktop/mobile software independent not invasive and flat probe causing less possible skin rubbing elimination of the gel requirement in echo Doppler technology reduction or semplification of the auscultation point research process able to resist to fast and significant body movement (dance or acrobatics, for example) possible sound slowing interaction focus on sound quality

87

sources

3.2 - the technology behind auscultation

3.Concept design

Mario J. Mc Loughlin, Santiago Mc Loughlin Cardiac auscultation: Preliminary findings of a pilot study using continuous Wave Doppler and comparison with classic auscultation [online, article] http://www.internationaljournalofcardiology. com/article/S0167-5273(12)01346-0/fulltext

3.1 - a problem to solve: heartbeat sound amplifi­­ca­tion out of medical appli­ca­tion

Neramitr Chirakanphaisarn, Thadsanee Thongkanluang, Yuwathida Chiwpreechar Heart rate measurement and electrical pulse signal analysis for subjects span of 20–80 years [online, article] http://www.sciencedirect.com/science/article/ pii/S2314717216300265

16 songs with a heartbeat [online, website] http://www.songfacts.com/blog/writing/16_ songs_with_a_heartbeat/ Gianella Demuro Offrimi il cuore Nero Project, Antonello Fresu, 2012

Tarun Agarwal Heartbeat Sensor – Working & Application [online, article] https://www.elprocus.com/heartbeat-sensor-working-application/

Helen Thomson, NEW SCIENTIST Listening to your heartbeat helps you read other people’s minds [online, article] https://www.newscientist.com/article/­ 2129367-listening-to-your-heartbeat-helpsyou-read-other-peoples-minds/

Stethoscope [online, article] https://en.wikipedia.org/wiki/Stethoscope Liat Clark Meet the doctor bringing cheap, 3D printed medical devices to Gaza [online, article] http://www.wired.co.uk/article/3d-printed-stethoscope-gaza

Heartbeat sound listening [online, website] https://www.youtube.com/results?q=heartbeat+sound+listening&sp=SABQFOoDAA%253D%253D

88

Kashmira Gander Gaza doctor Tarek Loubani creates 3D printed stethoscopes to alleviate medical supply shortages caused by blockade [online, article] http://www.independent.co.uk/news/world/ middle-east/gaza-doctor-tarek-loubani-creates-3d-printed-stethoscopes-to-alleviate-medical-supply-shortages-10495512. html

S.M. Debbal, F.Bereski-Reguig Frequency analysis of the heartbeat sounds [online, article] https://www.researchgate.net/publication/228412447_Frequency_analysis_of_ the_heartbeat_sounds Active Signal Technologies [online, website] http://www.activesignaltech.com/Stethoscope2.html

Andy Boxall 15 Year-old creates a 3D printed iPhonecase that’s also a stethoscope [online, article] https://www.digitaltrends.com/mobile/stethio-case-turns-your-iphone-into-a-stethoscope/

One Digital Stethoscope [online, website] http://www.thinklabs.com/one-digital-stethoscope

3.3 - the possible heart­ beat sound

Houtsma, A. J., Curry, I. P., Sewell, J. M., & Bernhard, W. N. A dual-mode noise-immune stethoscope for use in noisy vehicles ARMY AEROMEDICAL RESEARCH LAB FORT RUCKER AL. [online, article] http://www.dtic.mil/docs/citations/ADA481444

Farshad Arvin, Shyamala Doraisamy, and Ehsan Safar Khorasani Frequency shifting approach towards textual transcription of heartbeat sounds [online, article] https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC3396354/

Bridger, K., Cooke, A. V., Kohlhafer, D. J., Lutian, J. J., Sewell, J. M., & Strite, R. E. U.S. Patent No. 8,265,291. Washington, DC: U.S. Patent and Trademark Office. [online, patent] https://www.google.com/patents/US8265291

Phua K., Chen J., Dat T., Shue L. Heart sound as a biometric Pattern Recognition, 2008 Arvin F, Doraisamy S. Real-time segmentation of heart sound pattern with amplitude reconstruction IEEE EMBS Conference on Biomedical Engineering and Sciences, 2009

David A. Van Alstine Electronic Stethoscope [online, patent] https://www.google.com/patents/ US20140254814#backward-citations 89

Leading 10 causes of death in low-income countries worldwide in 2015 (in deaths per 100,000 population) [online, website] https://www.statista.com/statistics/311934/ top-ten-causes-of-death-in-low-incomecountries/

Heart sounds [online, article] https://en.wikipedia.org/wiki/Heart_sounds Jiang Z., Choi S. A cardiac sound characteristic waveform method for in-home heart disorder monitoring with electric stethoscope Expert Systems with Applications. 2006

Bechara, A., & Naqvi, N. Listening to your heart: interoceptive awareness as a gateway to feeling. Nature neuroscience, 2004, 7(2), 102-103. [online, article] https://www.nature.com/neuro/journal/v7/n2/ full/nn0204-102.html

Khorasani E, Doraisamy S, Arvin F. An Approach for Heartbeat Sound Transcription International Conference on Computer Technology and Development, IEEE. 2009

Liu, Hao, Jun Hu, and Matthias Rauterberg Music playlist recommendation based on user heartbeat and music preference Computer Technology and Development, 2009.

ICCTD’09. International Conference on. Vol. 1.

IEEE, 2009. [online, article] http://ieeexplore.ieee.org/abstract/document/5360027/?reload=true

Heartbeat sounds [online, website] https://www.soundsnap.com/tags/heartbeat

3.4 - who could be intere­ sted in heartbeat sound?

Gianella Demuro Offrimi il cuore Nero Project, Antonello Fresu, 2012

Dunn, Barnaby D., et al.

Listening to your heart: how interoception shapes emotion experience and intuitive decision making Psychological science, 2010. [online, article] http://journals.sagepub.com/doi/pdf/­ 10.1177/0956797610389191

Distribution of the 10 leading causes of death in the United States in 2015 [online, website] https://www.statista.com/statistics/248619/ leading-causes-of-death-in-the-us/

90

Fukumoto, Makoto, and Junichi Imai Evolutionary computation system for musical composition using listener’s heartbeat information IEEJ Transactions on Electrical and Electronic Engineering 3.6, 2008: 629-631. [online, article] http://onlinelibrary.wiley.com/doi/10.1002/ tee.20324/full

Leonid Miakotko The impact of smartphones and mobile devices on human health and life [online, article] http://www.nyu.edu/classes/keefer/waoe/miakotkol.pdf Breadboarding the circuit [online, website] https://learn.adafruit.com/pulse-sensor-displayed-with-neopixels/breadboarding-the-circuit

Marie Tricaud [online, website] http://www.marietricaud.com/touche.html Wiens, Stefan, Elizabeth S. Mezzacappa, and Edward S. Katkin.

Heartbeat detection and the experience of emotions Cognition & Emotion 14.3 (2000): 417-427. [online, article] http://www.tandfonline.com/doi/abs/­ 10.­1080/026999300378905

3.5 - possible heartbeat interaction Christian Vasile UI Principles for Great Interaction Design [online, article] https://designmodo.com/ui-principles-interaction-design/ The Touch [online, website] http://www.thetouchx.com/

91

4.Project development 92

4.1 - system requi­re­­ments: heart sound system performances In the previous chapter a final brief was de­ fined: “A device able to live amplify real and personal users heartbeat sound, which can be used both for live performances and for private listening” Once the goal is settled, solutions could be multiple. This paragraph aims to show the process of considering every potential solution until the definition of the final concept.

rubber band

The fisrt step is to develope on a deeper level the requirements: –– wearable (in direct skin contact, allowing hands freedom and being portable for 3-4 hours) How many ways this device could be wearable? the following sketches show some possible ideas. The issue is quite relevant be­cause it

t-shirt

3 points band

necklace 93

MOODBOARD

wearable technology heart contact related

94

95

figure 4.1 - wearable technology heart contact related moodboard

miniaturized as possible. The user will wear all the technology in the same part of the body, minimum weight and size are really relevant.

2. the device is composed by two main elements: the sensor and the signal /audio processor. The sensor is the wearable part and it could be really small; the signal processor, can be bigger and wearable is not a mandatory feature.

There are advantages and disadvantages behind this choice, and just prototyping the concept could probably provide the final answer. If the compact device is more communicative and easy to handle, it could

tappet

figure 4.2 - shaping system of a compact device

clip (big headphones concept) represents on of the the highest challenge and innovation task of the full project. A moodboard for technology on skin is also showed at the next pages.

But this issue is related to a focus which is now relevant to elaborate. The focus is also strictly related to technology; in fact in the meanwhile we start to think about wearable, technology miniaturisation is the key factor for this kind of technology.

figure 4.3 - shaping system of a multiple components device

In particular we are in front of two options: 1. all-in-one compact device, as much 96

cost a lot more for the miniaturization required and it could anyway keep a considerable weight, which of course would complicate the wearable requirement until unpredictable complexity. From the other side, the splitted device gains a lot in wearable and it keeps low price and general low design complexity; the disadvantage of it, is a possible complexity experienced by the user that could be involved in pairing/connecting activity between dif­fe­ rent components. During the concept phase, sketches will consider the first option in order to reduce complexity in understanding.

After all, the second option is just split of the first one: the scheme at the next page shows visually this concept.

figure 4.4 - system assembly options

option 1

–– adaptable both for listening and amplification activities

option 2

bluetooth/small jack audio output This requirement has a simple solution: the device will have an integrated bluetooth audio signal for listeners, and a small jack audio output on the external shell; in live performances amplification and recording ac­ti­vities cables are still largely emploied, because of their higher quality and reliability.

97

This solution has also flexibility, because user could listen by jack audio output if they have specific headphones or audio system, meanwhile performers could choose bluetooth audio signal if they are going to be very dynamic and cable around their body could impede their movements. Nowadays the bluetooth audio signal is becoming the standard, especially taking a look on smart­ phones or cars audio system evolution. In li­ve music performances, if the bluetooth tech­ nology is available, few users really go for it in professional applications, mostly because of reliability, but somehow, also by the attraction of the cables, in this market.

device could be designed to be rechargeable with an integrated, not easy accessible lithium battery. This would surely increase the final cost, but it is also a way to follow the trend of smart devices and adding quality to the product appereance and interaction. Following this way of thinking a contact wireless charging would be the best option. About the autonomy, dimensions are more important. It wouldn’t be such a problem for the user to go under charging after something like 15 hours of interaction with the device. This would be an other dimensioning parameter into the final battery selection.

–– small dimensions (limits 70x70x15mm, excluded a possible external wearable system)

–– light weight (under 85g, battery included)

rechargeable battery

dimensions This in the case of a compact device; a bigger one wouldn’t be acceptable for wearable issue.

Those limits were settled considering the standard general size of the whole technology re­quired. At this level of development it is still an open question how the technology would be arranged inside, how dimensions could be minimized to their limit and how the final

It’s quite common the battery cover the most of the final device weight. Considering a device like Angelsounds® the choice of a 9 volt battery is surely a cheap and acceptable solution for a device designed to be handled.

But in the case of a wearable device, the battery size and weight reduction is a key factor for the entire product design. In particular, the 98

shape will influence the final sizes. A cylinder is for the moment the most attractive and acceptable solution; just going deeper in technical details would realize its feasibility.

In addition, because the highest percentage of the technology is electronics, it’s worth to mention that dimensions, in this case, are directly proportional to budjet and final wished production cost.

Miniaturization in this field it’s a matter of budget, of available components on the market and continuos tech­nological de­ve­ lopment; that’s why it’s re­asonable to say that the project should fol­low the standard quality/price good ratio. In other words, mi­ niaturization is a relevant task, but not the reason to double or triple the pro­­du­ ction cost.

Why a possible wearable system is exluded from the sizing requirement? because it would be external to the technology and in order to be really comfortable and functional, it could be quite relevant in size, without having any relationship with the device itself.

Moreover, in the case of a separation between the sensor and sound processing board, the sensor could still keep the limits 70x70x15.

bluetooth audio pairing interaction under pairing mode or not. Unless a different from the standard interaction with pairing would be designed, a pairing button, maybe with other functions, and a different led light behaviour under pairing mode will be present into the device. In general some standard choices for a quite new technology could be a way to increase user-friendly interaction, because users could feel more comfortable in finding something they already know inside a totally new device. Like it already happens in many devices on the market the presence of a button could include many different functions for the device itself.

Different level or number of pressure designed for a single button could drive most of the basic device interactivity; this considering that a touch display would be useless and there are still many people loving buttons.

In the case of a separation between the sensor and sound processing board, this feature will be surely part of the board.

–– possible wireless audio transmission This requirement was already analyzed inside the “adaptable” issue. In details, wireless audio transmission should be considered as an option but not as an obligation, or some specific target would be limited without any advantage.

Wireless audio transmission bring within itself usually a pairing button, especially in the most common bluetooth technology, where usually a different led light behaviour helps the user to understand whether the device is 99

–– volume/ultrasound intensity controller

the moment the most effective interaction with volume is supposed to be inspired by the normal volume control a user has with many high quality stereos or amplifiers, that is a wheel volume controller. Anyway the automation of the volume in starting heartbeat sound research phase is a matter of coding a process into the audio bo­ard, it wouldn’t actually change so much the final product appereance, it would be just a smarter function.

–– 1 small jack audio output

volume controller This feature is really important for the starting heartbeat sound research task. In fact, during this activity, if the device would be at its maximum volume and the sensor would based on ultrasound, background noise mostly coming from skin rubbing would be very annoying. Now the solution could be double: 1. the device set automatically the appro­ priate volume level when the hearbeat sound is detected 2. the user control the intensity during the whole interaction process There are advantages and disadvantages; so­ me technical tests should be performed for a final choice in between those two options.

What we could set for the moment is that a volume controller is mandatory, and maybe the solution would be an hybrid of the two options.

It is mandatory because not every listening or amplification device has an integrated vo­ lume controller, so since we are not able to be sure about the user owning for example headphones with volume controller, the device requires this feature as an integrated one. As it was already described in the paragraph 3.5, for

small jack audio output positioning As it was already mentioned in “adaptable” requirement, flexibility for multiple categories of users is really important. The simple small jack audio output has just an issue, the best possible positioning around the external shell in order to don’t impede user movements.

In the case of a separation between the sensor and sound processing board, this feature will surely be part of the board, meanwhile an other cable with same issue will connect the sensor with the audio board.

100

–– battery power source

it from the huge amount of recent new born devices strictly related to desktop/mobile app interfaces. This doesn’t mean that an app for smartphone in order to interact with the device it couldn’t be designed: the user has just to be free to choose! He has to be able to use the device without the app, since the app will provide just secondary and additional features, maybe for a certain kind of targets, interested into some specific interactions (the speeding/ slowing heartbeat sound control is the best example for this). In other words, the device wish to be perceived has a sensor and a controller, meanwhile the eventual related app, would be a development platform, pro­posing deeper interaction, engagement and technology exploitation.

A similar interaction is actually easy to be observed in most of the fitness smart bracelet interaction: they have display and they are autonomous. Smartphone apps are where they collect data and leave the user free to analyze them on a deeper level, just if he wants it.

no cables for energy supply No permanent cables for energy supply. The device would be energetically autonomous for a certain amount of hours. For a first divice settlement, an autonomy of 5-6 hours is enough for the kind of interaction required by the technology.

–– desktop/mobile software independent

–– not invasive and flat probe causing less possible skin rubbing

app independance This requirement it will provide a specific personality to the device and it will distinguish

not invasive and flat probe 101

What in medical language is called the probe is actually the component containing the ultrasound diffusors; the probe is what goes in direct contact with human body. Considering what the previous requirement settled until now, there wouldn’t be such a distinction in between the probe and the rest of the device.

The probe, let’s say, is the “macrosensor”, and like it happens for example in fitness smart bracelets, sensors are integrated into the whole product itself.

In order to don’t cause skin rubbing, this element would be basically composed by a soft external surface and a harder internal one, containing the ultrasound diffusors. A specific attenction has to be a material requirement for the soft part: it doesn’t have to influence the ultrasounds waves transmission.

The flatness is related to the idea the device is not anymore handled, so has much it would be thin and attached to human chest, the more it will be not invasive for the user.

This requirement is strictly related to the previous one. The ambition of eliminating the gel requirement in echo Doppler technology is a mandatory task in the case of ultrasound technology employment. How a wearable system could fit to the need of gel application before wearing? It can’t, so the general idea for a concrete solution to this problem is relied on the soft material covering the flat probe.

Thanks to nano­technology or just smart material se­le­ction, it would be possible to find a material really fitting the task, maybe inno­vating an entire market. In ad­di­tion the gel is mostly used for pregnant woman: so this specific target is of course, very delicate and it requires a higher level of attention. But in the case of different target, it is reasonable to adfirm that as soon as the probe would be just able to smoothly flow on skin and provide the correct transmission of the ultrasound waves, the issue would be immediately solved.

If this idea would fail after technical trials, the ”b” plan is considering the whole inte­ ra­ction process. In fact, some test with the An­gelsounds® already confirmed the gel is just really useful during the heartbeat sound research phase; once the point is founded (and usually it would be always the same for the entire product life), the gel it doesn’t really affect the sound quality anymore, if the device is stable on a certain chest point; this would ask also for a more intense study of the wearability of the entire device.

Since it is still an option to consider an other technology in order to capture the sound, it is not sure a soft probe it would be mandatory.

In fact, “in the case of an accelerometer sensor, this would stay in direct contact with the skin. The issue would be just holding it in the point of auscultation, eventual skin

–– elimination of the gel requirement in echo Doppler technology

soft probe 102

rubbing or general undesired sounds could be later excluded by a low-bass filter” (Acoustic department, INRIM, Turin, 2017) like it was anticipated in paragraph 3.3.

–– reduction or semplification of the au­ scultation point research process

wearable extra accessory issue and in particular to the question: wearable for who? The device would surely attract more different users with such a performance. In particular the solution could be standard, if it would be always integrated with the device, or optional, if it would be an extra accessory, as it was anticipated before.

auscultation point research This is probably on of the most critycal inte­ raction aspect. As it was already highlighted in paragraph 3.3, the research of the best auscultation point is quite demanding for the users. Also in a simple product like An­ gel­­sounds® this phase is critycal, that’s why they write everywhere «don’t worry if you don’t ear anything, you’re just not searching in the right place». It’s quite easy to understand why there is also a psycological issue into this requirement: with heartbeat we are talking about one of the first life signal, if your heartbeat is silent you are dead!

The idea of an extra and more efficient accessory is more on the wave of a relevant product like the recent action cam technology.

The GoPro® market strategy for example, is about offering a high and continuosly updated quality sensor, which is the core business of the entire company and in addition a relevant amount of accessories providing multiple uses of the same sensor. Different users are happy to choose the perfect accessory designed for their need. Possible targets are too many for even thinking about designing an all-in-one solution.

–– able to resist to fast and significant body movement (dance or acrobatics, for example)

–– possible sound slowing and/or speed­ ing interaction

The performance is strictly related to wearable

Onto this level of development this feature is still under research. But in the case it 103

–– focus on sound quality

would be finally included into the device, the interaction would surely rule its configuration.

Since we are probably talking about an op­tional interaction, the possibility of leaving the basic sensor free of this feature it’s a reasonable choise; in fact, thanks to an app, this feature could be easily performed by a smartphone touch display without charging the device of unwished complexity. This basically happens into a considerable amount of new contemporary smart devices.

Moreover an app could also increase the user interaction possibilities; in other words, he could be free to choose in between direct device interaction or indirect app interaction with the technology.

By the way, in the case of an integration, the pairing button or the volume controller, already included in the external shell, would be surely considered in order to include this function, as a secondary one.

It’s just enough to specify it’s a key factor.

Since the sound is digital, the task would prabably and hopefully be a lot related to coding and digital sound processing in ge­ neral, but if some components contained onto the audio board would ask for more room in order to improve sound quality, the full device would be immediately reframed in order to pursuit this goal. Higher sound quality by this new device is considered an other relevant key factor for innovation in this technology development. The contemporary sound quality of a device like the An­gel­­sounds® is quite low.

4.2 - system tech­nology: suitable solution for multiple users The main heart auscultation technology available on the market were highlighted in the previous chapter. Now the objective is to understand which one could fit to the concept requirements and what could go even further, from an innovation point of view in heartbeat auscultation technology.

This study had the great advantage to get in contact with the acoustic department of INRIM (national institute of metrologic research) in Torino, Italy. The amount of knowledge acquired there is so relevant that there wouldn’t be enough space inside the context of this work, but it will surely be a great source of informations for the further project development. The engineer Pavoni Belli is a researcher at the INRIM acoustic department and the expert in charge of following the work;

app additional controller

104

figure 4.5 - Andrea Pavoni Belli managing with acoustic departments equipment at INRIM institute

figure 4.6 - analysis of different technological options at INRIM institute

He was also an important professor at APM school in Saluzzo and he accidentally already experienced heart sound amplification during his life with a very interesting personal story related to the topic he is more than happy to tell it to every inquiring person.

the stethoscope technology; every aspect of this research was shared and merged with the experience of professor Pavoni Belli, until the definition of a further level of knowledge in heart sound auscultation technology.

In general, the collaboration with INRIM revealed immediately some critical elements:

In INRIM we started from the analysis of the An足gel足足sounds速 device until the observation of 105

- size vs cost - cost vs technology - application vs technology

that other solutions are possible but they require investment and research definitely out of the topic of this work; from the other side, the ultrasound technology is definitely the most performative solution for a starting development: it’s a quite affordable solution and with the right approach to the wearable issue, it could be really efficient, even for quite dynamic users.

All those contrasts (also related to the project requirements) were been a precious filter for excellent unaffordable options. Since INRIM is a metrologic institute, it emploies a huge amount of highly professional technologies unsuitable for normal consumer market. So we started very soon minding also about the market and the hypothetical production costs, because there would have been any advantage in considering very expensive technology for a possible prototype, discovering later that it was going to be later impossible to make that technology affordable for mass-production.

The quality of sound is related to some technical parameters like the amount of hertz coming out from the probe or the way how the return signal is processed by different board.

An additional sound EQ to the standard output of this technology could improve further the desired sound quality. So Doppler effect would be the core of the system, paired with an additional sound EQ by coding; now the issue is how to update this technology and drive it to the highest possible perfomances for the requirements we need. The mentioned right EQ value where discovered at INRIM thanks to SoundForge, where we analyzed a track recorded by a fetal doppler device called “UnbornHeart�, particular valuable for its small dimensions and performances.

The next schemes at the following pages are the visual result of several meetings at INRIM institute; during those meeting all the possible technological solutions for the issue were analyzed considering every critical aspect until the definition of four possible technical approach to the problem. We moved a lot further from the ultrasound technology, which became just one of the options. But in the end, the test of other technologies demonstrated

figure 4.7 - heartbeat audio EQ in order to reduce background noises

106

prototyping

ULTRASOUND OPTION

Just an electronic engineer with some knowledge in sound could really manage with this ultrasound technology. The prototyping would be about disassembly the fetal doppler device and apply it to a new external configuration with additional functions. bli nk

ing )

It completely etely adopt a the system of an already existin existing technology. The innovation on iss about a making it wearable,e, for or different d target and with a more interesting nteres res interaction.

The heartt wouldn’t l sound very natural, audio quality could be improved with audio live processing by filters.

lig ht (h vo

lum

ec

on tr o l

ler

+o

n/o

ff

2 (3, ultra 3M so Hz) un CW d di Do ffus ppl or er s Ă˜ 10m m pro ce ss or2

1j

RG BL

ac

ED

ka

ud io

it will require a cover and it has to be firmly fixed to chest in order to work

ear

inp ut

tbe

at

arduino or similar

OFF

ON

au dio eto oth

blu

130g

lit 34x hium 62x re 5m cha m rg

custom audioboard managing with ultrasound signal and amplification

ab le

pro

ba tte ry

70x ces 70m sor m 1

mo du

le

5g

30g 107

technology total weight about

165g

2

prototyping

ACCELEROMETRIC OPTION

An electronic engineer with some knowledge in sound could manage faster and better with it. In few time the prototyping risk failure because it requires different knowledges and higher budjet.

uls lig ht (p

ud io

ED

ka

RG BL

ac 1j

m

ff n/o +o ler on tro l

or

ec

ss

lum

ce

vo

pro

M ana EMS log acc inp ele ut/ rom low e g ter

10x

10m

it will require a cover and it has to be firmly fixed to chest in order to work

inp ut

arduino or similar

ing

wit h

hea

rtb

eat

)

The most st interes interesting and the more ated. The accelerometer can complicated. eartb vibration on the catch thee heartbe heartbeat thout hout fee chest without feedback problems; The system requires aamplifier and proper i filt frequencies filters. This application for accellerometer is unexplored so it requires research, time and experiments. The heart sound quality could be high and accurate.

OFF

ON

au d eto oth

blu

130g

lit 34x hium 62x re 5m cha m rg

ab le

ba tte ry

am pli

io

fie r

mo d

ule

5g

30g 108

technology total weight about

165g

109

prototyping

ACOUSTIC OPTION

The task is about bulding a wearable stethoscope with innovative insulation proporties for unwished background noises. Sound output will be easier to manage.

uls

ing

inp ut

lig ht (p

ss

me

or

co ntr ol

ler

+o

n/o

ff

RG

BL

ED

ac 1j

arduino mini or similar

vo lu

pro ce

ac ele oust ctr ic et iso mi la cro ted pho s ne teth os c

op

et

ec

hn olo

ka ud io

gy

it will require a cover and it has to be firmly fixed to chest in order to work

wit h

hea

rtb

eat

)

idea of merging tthe stethoscope technology with the acoustic insulation technology. The acoustic system stem is completly passive and it employs a quite te sstandard microphone technology. ogy. The wweight on the chest could be a relevant nt issue. issue Heart sound quality could be the best possible.

OFF

ON

65g

lith

34x ium 62x re 5m cha m rg a blu ble b eto a oth ttery au dio mo du

le

100?g

30g technology total weight about 110

195g

111

4

prototyping

DOUBLE DEVICE OPTION

this option can’t be prototyped in the same way because it requires tipycal industrial electronic solutions unavailable for the current project development. The prototype would have a totally different configuration; it could produce similar output but with different interaction

(pu ht lig ED

m

BL

0m

OFF

ON

lith

ium

lith

rec ha

ium

rga

ble

rec ha

rga

ble

ba tte ry

ba

tte r

y

Wi

Fi

vo lum ec on tro ll

er+

tra ns mi ss

on

ion

/of

f

RG

2 (3, ultra 3M so Hz) un CW d di Do ffus ppl or er s Ă˜ 1

it will require a cover and it has to be firmly fixed to chest in order to work

lsin

gw ith

hea

rtb

eat

)

the sensor sor on bbod body includes a board in order orm m wir wirele to perform wireless technology; miniaturizaeree will be b the rule. tion there er device re An other receives the signal and it is the center ter of the sound transmission to other ti sources and the interactive amplification object. It could still be a wearable or it can also become something like a control station near the user.

?g ?g

technology total weight about 112

210g

113

figure 4.8 - heartbeat audio EQ in order to reduce background noises (20 band)

option and sound quality would be even higher.

The final choice is also related to size, in fact an EQ pedal, even the most miniaturized one, would be anyway quite bulky. Below an example of an EQ pedal specific for bass, with low frequences management (this needs make it even bigger). Moreover it is hard to obtain the same achievable EQ you can get by a professional audio software. But simulating an EQ digitally could be quite complicated without a strong coding and programming background, that’s why the EQ real pedal is stil an option for the working prototype.

The EQ filter will be in between the ultrasound board and the general device board. A digital version of it charged into tho code of audio processing is an option, but for the prototype, the idea of replacing it phisically with the technology of an EQ pedal is also a valid figure 4.9 - example of a EQ pedal usually developed for bass

A detailed list of the essential components is now necessary for the further project development.

TECHNOLOGICAL AND INTERNAL COMPONENTS –– ultrasound diffusor –– ultrasound intensity controller (volume controller related) –– bluetooth audio system –– board for ultrasound output management –– board for the additional interaction and sound management –– lithium battery 114

INTERACTIVE AND EXTERIOR COMPONENTS –– external hard shell –– battery case and recharging system –– RGB led light system(heartbeat pulsation) –– soft material in direct contact with skin (covering the ultrasound diffusors) –– volume controller –– one small jack audio output –– bluetooth pairing button –– wearable system –– tap sensitive surface (on/off)

thinking about this concept of art into the project. It’s the idea of the heartbeat as background sound of various activities, of living and being in general; espress special abilities togheter with this emotional sound background, so acting and performing with it.

In addition HART has also a not very wellknown meaning of “a male deer, especially a red deer”(Cambridge Dictionary, 2017), which can be considered irrelevant or strange enough to be included as an element of curiosity, probably more for native english speakers.

4.4 - final concept:­HART, the heartbeat sound amplifier

“The heartbeat sound amplifier” is a sort of claim which sounds now quite explicit and efficient for the kind of object is going to be designed. If HART could be a cryptic name, from a communication strategy point of view, it is clever to reveal it immediately with a strong and understandable claim.

“HART” is an invented word. Phonetically it has a strong link with the word “heart” but as soon as you see it written somewhere you don’t link it immediately to that word and this is a desired effect; the link occurs during the pronunciation, when the mind hear that sound: this makes the name itself linked to sound.

There is an other interesting aspect about the spelling of the word: “ART” is included in the name not by chance. We know this project started with a primary focus on artistic applications, so keeping the word inside the name is a declaration of this first intent. But “art” is a quite generic word if we consider it with the meaning of “an activity through which people ex­ press particular ideas” or moreover “a skill or special ability”; even the archaic meaning of the second person singular of the present tense of “be” (Cambridge Dictionary, 2017) in an interesting way of

At the following pages some shaping and configuration approach are showed as different concepts. In the end the choice is also related to the kind of target that is going to be considered as the primary one.

Some specific approach to the shaping process are on the core of the following proposals: - required wire connection between sensor and board as an advantage for a functional element in wearable requirement (not just a connection cable) - hierarchical shape of two elements (one “mother” of the other, in other words, the board will be “mother” of the sensor) - flexibility in usage and wearable feature - emotional and minimal style in contrast - affordance, the device should tell by itself to the user how it works and especially how the user has to wear it.

115

116

117

118

119

the heartbeat sound amplifier It’s time to listen to your heart, in the true sense of the sentence. HART is the first wearable heartbeat sound controller.

Our heart produces the most relevant involuntary sound in our body, the HART concept is to deliver that sound to new unexplored applications such has amplification during artistic live perfomances or private listening for activities with the goal of a higher connection with our body.

APP for advanced experience

private listening

120

private listening

elasti

sensor

ic adjustable band BODY ADAPTABLE

join with a “click”

processor

battery

WEARABLE

PORTABLE

121

The device showed at page 120-121 is a temporary solution for the prototype.

It has to be considered the container of the HART technology and a rough preview of what its interaction could be.

Testing is the primary goal; going deeper in aestethical process is a waste of time at this phase of development, the idea is about the creation of a practical solution, very effective in performance, which in the future will be surely changed by aestethical process, also related to budget. Contemporary approach by size and solution in the current configuration are related to relatively small budget and possibility to interact manually with the technological system. Wearable design today is under continuos evolution, but there is quite a big gap in between what we imagine and what we can really handle from a low budget protoyping point of view. We want to test HART providing is final output with high fidelity with what will be the final massive product, that’s why now the appereance of this technology is just a functional container.

SETTING INTERACTION 0. Once the user unbox the system, he finds the device, the wearable system and a charging cable 1. the user wear the device the interaction is similar to a seat belt interaction. Phase of connection are two, from one side on the hands, from the other side directly on the chest.

2. the user connects his headphones to the jack audio output (or if his headphones are wireless, he interacts with the pairing bluetooth button in order to connect the devices). In the case of bluetooth audio transmission the user has to be aware of the lower quality audio performance of the system. moreover the user should know that headphones more performative with low frequences would be the best choice to pair with the device (an adavnced version of the technology could also provide specific headphones designed for heartbeat listening)

4.4 - system interaction: how do we get in touch with heart sound?

3. the user turns on the device (the device is on with a double tap on the sensor). Volume is automatic, by coding and in this phase it doesn’t go very high because it’s just a sound checking phase. As soon as there would be a good heartbeat sound capture, the volume will start to be higher. In the meanwhile, LED light start to blink fol­ lowing heartbeat rhythm.Blinking is coordinated with the good quality of the sound capture, so it’s also a reference for the perfect sensor positioning

At this level of development, it is now worth to talk about a storyboard of the interactions.

As it was already mentioned in previous ana­ ly­sis, goal interaction with this technology are basically two: amplification and li­ stening. But in order to achieve them the user needs to perform some actions before and after.

122

4. the user looks for his favourite point of auscultation, listening his own he­ art­beat by the headphones and moving the probe around his chest by sliding the belt on his bust.

during his heartbeat amplification and on the kind of headphones he’s useing).

During sound amplification activities, it is common to observe users choosing to listen to some specific sounds and being isolated from others. It is a good example the situation of the singer on a live stage: he usually wants to ear his own voice higher than everything else, than what the rest of the band is playing, in background. This kind of preferences could be very complicated and quite specific for every kind of user. In this case it’s just enough to let the “amplification user” free to choose if he wants or not listening to his heartbeat during his activity, the rest will be done by a mixer, that is what happen with all the rest of music instruments. In addition, this task could also be performed by a standard live stage audio system where specific headphones can send specific audio channels to specific musicians, this means everything could be wireless and everything is a bit out of the device performance.

The points 0 and 4 are basically just related to the first usage of the technology. In fact, once the user knows his favourite auscultation point, he won’t need to look for it anymore, he will immediately place the device in the right place. So the action 4 will be deleted in future standard interaction with the technology, unless the the user would like to try new positions. In that case the automatic volume adjustment with its LED light feedback will help him in correct point selection.

LISTENING (user group 1) 5. the user just keep listening his own heartbeat by the headphones he was using before. If he would be interested in changing the volume, some headphones usally have that controller integrated or the smartphone app would provide also that possibility. But in general the volume will be regulated automatically, by some factors like the good quality of the sound capture and/or the bpm rate.

If for example the user is running, the general noise around could disturb the activity, so automatically the device will be able to regulate is own volume by this BPM check. But it will be anyway always possible to disable all those automatic activities by the smartphone app.

6. the user connect the device to a mixer or what ever audio amplification sy­ stem by a jack cable (for higher sound quality). He has still the option of audio bluettoth transmission, but in the case of a live stage for example, this is rarely a real option because of quality and reliability issues.

7. the user can now perform his activity without taking care of the device.

Volume adjustments could be performed by someone else on the mixer, the only important thing is that the internal device volume is settled on a reasonable level (usually close to the maximum).

AMPLIFICATION (user group 2) 5. the user disconnect the headphones or keep listening to them (this depends on the activity he is going to perform 123

This will be performed automatically by the technology as a coding instruction.

If he is annoyed about the ligh effect, that can be also turned off by the app. In addition a different way of tapping could be planned in coding in order to perform the same activity.

is focused on. Anyway, a preview of it, will be showed in some concept screenshots further inside this paragraph. figure 4.10 - HART lighting interaction system

red light glowing following heartbeat sound possibility of changing the color by RGB setting by app

shade of red light sound visualizer effect

TURNING OFF INTERACTION One other double tap to the touch sensitive surface will be enough to end the interaction with the device; it is the same for turning it on, exactly like a on/ off switch button, but in a gesture. In addition the LED light will stop blinking, so the user will immediately know the device is off by a visual feedback and moreover by the sound off on his listening output.

possibility of multicolors effect by app setting

light off fixed red light white light temporary white smooth blinking

turning on

and heart sound searching

OPTIONAL SLOWING/SPEEDING HEART RATE BPM This is an optional activity, which doesn’t in­fluence the whole interaction process described before.

The activity would be based on the smartphone app and it will be mostly performed for “listening users” because for those interested just in amplification for artistic performances, the feature is not so interesting. Basically, after a smartphones blue­tooth pairing with the device, the user will be able to slow down or speed up his own heart rate. It is important to keep on display a frequency of his real heartbeat; in other words the user has to know constantly the difference between his real bpm and his fake one.

Other secondary functions and parameters could be con­trolled by the app.

The app design is not part of this research, because it is considered a further optional deve­lopment of the core technology this work

working mode

battery level 25%

charging mode

RECHARGING PHASE It is performed by a simple mini USB cable.

The USB door, could be hidden by a classic rubber cup out of the recharging phase, like it happens in many wearable devices.

The option of a wireless recharging station is attractive but more expensive; moreover 124

this kind of solution fits more to devices you use everyday, which is not the case of this technology at the moment.

The battery will guarantee 6 hours life, before charging need; by the way the device could also work connected to electricity if the user dispose of a long micro USB cable and its specific use of the devices require so much time of activity.

When the device is under charging mode, the light visual signal changes: it turns in blue colour and it indicates the level of charging until its end. The scheme at the next column is a preview of how this visual signal from the device could work.

figure 4.11 - HART safety aesthetics of interaction

REST PHASE When the device is not being used, it is important to keep it safe inside a cover.

This solution is mostly related to the need of protecting the technology from dust or dirt in general because they could affect the sensor performance. In addition, since it could happen very often this device it will be used out of domestic environment, this case has to provide portability to the device; it will become the way to transport it.

AESTHETICS OF INTERACTION Last but not least, what about the ae­sthe­ tics behind the whole interaction process?

Since this project starts from the idea of a totally new application for an already exhisting technology, it is worth to consider two main aesthetic aspect of its possible interaction: –– the old application (right column on this page) –– the hypothetical background of the new one (1st column at the next page)

minding about all the previous analysis related to target and interaction. Intuitive operations, minimalism and familiarity are here considered as advantages, but this has to be linked to a general idea of interacting with something totally new, smart and innovative.

Minding about some market historical case

This aesthetical hypothesis is considered 125

memorable, this doesn’t mean aesthetics should be familiar. It is in fact a risk to provide an aesthetic to the device which could be confused with an other technology; since the application is new, also the device should look innovative as a tool.

An other relevant aspect is that this tech­ nolo­gy is managing with an interior quite unexplored part of our body. Some users, the most anxious, could feel a little bit scared about placing this object on their chest and earing a sound they have never listen before in this way. This is how safety becomes a keyword influencing the aesthetics of interaction. The double-tap related to the on/off button and the sliding for the volume bring the user to interact with sound and touch with something which aims to become part of their body for a while. They don’t have to feel this device as something so external they are wearing; the feeling has to be closer to an extension, a temporary upgrade of this body activity: the heartbeat. In this way of thinking the double-tap produces a sound familiar with the heartbeat sound, and touch in general is related to body more than other senses; double-tap is also doctor activity for auscultation, anything safer?

Soft and smooth are keywords for the whole final selection of appereance and interactivity of the various components involved in the system: lighiting, capacitive surfaces and materials in direct contact with skin.

figure 4.12 - HART usability aesthetics of interaction

studies, as much a tech­nology looks already simple and familiar, so faster it will be easily accepted by its new users; in other words disruptive aesthetics and interactions are rarely efficient, especially for devices without familiar reference for the consumers. But if interaction should be intuitive and some how

The storyboards and the app concept at the following pages resume and point out all what was declared until now regarding the interaction with the future heartbeat sound controller.

126

figure 4.13 - HART system feedbacks and signal

INTERACTIVITY STORYBOARD interaction setting activity CONNECTING

interaction just first setting activity POSITIONING

1

0

the user looks for his favourite point of auscultation

the user connects the audio jack directly to the wearable device or to the phone after a bluetooth pairing. In the second case, audio quality would be lower.

device

sensor

APP

wearable system

5

LEGEND

interaction additional app heartbeat sound management

VISUAL SIGNAL

interaction

4 visual feedback

MAIN FUNCTIONS: - speeding/slowing (by sliding gesture on the touchscreen) - recording - sharing recorded track - battery level - bluetooth connection - setting instructions - light settings

3

6 interaction setting activity ON/OFF

7 interaction device removal 8 interaction recharging 127

x2

double tap with 2 fingers it is a gesture linked to the heartbeat by sound and it also reminds the doctor gesture on the body of the patient during standard auscultation methods

TOUCH/ACOUSTIC SIGNAL

VISUAL/TOUCH SIGNAL

interactivity

2 interaction wearing

VISUAL SIGNAL

HEADPHONES CONNECTION standard audio jack input

SETTING OPERATIONS

WHAT’S INSIDE THE BOX device, charging cable, cover and elastic belt

0

1a click 1

1st BELT FIXING ON DEVICE click interaction 2nd BELT FIXING ON BODY best under chest position

2 3

128

x2

TURNING ON double tap

figure 4.14 - HART storyboard

1b

click 2

ON

AMPLIFIER CONNECTION jack cable connection

8 a

x2

OFF

b

TURNING OFF double tap

129

LISTENING relaxing with heartbeat sound

OUTPUT OPERATIONS

7

SECONDARY

6

CHARGING APP CONNECTION PLAYING MUSIC A-microB USB cable wider interaction following heartbeat rhythm

LOOKING FOR THE BEST POSITION sliding the belt under the clothes

4 5

figure 4.15 - HART main app preview

130

figure 4.16 - HART app functions preview

131

4.5 - technical drawings: HART pre-engineered version

are just simulated in size and appereance; they are showed on a macro-level looking more on the relationships among each other than on what’s exactly inside them. General size of the whole system are calculated by electronic feasibility constraints, considering the potential of an industrial production, which consider a good quality/cost ratio.

The following drawings are the most approximated configuration of an hypothetical engineered product. This pre-engineering process is the best way to approach the complex amount of components involved in the system. It’s goal is to establish the guidelines in between the prototype and the final industrial product. Internal components

Below the elements composing the HART system (fig. 4.17). The next pages show technical drawings and rendered views of the pre-engineered version of HART.

INSIDE THE BOX

elastic chest band

charging cable MICRO USB TO USB TYPE A

device padded bag brings everywhere all the components keeping them in a safe place

figure 4.17 - HART system elements

132

figure 4.18 - HART dynamic system

89.0

°

90

90

°

32.0

TOP VIEW compact configuration

56.0

158.0

FRONT VIEW scale 1:2 dimensions in mm

133

5.0

2.5 16.0

TOP VIEW

25.0

3.0

66.5

3.0 13.0

25.0

6.5

57.0

R5.0

66.5

R8.0

R7.0

LEFT VIEW

134

FRONT VIEW

figure 4.19 - HART orthographic projections

42.7

1.0

R3.0

4.7

Ø6.0

Ø2.5

158.0

Ø6.0

16.6

Ø5.0

28.0

48.7

7.5

19.7

Ø4.5

scale 1:1 dimensions in mm

RIGHT VIEW

135

lighting surface

ISOMETRIC VIEW SW

processor case sensor case battery case

chest band joint

scale 1:1

136

figure 4.20 - HART axonometric projections

soft side in contact with skin

rotational hinge

ISOMETRIC VIEW NW

137

13.0

A

A

1.0

1.0 5.0

SECTION AA

4.0

2.0

4.0

6.0

76.0 6.0

5.0

DETAIL 1 scale 2:1 76.0

scale 1:1 dimensions in mm

DETAIL 1

138

B

figure 4.21 - HART sections

B 11.0

63.5

SECTION BB

3.0

DETAIL 2

25.0 63.5

3.0

Ø16.0

2.0

1.5

DETAIL 2 scale 2:1

139

EXPLODED VIEW

energy supply soft part

main cluster rear body main cluster soft part

sensor soft part CW Doppler 3,3 MHz ultrasound

screw 2x5

energy supply rear body buckles system

male connector band connector

elastic band screw 2x8

scale 1:2

140

figure 4.22 - HART exploded view

elastic band band connector male connector

buckles system rotation support bracket Doppler audio board full rotation pin

main cluster front body

main board

hollow rotation pin

sensor rear body

lithium battery

control circuit micro USB

light/sensors board

circular LEDs system energy supply front body

141

sensor front body

figure 4.23 - HART renderings

142

143

4.6 - a working prototype: HART first system setting

by ultrasound doppler technology required the selection of the perfect starting system from an already existing device.

A complete redesign of that technological system would be in fact out of the topic; on the beginning of this work, we tried in every way to understand if that technology could be re-generated as new by standard open source electronics, but the answer is no: there are no Arduino, Raspberry Pi or similar project developed, and the single components on the market are quite unavailable for small production. Moreover coding task should start totally from zero, with a quite high skills for electronics and programming. In the future it will be surely a primary starting point, probably for HART working prototype 2.0 or 3.0, but for the demonstration and the HART 1.0, a completely new assembly of that board would slow the design process until unacceptable levels. The selected device is the Unborn Heart™ fetal doppler device by Odosoft Ltd, an European Mobile Software Company from Oulu, Finland, that has developed the UnbornHeart™ Fetal Doppler mobile app for pregnant women.

We are interested on its audio signal, the rest will be removed and/or replaced by different elements.

Why a working prototype construction is a key goal for the development of this project?

Since this technology is totally new for the target group it has been designed for, a working prototype will be the best way to discover if all what was analyzed until this moment is able to get a coherent feedback by the real users.

Honestly, the economical budget is not enough to pursuit very miniaturized technology and the amount of technical knowledge (mostly electronic and sound engineering) required is something achievable just by a structured team composed by different experts. That’s why this first prototype will be now functional until the level of providing the sound experience to primary target group (artists). It will be wearable but maybe a bit uncomfortable for the wrong size of the internal component; it will be functional for the core features of the device, from an interactive point of view; it will be a study system able to be the right starting point for further engineering process and possible startup ecosystem. This general situation creates the need of two different approaches to the prototype assembly: there will be a demonstration prototype, build as a study model of a working prototype which will be the core that will be developed in different versions until it will be ready for a small industrial production. This working prototype as­ sem­­bly will start with the name of HART 1.0.

Since the signal provided by the Unborn Heart™ device is not acceptable for a live performance and the original plan was to modify it by a specific equalization we studied in INRIM institute, the temporary idea for the demonstration prototype assembly is to use a computer software in order to filter the audio signal. Building a dedicated EQ filter working like a standard pedal and integrated into the whole system will be the first step in between the demon­

Firt of all, performing heartbeat sound capture 144

figure 4.24 - Unborn Heart™ fetal doppler device by Odosoft Ltd

soon an higher level of similarity in comparison with the future industrial product output. A circular LED light strip will be programmed by Arduino IDE software in order to react with heartbeat sound as it was planned in paragraph 4.4 about interaction.

stration prototype and the HART 1.0; the solution now is expensive and it can be quite accurately replaced by a computer software.

Let’s talk about light interaction: this is fortunately the only thing able to reach 145

figure 4.25 - LED sound reactive system with TRINKET PRO

In order to get diffuse light, a material with specific opacity will be close to the light source, because the wished result is a con­­ tinous and smooth lighting effect.

If it wouldn’t be too difficult a different lighting effect will be coded for the turning on/off and the charging mode already in the

demonstration version, but surely for the HART 1.0. The use of a TrinketPro 5V by Adafruit will reduce a lot the general sizes of this part of the system. An Arduino Uno will be employed for the starting tests. figure 4.26 - desired lighiting effect

figure 4.27 - NeoPixel circles

146

figure 4.28 - PRO TRINKET 5V pinout

Regarding the power, the system of the de­ monstration prototype version is of course “quite plugged”. But the general idea is to exploit a standard battery pack like a classic power bank in order to power all the system for the HART 1.0. The issue is related to how that battery pack can simultaneously power the doppler device, the lighting system with its board related and the EQ filter.

The visual schemes at the following page show how the demonstration and the working prototype HART 1.0 systems are going to be configured and what kind of internal components had been selected for their construction.

147

DEMONSTRATION PROTOTYPE

SENSOR piezoelectric ultrasonic transducer

Ø10mm (3,3MHz) CW Doppler

OFF

ON

piezoelectric for double tap turning on

148

BRAIN

OUTPUT

volume/power knob OFF

current LED removal

ON

fetal doppler board

current disposable battery

active EQ

X23

RGB LED light system heartbeat sound lighting effect

Arduino UNO

TrinketPro 5V rechargable battery power bank

RGB LED light system pulsing when the device is on under heartbeat sound research battery recharging feedback visual audio signal -

149

HART 1.0

SENSOR piezoelectric ultrasonic transducer

Ø10mm (3,3MHz) CW Doppler

OFF

ON

piezoelectric for double tap turning on

150

BRAIN

OUTPUT

no volume/power knob (it will be always on, at maximum volume) OFF

current LED removal

ON

active EQ

fetal doppler board

current disposable battery removal

DC Stepper 5V to 9V

lithium rechargable battery power bank transistor TIP31

TrinketPro 5V

RGB LED light system pulsing with heartbeat when the device is on visual audio signal battery recharging feedback -

151

MAKING OF The following pictures report some crucial steps related to all the work of about 6 months of research on feasibility and hyphothesis for turning HART concept into reality. Wrong approaches and initial mistakes are also showed, with the convinction that it will be always useful to mind about them in the future. figure 4.29 - HART prototype making of reportage

152

153

154

155

156

157

*further pictures are available in the digital version of the document

158

conclusion HART is the base platform of development of an immediate final product design process. In the near future it has the goal to meet feasibility, production and marketing strategy issues, probably inside a start-up environment.

What on the beginning of this work was being considered relatively easy, became quite complex and wide; wider than expected, because of many unpredictable factors: a bigger potential target, a technological and technical complexity which requires a team of experts, the unavoidable need of industrial solutions beside the starting idea of possible small “makers” production.

It started with a complex question and it ended with a relatively simple answer. This should already sound as a good result of the entire process and a positive ingredient for the future. Personal heartbeat sound available for live amplification or personal listening can be really delivered to the identified target; it just need an immediate further step in understanding its final configuration as a working full performative prototype, with the complexity of miniaturization unfortunately involved in the issue. Can we play music more instinctively? Yes, we just have to learn how to manage with an unpredictable and completely new rhythmic source. Some artists are already doing it, and out of artistic purpose, following heartbeat sound, we can generally act more instinctively, we can listen better to our body or to someone else body, it could be one of the way to get more into it, opening an other new innovative door to the world of bionic technology in relation with sound, which someone like to call bio-creativity. This project can let us discover better why there is such a strong and quite unexplore relationship in between music genres BPM and heartbeat BPM related to different human activities.

HART needs now to be prototyped onto a higher level which brings it to be ready for real test with the users. The huge amount of knowledge acquired, the mistakes during the path and the core solution inside the pre-engineered product version are the fundamentals of the further development.

HART is open to new contaminations and to everybody interested in heartbeat sound new applications.

Tutto è cominciato con una domanda complessa concludendosi con una risposta relativamente semplice. Questo dovrebbe già considerarsi un buon risultato e un ingrediente positivo per il futuro. Il suono reale del battito cardiaco disponibile per performance dal vivo o ascolto personale può realmen-

HART is the achievement of a mix of very different inputs. It is still under development believing that new points of view can have their place in something so related to everyone like our own heartbeat 159

di produzione e commercializzazione, probabilmente all’interno di un sistema startup. Ciò che all’inizio di questo lavoro era stato considerato relativamente semplice, è successivamente diventato abbastanza ampio e complesso, più ampio di quanto atteso a causa di molti fattori imprevedibili: un target potenziale più vasto, una complessità tecnica e tecnologica che necessita di un team di esperti, l’inevitabile bisogno di una produzione industriale a discapito dell’idea iniziale di una possibile autoproduzione con soluzioni dal mondo dei makers. Adesso HART deve essere prototipato ad un livello più alto per poter essere pronto ai test reali con gli utenti. La grande quantità di conoscenza, tutti gli errori commessi lungo il percorso e le soluzioni chiave all’interno della versione pre-ingegnerizzata del prodotto saranno i fondamenti dello sviluppo futuro.

te essere consegnato al target identificato; c’è solo bisogno di capire quale sia il passo immediatamente successivo per la configurazione del prototipo funzionante definitivo, purtroppo con la complicazione necessaria, di un livello di miniaturizzazione accettabile coinvolto nella questione. Possiamo suonare musica più istintivamente? Si! Dobbiamo solo capire come relazionarci ad una fonte ritmica completamente nuova caratterizzata da imprevedibilità.

Alcuni artisti lo stanno già facendo, e al di fuori di intenti artistici, seguendo il suono del battito cardiaco possiamo generalmente agire più istintivamente, possiamo ascoltare meglio il nostro corpo o anche quello di qualcun’altro.

Può diventare un modo nuovo di interagire con il corpo umano aprendo una nuova porta verso il mondo della biotecnologia in relazione alla creatività del suono, un concetto che qualcuno ama chiamare bio-creatività. Questo progetto potrebbe farci scoprire come mai sembri esserci una forte relazione fra i BPM dei generi musicali e i BPM relativi alle differenti attività umane.

HART è aperto a nuove contaminazioni e chiunque altro sia interessato a nuove applicazioni per il suono del battito cardiaco.

HART è il raggiungimento di un mix di numerosi stimoli. È ancora in fase di sviluppo, con l’idea che qualcosa di così largamente condiviso come il battito cardiaco, possa beneficiare dell’opinione di chiunque. HART vuole essere la piattaforma di sviluppo di un prodotto che mira ad una successiva rapida realizzazione sul mercato. Nel futuro prossimo l’obiettivo è focalizzarsi su questioni di fattibilità, strategia 160

4.7 - future development: HART engineering and potential

until now is a primary setup which needs the feedbacks and the experiences of the users.

Some of the interesting tests we want to perform with HART 1.0 are: –– survey to 30 artists asked to perform during a 20 minutes session with their own heartbeat –– one entire artistic performance of an artist based on heart­beat, survey to the audience in order to understand the impact of passive heartbeat listening on people –– 15 minutes listening session for 30 standard users with different back­ ground, for a first check on potential private listeners. People interested in heartbeat listening during activities in their ordinary life.

–– slowing/speeding personal heartbeat sound during listening (users BPM are tracked during short listening session).

The goal is to discover if their real BPM change in accordance with what they listen to (their own heartbeat slowed or speeded). This could confirm the presence of the function into the app or adding it on the device itself to open interesting application to unpredicted users.

HART wishes a future development. This re­ search stops at this moment because a project like HART can’t be closed in the standard timing setup of a final project work for a master thesis. The wish for the future is about a continous update and improvement process for the whole concept, until surely the primary goal of a full functioning HART system, immediately available for specific test environment, and for its primary target group, artists.

This could be actually the way how HART can start to move closer to reality than now.

Its potential has to be shown in artistic performances for projects like Offrimi il cuore

by Antonello Fresu. It isn’t a new discovery that artistic environments are the engine for big revolutions; the requirements there, are less restrictive meanwhile the communication impact is high.

HART is unfortunately an high cost tech­ nology; this study has the worthiness of this certainty. Miniaturization is the main reason of this high cost: since HART needs to be wearable, this primary requirement makes everything more complicated. So the project needs fund­ing, and this is one of the reason for such a focus on prototype system. The prototype is the way how the concept is truly deliverable to real user experience, which is what create business attention to the project.

The working prototype will be the core of an hypothetic startup development mean­ while the tests with it will be the upgrade design engines. What has been designed

The pre-engineered version of HART has been planned to be the development platform for a standard request to an electronic engineer with experience in sound engineering, or a full company with experience in this kind of project development. They are professional figure which is unavoidable to individuate for the development of the industrial version of the product, or even a full working prototype already miniaturized until a reasonable level 161

HART has been designed, until all the fund­ amental analysis a designer can perform.

Design engineering is the environment where this project has seen is birth; this is why, regarding some arguments, the analysis went even deeper. HART had the power or the luck of getting in touch with interesting and interested parties. The present is promising; everything could be planned or predicted but the future is unknown, and almost like the heartbeat during a performance, wayward.

for wearability and artistic performances.

Further design engineering process would be focused on: –– final decisions about the internal com­ ponents in terms of producers and per­ formances –– final material selection –– production cost analysis Moreover funding and marketing strategies has to be involved in the whole process. HART can’t really manage with a maker production for its whole life process. This was a wrong idea at the beginning of this work, because the technology involved hasn’t been considered so complex. There is high electronic and high acoustic engineering inside such a project, this has to be declared without filters and with respect for that amount of needed knowledge. Also a mechanic passive solution (minding on the tentative inspiration of the stethoscope), if it somehow nullifies the need of electronics, make higher and higher the acoustic engineering knowledge, which can be even more complex.

A primary evaluation of the cost of the whole device, considering a small industrial production is around 120€.

Which is also a sort of limit budget for the manufacturing cost, in order to keep HART under an acceptable final price for the future customers. HART general orientation in the market is to start on a medium positioning in between a relatevely cheap product and a high quality technology; but in order to understand where of the two tendencies, the final output should orient itself, further development and especially opi­ nions by testers and first customers will be fundamental.

162

163

sources

Sira-Ramirez, Hebertt J., and Ramón Silva-Ortigoza Control design techniques in power electronics devices Springer Science & Business Media, 2006

4.2 - system tech­nology: suitable solution for multiple users

4.Project development

Led, Santiago, Jorge Fernández, and Luis Serrano Design of a wearable device for ECG continuous monitoring using wireless technology In Engineering in Medicine and Biology Society, 2004. IEMBS’04. 26th Annual International Conference of the IEEE, vol. 2, pp. 3318-3321.

IEEE, 2004. [online, article]

4.1 - system requi­re­­ments: heart sound system performances Kate Hartman Make: Wearable Electronics: Design, pro­ totype, and wear your own interactive gar­ ments Maker Media, Inc., 2014

Lee, Young-Dong, and Wan-Young Chung Wireless sensor network based wearable smart shirt for ubiquitous health and activity monitoring Sensors and Actuators B: Chemical 140, no. 2 (2009): 390-395. [online, article] http://www.sciencedirect.com/science/article/ pii/S0925400509003724

McCann, Jane, and David Bryson, Smart clothes and wearable technology Elsevier, 2009 Tao, Xiaoming, Wearable electronics and photonics Elsevier, 2005 Cauwels, Patrick J., Rachid M. Alameh, Jeong J. Ma, and Paul R. Steuer Wearable Band with Ease of Adjustment U.S. Patent Application 13/491,054, filed June 7, 2012. Harvard [online, patent] https://www.google.com/patents/US201­ 30­326790

Hahn, Robert, and Herbert ReichlBatteries and power supplies for wearable and ubiquitous computing In Wearable Computers, 1999. Digest of Papers. The Third International Symposium on, pp. 168-169. IEEE, 1999 [online, article] http://ieeexplore.ieee.org/abstract/document/806705/ 164

Create arduino stereo neopixel ring vu meter [online, website] https://create.arduino.cc/projecthub/ericBcreator/stereo-neopixel-ring-vu-meter-b28e78

ST microelectronics [online, website] http://www.st.com

4.3 - final concept:­HART, the heartbeat sound controller

4.7 - future development: HART engineering and potential

Cambridge dictionary [online, website] https://dictionary.cambridge.org/dictionary/ english/art

Müller, Roland M., and Katja Thoring Design thinking vs. lean startup: A comparison of two user-driven innovation strategies [online, article] Leading Through Design 151. 2012

4.4 - system interaction: how do we get into heart sound?

Product Design 101 for Developers, Startups & Designers [online, article] https://www.blazemeter.com/blog/product-design-101-developers-startups-designers

Tidwell, Jenifer Designing interfaces: Patterns for effective interaction design O’Reilly Media, Inc.”, 2010

4.6 - a working prototype: HART first system setting Adafruit [online, website] https://www.adafruit.com/

Instructables [online, website] http://www.instructables.com/ Rashid, Muhammad H. Power electronics handbook: devices, circuits and applications Academic press, 2010 165

5.References

5.1.Reasearch 5.1.1 - a focus on musical instruments design

5 futuristic instruments changing how we make music [online, article] https://www.fastcodesign.com/3041135/5futuristic-instruments-changing-­how-wemake-music

Ben Hayes When analogue, digital and acoustic sound meet [online, article] https://roli.com/stories/when-analogue-meets-digital

Interaction design brief [online, article] https://www.interaction-design.org/literature/ book/the-encyclopedia-of-human-computer-interaction-2nd-ed/interaction-design-brief-intro

Sachs, Curt The history of musical instruments Courier Corporation, 2012

Interaction design brief [online, article] https://www.techopedia.com/def inition/27248/interaction-design-ixd

The 10 most curious and successful kickstarter instruments [online, article] https://www.gadgetdaily.xyz/the-10-most-curious-and-successful-kickstarter-instruments/

Rudolf Laban Choreutics Ed. Lisa Ullmann. Macdonald & Evans, 1966 Fernando Lazzetta Meaning in Musical Gesture. Trends in gestural control of music 2000

Introducing the artiphon instrument [online, website] https://www.kickstarter.com/projects/artiphon/introducing-the-artiphon-instrument-1

Mark Leman An Embodied Approach to Music Semantics Musicae Scientiae, 2010

Stanislavsky, Constantin.

Creating a Role Trans. Elizabeth Reynolds Hapgood. New York: Theatre Arts Books, 1961 Dualo, the new musical instrument for all [online, website] https://www.kickstarter.com/proje­c ts­ ­­/63006320/dualo-the-new-musical-instrument-for-all

Godøy, Rolf Inge, and Marc Leman Musical gestures: Sound, movement, and meaning Routledge, 2010

167

C. Cadoz and M. M. Wanderley.

Trends in Gestural Control of Music Centre Pompidou Paris, IRCAM - Centre Georges Pompidou, 2000

Mogees [online, website] http://www.mogees.co.uk/

5.1.2 - what music are we playing?

Kickstarter [online, website] http://www.kickstarter.com/

Kusek, Dave, and Gerd Leonhard The future of music: Manifesto for the digital music revolution 2005

Drumpants [online, website] http://www.drumpants.com/ Analog digital what’s the difference [online, article] https://www.recordingconnection.com/reference-library/recording-education/analog-digital-whats-the-difference/

9 ways musicians actually make money today [online, article] http://www.rollingstone.com/music/ lists/9-ways-musicians-actually-make-money-today-20120828

Why this is a great time to be in music [online, article] http://www.savvymusician.com/blog/2011/01/ why-this-is-a-great-time-to-be-in-music/

The new economics of the music industry [online, article] http://www.rollingstone.com/music/news/ the-new-economics-of-the-music-industry-20111025

Andrea Doria La musica il suono e le frequenze: informazione coerente a 432 hertz [online, article] http://www.scienzaeconoscenza.it/blog/consapevolezza-spiritualita/musica-suono-frequenze-432-hertz

2015 music industry trends [online, article] http://www.lawyersrock.com/2015-music-industry-trends/ Statista [online, website] https://www.statista.com 2015 music industry trends [online, article] http://thenexttech.startupitalia.eu/55­3 120160113-kurv-musica-chitarra-virtuale 168

5.1.3 - who are musicians today?

Seaboard-rise [online, website] https://roli.com/products/seaboard-rise

The future and role of music in modern society [online, article] http://www.savvymusician.com/blog/2010/04/ the-future-and-role-of-music-in-modern-society/

Nrityadhol [online, website] https://nrityadhol.wordpress.com Aaron-sherwood [online, blog] http://aaron-sherwood.com/blog

Are todays artist more uniform and less musically adventurous yhan those of yester year? [online, article] http://www.bulletproofmusician.com/are-todays-artists-more-uniform-and-less-musically-adventurous-than-those-of-yesteryear/

5.1.5 - new sound sources What causes involuntary movements? 14 possible conditions [online, article] http://www.healthline.com/symptom/involuntary-movements

Kickstarter [online, website] http://www.kickstarter.com/

Offrimi il cuore | Roma Jazz Festival 2012 [online, video] https://www.youtube.com/watch?v=CKa8TlaltD8&feature=youtu.be

Steim [online, website] http://steim.org/what-is-steim/

Polotti, Rocchesso Sound to Sense, Sense to Sound, A State of the Art in Sound and Music Com­puting Pietro Polotti and Davide Rocchesso, editors IUAV, Dipartimento delle arti e del disegno industriale, 2008

XTH [online, website] http://www.xth.io/

5.1.4 - sound and movement

Trevor Pinch, Karin Bijsterveld The Oxford Handbook of Sound Studies Oxford University Press, 2011

Theremin [online, website] https://nl.wikipedia.org/wiki/Theremin

Lane Arye Unintentional Music: Releasing Your Deepest Creativity Hampton Roads Publishing, 2002 169

Arslan Burak, Andrew Brouse, Julien Castet, Remy Léhembre, Cédric Simon, Jehan-Julien Filatriau, Quentin Noirhomme [online, article] A real time music synthesis environment driven with biological signals ICASSP06 - IEEE International Conference on Acoustics, Speech, and Signal Processing, May 2006, Toulouse, France, 2006 A. Tanaka

Eindhoven University of Technology Personal touch: Hearing a heartbeat has the same effect as looking each other in the eye ScienceDaily [online, article] www.sciencedaily.com/releases/2011­­/04/1104 07093118.htm (accessed July 20, 2017).

Joris H. Janssen, Jeremy N. Bailenson, Wijnand A. IJsselsteijn, Joyce H.D.M. Westerink Intimate Heartbeats: Opportunities for Affective Communication Technology IEEE Transactions on Affective Computing 2010 Eduardo R. Miranda On Biophysical Music Guide to Unconventional Computing for Music.

Berlin: Springer, 2017.

Musical perfomance practice on sensor-based instruments in Trends in Gestural Control of Music, M. M. Wanderley and M. Battier, Eds., pp. 389–406.

IRCAM, 2000 Y. Nagashima Bio-sensing systems and bio-feedback sys- tems for interactive media arts in 2003 Conference on New Interfaces for Musical Expression (NIME03), Montreal, Canada, 2003, pp. 48–53

Computer Music Journal Biophysical Music: Sound and Video Anthology MIT Press, 2015 [online, article] http://www.mitpressjournals.org/doi/ pdf/10.1162/COMJ_a_00333

R. Douglas Fields The Power of Music: Mind Control by Rhythmic Sound [online, article] https://blogs.scientificamerican.com/guestblog/the-power-of-music-mind-control-byrhythmic-sound/

5.2.Focus design 5.2.1 - a focus on musical accompaniment

Mogees [online, website] https://www.mogees.co.uk/

Accompaniment [online, website] https://nl.wikipedia.org/wiki/Accompaniment

170

5.2.2 - sound by our body

Tempo indications and beats per minute (BPM) reference for social dance genres [online, article] http://www.beatsperminuteonline.com/en/ home/bpm-beats-per-minute-reference-fordance-genres

Dave Kusek, Gerd Leonhard The future of music: Manifesto for the digital music revolution Ed. Susan Gedutis Lindsay. Boston: Berklee Press, 2005

Fradà, G., and G. Fradà Semeiotica medica nell’adulto e nell’anziano Piccin, IV edition, 2009

Haga, Egil Correspondences between music and body movement [online, thesis] 2008

Polotti, Rocchesso Sound to Sense, Sense to Sound, A State of the Art in Sound and Music Com­puting Pietro Polotti and Davide Rocchesso, editors IUAV, Dipartimento delle arti e del disegno industriale, 2008

Sam Loman illustration of human body as metro map [online, website] http://www.just-sam.com/

Ascoltare il battito del proprio cuore fa capire meglio le emozioni degli altri [online, article] http://www.ansa.it/canale_saluteebenessere/ notizie/medicina/2017/05/02/ascoltare-il-battito-del-proprio-cuore-fa-capire-meglio-le-emozioni-degli-altri_e88b0d3c-c80843a6-8e0c-31aa46d7830b.html

Daniel J. Levitin This Is Your Brain on Music 2007 This Is Your Brain on Music [online, essay] http://www.ldv.ei.tum.de/fileadmin/w00bfa/www/Vorlesungen/Brain_Mind_Cognition/Your_Brain_On_Music_Essays_ Group_A_2011.pdf

Time [online, website] http://www2.siba.fi/muste1/index. php?id=102&la=en

TEMPO-RADIO project [online, website] http://www.gemmaroper.com/TEMPO-RADIO Tempo [online, website] https://en.wikipedia.org/wiki/Tempo

171

5.2.3 - musical instru­ments for the body: market situation

http://josephmalloch.com/

Musical Instruments Market: Global Industry Analysis and Opportunity Assessment 2016-2026 [online, article] http://www.futuremarketinsights.com/reports/musical-instruments-market Global Electronic Musical Instrument Market 2017-2021 [online, article] http://www.prnewswire.com/news-releases/ global-electronic-musical-instrument-market-2017-2021-300410269.html

Future Sounds Like This: 10 Odd Modern Musical Instruments [online, article] http://weburbanist.com/2009/11/17/the-future-sounds-like-this-10-magnificently-modern-musical-instruments/

Drumpants [online, website] http://www.drumpants.com/

Joseph Malloch, Ian Hattwick, and Marcelo M. Wanderley Instrumented Bodies: Prosthetic Instruments for Music and Dance In A Framework and Tools for Mapping of Digital Musical Instruments, Ph.D. thesis, Music Technology, McGill University, Montréal, 2013

Farmer footdrum [online, website] https://www.footdrums.com/

Emma Tucker Wearable pods let musicians compose and perform tracks as vibrations on your skin [online, article] https://www.dezeen.com/2017/06/30/ royal-college-art-graduate-marie-tricaud-touche-system-musicians-compose-perform-vibrations/

Node:0 [online, website] https://www.indiegogo.com/projects/node-0of-the-sonic-fractal-matrix-a-vision-of-anew-model-for-music#/ XTH Sense [online, website] h t t p s : / / w w w . k i c k s t a r t e r. c o m / p r o jects­/1814357959/xth-sensetm­-the-worldsfirst-biocreative-instrumen/updates

5.3.Concept design

GePS, Gesture-based Performance System [online, website] http://geps.synack.ch/

5.3.1 - a problem to solve: heartbeat sound amplifi­­ca­tion out of medical appli­ca­tion

Joseph Malloch [online, website] 172

16 songs with a heartbeat [online, website] http://www.songfacts.com/blog/writing/16_ songs_with_a_heartbeat/

Neramitr Chirakanphaisarn, Thadsanee Thongkanluang, Yuwathida Chiwpreechar Heart rate measurement and electrical pulse signal analysis for subjects span of 20–80 years [online, article] http://www.sciencedirect.com/science/article/ pii/S2314717216300265

Gianella Demuro Offrimi il cuore Nero Project, Antonello Fresu, 2012 Helen Thomson, NEW SCIENTIST Listening to your heartbeat helps you read other people’s minds [online, article] https://www.newscientist.com/article/­ 2129367-listening-to-your-heartbeat-helpsyou-read-other-peoples-minds/

Tarun Agarwal Heartbeat Sensor – Working & Application [online, article] https://www.elprocus.com/heartbeat-sensor-working-application/ Stethoscope [online, article] https://en.wikipedia.org/wiki/Stethoscope

Heartbeat sound listening [online, website] https://www.youtube.com/results?q=heartbeat+sound+listening&sp=SABQFOoDAA%253D%253D

Liat Clark Meet the doctor bringing cheap, 3D printed medical devices to Gaza [online, article] http://www.wired.co.uk/article/3d-printed-stethoscope-gaza

5.3.2 - the technology behind auscultation Mario J. Mc Loughlin, Santiago Mc Loughlin Cardiac auscultation: Preliminary findings of a pilot study using continuous Wave Doppler and comparison with classic auscultation [online, article] http://www.internationaljournalofcardiology. com/article/S0167-5273(12)01346-0/fulltext

Kashmira Gander Gaza doctor Tarek Loubani creates 3D printed stethoscopes to alleviate medical supply shortages caused by blockade [online, article] http://www.independent.co.uk/news/world/ middle-east/gaza-doctor-tarek-loubani-creates-3d-printed-stethoscopes-to-alleviate-medical-supply-shortages-10495512. html

173

scope2.html

Andy Boxall 15 Year-old creates a 3D printed iPhonecase that’s also a stethoscope [online, article] https://www.digitaltrends.com/mobile/stethio-case-turns-your-iphone-into-a-stethoscope/

One Digital Stethoscope [online, website] http://www.thinklabs.com/one-digital-stethoscope

5.3.3 - the possible heart­ beat sound

Houtsma, A. J., Curry, I. P., Sewell, J. M., & Bernhard, W. N. A dual-mode noise-immune stethoscope for use in noisy vehicles ARMY AEROMEDICAL RESEARCH LAB FORT RUCKER AL. [online, article] http://www.dtic.mil/docs/citations/ADA481444

Farshad Arvin, Shyamala Doraisamy, and Ehsan Safar Khorasani Frequency shifting approach towards textual transcription of heartbeat sounds [online, article] https://www.ncbi.nlm.nih.gov/pmc/articles/ PMC3396354/

Bridger, K., Cooke, A. V., Kohlhafer, D. J., Lutian, J. J., Sewell, J. M., & Strite, R. E. U.S. Patent No. 8,265,291. Washington, DC: U.S. Patent and Trademark Office. [online, patent] https://www.google.com/patents/US8265291

Phua K., Chen J., Dat T., Shue L. Heart sound as a biometric Pattern Recognition, 2008 Arvin F, Doraisamy S. Real-time segmentation of heart sound pattern with amplitude reconstruction IEEE EMBS Conference on Biomedical Engineering and Sciences, 2009 Heart sounds [online, article] https://en.wikipedia.org/wiki/Heart_sounds

David A. Van Alstine Electronic Stethoscope [online, patent] https://www.google.com/patents/ US20140254814#backward-citations S.M. Debbal, F.Bereski-Reguig Frequency analysis of the heartbeat sounds [online, article] https://www.researchgate.net/publication/228412447_Frequency_analysis_of_ the_heartbeat_sounds

Jiang Z., Choi S. A cardiac sound characteristic waveform method for in-home heart disorder monitoring with electric stethoscope Expert Systems with Applications. 2006

Active Signal Technologies [online, website] http://www.activesignaltech.com/Stetho-

Khorasani E, Doraisamy S, Arvin F. 174

An Approach for Heartbeat Sound Transcription International Conference on Computer Technology and Development, IEEE. 2009

Liu, Hao, Jun Hu, and Matthias Rauterberg Music playlist recommendation based on user heartbeat and music preference Computer Technology and Development, 2009.

ICCTD’09. International Conference on. Vol. 1.

IEEE, 2009. [online, article] http://ieeexplore.ieee.org/abstract/document/5360027/?reload=true

Heartbeat sounds [online, website] https://www.soundsnap.com/tags/heartbeat

5.3.4 - who could be intere­sted in heartbeat sound?

Dunn, Barnaby D., et al.

Listening to your heart: how interoception shapes emotion experience and intuitive decision making Psychological science, 2010. [online, article] http://journals.sagepub.com/doi/pdf/­ 10.1177/0956797610389191

Gianella Demuro Offrimi il cuore Nero Project, Antonello Fresu, 2012

Fukumoto, Makoto, and Junichi Imai Evolutionary computation system for musical composition using listener’s heartbeat information IEEJ Transactions on Electrical and Electronic Engineering 3.6, 2008: 629-631. [online, article] http://onlinelibrary.wiley.com/doi/10.1002/ tee.20324/full

Distribution of the 10 leading causes of death in the United States in 2015 [online, website] https://www.statista.com/statistics/248619/ leading-causes-of-death-in-the-us/ Leading 10 causes of death in low-income countries worldwide in 2015 (in deaths per 100,000 population) [online, website] https://www.statista.com/statistics/311934/ top-ten-causes-of-death-in-low-incomecountries/

Marie Tricaud [online, website] http://www.marietricaud.com/touche.html Wiens, Stefan, Elizabeth S. Mezzacappa, and Edward S. Katkin.

Heartbeat detection and the experience of emotions Cognition & Emotion 14.3 (2000): 417-427. [online, article] http://www.tandfonline.com/doi/abs/­ 10.­1080/026999300378905

Bechara, A., & Naqvi, N. Listening to your heart: interoceptive awareness as a gateway to feeling. Nature neuroscience, 2004, 7(2), 102-103. [online, article] https://www.nature.com/neuro/journal/v7/n2/ full/nn0204-102.html 175

5.4.Project development

5.3.5 - possible heartbeat interaction Christian Vasile UI Principles for Great Interaction Design [online, article] https://designmodo.com/ui-principles-interaction-design/

5.4.1 - system requi­re­­ments: heart sound system performances

The Touch [online, website] http://www.thetouchx.com/

Kate Hartman Make: Wearable Electronics: Design, pro­ totype, and wear your own interactive gar­ ments Maker Media, Inc., 2014

Leonid Miakotko The impact of smartphones and mobile devices on human health and life [online, article] http://www.nyu.edu/classes/keefer/waoe/miakotkol.pdf

McCann, Jane, and David Bryson, Smart clothes and wearable technology Elsevier, 2009

Breadboarding the circuit [online, website] https://learn.adafruit.com/pulse-sensor-displayed-with-neopixels/breadboarding-the-circuit

Tao, Xiaoming, Wearable electronics and photonics Elsevier, 2005 Cauwels, Patrick J., Rachid M. Alameh, Jeong J. Ma, and Paul R. Steuer Wearable Band with Ease of Adjustment U.S. Patent Application 13/491,054, filed June 7, 2012. Harvard [online, patent] https://www.google.com/patents/US201­ 30­326790 Sira-Ramirez, Hebertt J., and Ramón Silva-Ortigoza Control design techniques in power electronics devices Springer Science & Business Media, 2006

176

5.4.2 - system tech­nology: suitable solution for multiple users

5.4.3 - final concept:­HART, the heartbeat sound amplifier

Led, Santiago, Jorge Fernández, and Luis Serrano Design of a wearable device for ECG continuous monitoring using wireless technology In Engineering in Medicine and Biology Society, 2004. IEMBS’04. 26th Annual International Conference of the IEEE, vol. 2, pp. 3318-3321.

IEEE, 2004. [online, article]

Cambridge dictionary [online, website] https://dictionary.cambridge.org/dictionary/ english/art

5.4.4 - system interaction: how do we get into heart sound?

Lee, Young-Dong, and Wan-Young Chung Wireless sensor network based wearable smart shirt for ubiquitous health and activity monitoring Sensors and Actuators B: Chemical 140, no. 2 (2009): 390-395. [online, article] http://www.sciencedirect.com/science/article/ pii/S0925400509003724

Tidwell, Jenifer Designing interfaces: Patterns for effective interaction design O’Reilly Media, Inc.”, 2010

5.4.6 - a working prototype: HART first system setting

Hahn, Robert, and Herbert ReichlBatteries and power supplies for wearable and ubiquitous computing In Wearable Computers, 1999. Digest of Papers. The Third International Symposium on, pp. 168-169. IEEE, 1999 [online, article] http://ieeexplore.ieee.org/abstract/document/806705/

Adafruit [online, website] https://www.adafruit.com/ Instructables [online, website] http://www.instructables.com/ Rashid, Muhammad H. Power electronics handbook: devices, circuits and applications Academic press, 2010

ST microelectronics [online, website] http://www.st.com

177

in the “Music Events” segment in 2016 by https://www.statista.com/ figure 1.5 - Statista report about revenue in the “Digital Music” segment in 2016 by https://www.statista.com/ figure 1.6 - Statista report about recorded music market revenue worldwide from 2005 to 2015 by https://www.statista.com/ figure 1.7 - XTH Sense performing figure 1.8 - XTH Sense bio-creativity inputs figure 1.9 - Leon Theremin figure 1.10 - Firewall visual/sound interaction figure 1.11 - Firewall fabric depth figure 1.12 - ROLY seabord figure 1.13 - Rudolf Laban figure 1.14 - Laban Theory Eight Efforts figure 1.15 - NrityaDHol appereance figure 1.16 - NrityaDHol setting figure 1.17 - Stimulus by Bart Hess figure 1.18 - Antonello Fresu looking for the right auscultation point figure 1.19 - “Offrimi il cuore” by Nero Project

Create arduino stereo neopixel ring vu meter [online, website] https://create.arduino.cc/projecthub/ericBcreator/stereo-neopixel-ring-vu-meter-b28e78

5.4.7 - future development: HART engineering and potential Müller, Roland M., and Katja Thoring Design thinking vs. lean startup: A comparison of two user-driven innovation strategies [online, article] Leading Through Design 151. 2012 Product Design 101 for Developers, Startups & Designers [online, article] https://www.blazemeter.com/blog/product-design-101-developers-startups-designers

All the pictures without a different indication for the reference have their source in Google Images.Their original look has been modified.

5.5. figures references

2. Focus design figure 2.0 - “Breath” by MILLO figure 2.1 - percussions figure 2.2 - one-man-band figure 2.3 - “The human body” exhibition figure 2.4 - human body metro map infographics by Sam Loman figure 2.5 - human body involved areas in most common musical instrument playing by the author

1. Research figure 1.0 - “Love Seeker” by MILLO figure 1.1 - Artiphon figure 1.2 - Dualo figure 1.3 - GePS interaction figure 1.4 - Statista report about revenue 178

figure 3.9 - Pi KeoHavong, dancer performing for “Offrimi il cuore” project figure 3.10 - example of a guitarist effects setup figure 3.11 - Touchè prototype figure 3.12 - lighiting project with Pulse Sensor and Arduino setup figure 3.13 - The Touch figure 3.14 - final brief by the author

figure 2.6 - dynamic and static acting for musicians by the author figure 2.7 - main music genres tempo vs BPM activity by the author figure 2.8 - interesting areas to focus on for a new musical instrument design by the author figure 2.9 - real products on the market by the author figure 2.10 - market analysis for personal artists experiments by the author figure 2.11 - market analysis for concepts still under development, proto­types and very young products by the author figure 2.12 - design orientation by the author figure 2.13 - more instinct in playing by Roli music setup

All the pictures without a different indication for the reference have their source in Google Images. Their original look has been modified.

4. Project development figure 4.0 - “Love Runs the World” by MILLO figure 4.1 - wearable technology heart contact related moodboard by the author figure 4.2 - shaping system of a compact device figure 4.3 - shaping system of a multiple components device figure 4.4 - system assembly options by the author figure 4.5 - Andrea Pavoni Belli managing with acoustic departments equipment at INRIM institute by the author figure 4.6 - analysis of different technological options at INRIM institute by the author figure 4.7 - heartbeat audio EQ in order to reduce background noises by Andrea Pavoni Belli and the author figure 4.8 - heartbeat audio EQ in order to reduce background noises (20 band) by Andrea Pavoni Belli and the author figure 4.9 - example of a EQ pedal usually developed for bass

All the pictures without a different indication for the reference have their source in Google Images. Their original look has been modified.

3. Concept design figure 3.0 - “Heart Slingshot” by MILLO figure 3.1 - “Offrimi il cuore” by Nero Project figure 3.2- René Laennec portrait figure 3.3 - heartbeat sound amplification technologies by the author figure 3.4 - what’s inside Angelsounds fetal Doppler by Jumper by the author figure 3.5 - different points of heartbeat sound auscultation figure 3.6 - Antonello Fresu fixing the device on artist chest figure 3.7 - Paolo Fresu performing with his heartbeat sound figure 3.8 - target analysis 179

figure 4.10 - HART lighting interaction system by the author figure 4.11 - HART healthy aesthetics of interaction by the author figure 4.12 - HART usability aesthetics of interaction by the author figure 4.13 - HART system feedbacks and signal by the author figure 4.14 - HART storyboard by the author figure 4.15 - HART main app preview by the author figure 4.16 - HART app functions preview by the author figure 4.17 - HART system elements by the author figure 4.18 - HART dynamic system by the author figure 4.19 - HART orthographic projections by the author figure 4.20 - HART axonometric projections by the author figure 4.21 – HART sections by the author figure 4.22 - HART exploded view by the author figure 4.23 – HART renderings by the author figure 4.24 - Unborn Heart™ fetal doppler device by Odosoft Ltd figure 4.25 - LED sound reactive system with TRINKET PRO figure 4.26 - desired lighiting effect figure 4.27 - NeoPixel circles figure 4.28 - PRO TRINKET 5V pinout figure 4.29 - HART prototype making of reportage by the author

for the reference have their source in Google Images. Their original look has been modified.

All the unlisted pictures in this chapter, like sketches, schemes or renderings were all created by the author. If they haven’t been numbered this decision came from the idea that they are enough clear to understand or in order to don’t complicate too much the layout and the general reading of this part of the work, which can be considered an entire image of a design process created by the author.

5.6. special sources All the street art pictures selected for every chapter front cover are online available pictures by the amazing work of an italian artist. His name is Francesco Camillo Giorgino, known as MILLO. https://www.millo.biz/ In relationship with the chapters they are: 0.Start: “Freedom” Vilnius, Lithuania, 2015 1.Research: “Love Seeker” Milano, Italy, 2015 2.Focus Design: “Breath” Torino, Italy, 2014 3.Concept Design: “Heart Slingshot” Milano, Italy, 2015 4.Project Development: “Love Runs the World” Kiev, Ukraine, 2016 5.References: “Kriebelstad” Heerlen, Netherland, 2015

5. References figure 5.0 - “Kriebelstad” by MILLO All the pictures without a different indication 180

This is a final thesis developed inside “Design & Engineering” Master Degree Course in Politecnico di Milano

The acoustic department of INRIM (Istituto Nazionale di Ricerca Metrologica) supported the project development section of this work.

Antonello Fresu “Offrimi il cuore” project is the engine of the concept design chapter. His artistic research inspired the whole work, like the good quality art usually does.

In order to know more about “Offrimi il cuore” the artistic activity of Antonello Fresu, further informations are available at following link: http://antonellofresu.it/

181

Acknowledgements My heartfelt thanks to:

Antonello Fresu, “Offrimi il cuore” project founder, he believed in the project immediately, sharing 10 years of experience in a project that is definitely one of the main engines of this work.

Sara Colombo, she followed and inspired me from the beginning, when everything was more than confused. Her positive and negative feedbacks, her pushes and brakes are the reason why this work was born with a solid foundation.

Barbara Del Curto, she guided me during the last crucial step of this work. Her help was essential to bring a complex research output until a concrete product design proposal.

Andrea Pavoni Belli, scientist operating in acoustic department of INRIM metrological institute in Torino, he has been “the light in the dark world of electronics and acoustic engineering” for a designer.

Alessandro Casalaspro, friend, designer and musician, he was soon amazed about the concept. He is the definitely the 1st heartbeat player, and the reference user to test the project development.

Matteo Rivelli, friend and medician, he sup­­ port­ ed scientifically the research on eve­­­­ry medical doubt, with enthusiasm and availability at really every hour, like just a real friend can do.

Alessandro Scisci and Antonio Laviola; their different approach and passion about ele­ctronics have been the main guide of the prototype development. It has been a pleasure to share with you the first steps of this technology.

Luca Manniti and Lucia Strazzeri, amazing friends and musicians I met in Amsterdam at the early beginning of this work. They gifted to me support and key-informations for the whole project research.

Tea Anisia Carpentieri, we shared the same path, the both of us on our final thesis mode, and everything was supported by our crazy and lovely way to face difficult moments together.

My family, you trusted me until the end, supporting me in every possible way, during this work and during 6 years of studies finally on their end. I’m lucky to have you, behind me.

All the people who temporarily offered their heart for experiments and prototype test, I hope you had fun, this was just the beginning!

“ thanks

How product design can generate concrete answers for new contemporary trends in the world of music? Innovating in musical instrument design today means looking for new sound sources.

The core output of the investigation is a question: “can we play music more instinctively?�. There are few musical instruments able to create sound by our instinctive or even involuntary body movement and activity, but the real heartbeat sound is quite unexplored in music production.

Heartbeat is the most relevant involuntary sound produced by our body. It could be an unpredictable rhythmic accompaniment and a new active sound source for music performances. The target could even be wider if beside music and art in general we add other possible applications such as sport monitoring, relaxing practices and basically every expressive activity in connection with human body. The final goal is about providing an acceptable live heartbeat sound signal to the selected target group. This is what HART means and what it hopefully will do in the early future.