SkyNote: An expert AI teacher

SKYNOTE: AN EXPERT AI TEACHER

From programs that claim to curate perfect playlists to software that composes machine-made music, artificial intelligence is gaining traction in the industry. Now, a team of researchers at the Centre for Performance Science has developed a system that promises to help violinists become better, faster. Upbeat finds out more about what machine learning can do for RCM students.

RCM violinist Katherine Yoon using the SkyNote system

The Centre for Performance Science has some good news for violinists. A team of engineers, designers and musicians has created a new technology that helps violin students build awareness of their technique, allowing them to avoid injury and acquire skills more efficiently.

The intelligent learning system, SkyNote, is one of a suite of technologies developed as part of the EU-funded Technology Enhanced Learning of Musical Instrument Performance (TELMI) project, and is the culmination of three years of collaborative research undertaken alongside Pompeu Fabra University in Barcelona and the University of Genoa, as well as the companies Highskillz and SAICO Intelligence.

The international team worked closely to find new ways of capturing and analysing violin performance, with the aim of providing valuable real-time feedback.

‘Expert performers have exceptional awareness and control of their bodies and instruments, which grants them not only the great consistency of quality professional musicians achieve, but also the confidence and freedom to be spontaneous’, explains Dr George Waddell, Research Associate in Performance Science at the Royal College of Music (RCM).

‘SkyNote helps violinists develop these skills by scrutinising their playing through the eyes of an expert teacher, one attuned to the technical aspects vital to their performance.’

The first step in creating this expert AI teacher was to find innovative detection technologies that could be used to collect a range of performance data, without interfering with playing. SkyNote tracks small markers that are attached to a violin and bow in order to unobtrusively monitor specific aspects of bowing including tilt, speed, weight, contact point, inclination, and direction.

Audio, too, can be analysed: the technology can assess pitch and intonation, dynamics, tone quality and rhythm of a range of instruments.

During practice, a camera records all of this data and sends it to specially developed software clever enough to spot errors in technique and provide continual feedback written to a digital score.

SkyNote helps violinists develop skills by scrutinising their playing through the eyes of an expert teacher, one attuned to the technical aspects vital to their performance.

The next challenge for the team was how best to feed this information back to the violinist in a way that would be both helpful and immediate. For this, researchers worked directly with a group of violin students and teachers, led by RCM violin professor Madeleine Mitchell, to design visualisations that would be meaningful to musicians.

Together, they created customised widgets that can be displayed either on a computer screen or directly on the musical score. Importantly, this visual feedback is available both as an overall performance evaluation and in real time as the musician is playing, enabling them to swiftly correct any errors during practice.

Real-time graphic feedback from SkyNote during a violin rehearsal

So far, SkyNote has proved to be a remarkable teacher. In fact, students have showed a twofold improvement after just 10 minutes of using the technology. And importantly, by helping students become more efficient in their practice, the new learning system may help reduce injuries, too.

‘We see this epidemic of injuries among musicians, with rates comparable to professional athletes in terms of injury that affects their ability to perform’, says George. ‘Expectations are constantly increasing, with far greater pressure on musicians to perform more difficult repertoire at a younger age. Extra pressure means more practising, which increases the risk of injury and burnout.’

By enabling musicians to quickly isolate problematic areas in their performance, SkyNote helps them correct any technical errors at an earlier stage, leading to more efficient practice and a reduced risk of injury.

There’s no cause for RCM professors to worry just yet, though – we are unlikely to see artificial intelligence machines taking over the classroom any time soon. George says that SkyNote has been designed to support, rather than lead, teaching.

‘Our goal is not to replace or diminish the role of music teachers’, he continues. ‘On the contrary, we want to empower students to be able to develop and refine their own fundamental skills and technique in the practice room so that the precious time spent with their teachers can focus on the greater issues of interpretation and helping a student find their unique voice.’