The majority of the processing of sensory information in our brains is thought to take place in the Cerebral Cortex, which contains a population of billions of neurons, each making thousands of synaptic connections with its neighbours. A single neuron can be thought of as a cell which generates a travelling spiking signal to these connected neighbours when the voltage on its membrane exceeds a certain threshold voltage, a process which is called ‘firing’. A neuron receiving several spikes simultaneously (or within a very small time-window) is likely to have its voltage pushed beyond the threshold level and will therefore in turn, send spike signals to its connected neighbours. Furthermore, connections (synapses) between most neurons which cause spiking signals tend to become potentiated and those which do not become depleted, a phenomenon known as ‘Synaptic Plasticity’. There are also many varieties of spiking behaviour; including ‘regular spiking’ and ‘bursting’ (short bursts containing many spikes very close together in time). The dynamics of millions of such adaptive, interconnected neurons thus provides an extremely rich behaviour, especially on a collective level of description [For a more comprehensive introduction, see for example Gerstner and Kistler 2002] and patterns of firing regularly occur in large groups of neurons. The following diagram, which is taken from a mathematical simulation of a group of 1000 coupled neurons [Izhikevich et.al (2004)] shows an example of such collective firing behaviour.
The neurons are numbered on the ordinate
y-axis (with neuron number 1 at the bottom, and neuron number 1000 at
the top) and the time, which runs from zero to 1000 milli-seconds (or
one second), is on the x-axis. This is therefore a simulation of one
second’s activity of this group of 1000 artificial neurons. Every
time a neuron fires a blue dot is placed on the graph at the appropriate
time on a line horizontally drawn from that particular neuron. The dots
on the graph can thus be regarded as firing ‘events’. In the particular
graph shown, because of the plasticity of the neural connections, many
of the events are centred in four bands, which appear as a pulse or
‘wave’ of spiking events in real-time.
The spiking events are indeterminate (not predictable in advance) but are certainly not random, and, as is the case with the above scenario, can be very correlated. A rhythmic pattern such as the one pictured above is very likely to be connected with the ‘polychronous’ firing of a particular group of neurons [Izhikevich et.al. (2004)], in which the firing of a particular neuron generates a sequence of events which stimulate a large number of neurons which form a closed group with the connections between these neurons being reinforced through repeated firing of the first neuron (the group of neurons fire not in synchrony but with polychrony). It is the musicality of these sequences of firing events, or rhythms (or ‘cortical songs’ [Ikegaya et.al. 2004]) which have become of great interest to a group of interdisciplinary researchers and practitioners based at the University of Plymouth, at the interdisciplinary Centre for Computer Music Research (including Eduardo Miranda, John Matthias, Jane Grant, Tim Hodgson, Andrew Prior and Kevin McCracken) and we have begun to develop a methodology in which the firing times of the neurons trigger either sonic events or instructions for performers to perform sonic (or musical) events.
The following sound recording, for example, takes as its source a recording from a microphone dragged across the pages of a book. Every time there is a sound ‘event’ from a firing neuron in the computer program, a small grain of sound (typically of the order of 100ms long) is taken at random from the source recording and placed in the output file at the time of firing:
We have developed three approaches; triggering sound grains (packets of sound within a range of duration of approximately 10-100ms [Roads 2002]) from a recorded sample every time a neuron fires (every time there is a blue dot in a diagram similar to the one above) as in the recorded example above, which we call ‘Neurogranular Sampling’ [Matthias and Miranda 2005], triggering an array of oscillators which control a granular synthesizer, which we call ‘Neurogranular Synthesis’ [Murray, Miranda and Matthias 2006] and triggering light actuators via MIDI which are interpreted via a score by live performers [Matthias and Ryan 2007], [Ryan et al. 2007].
In this article, we will concentrate on the latter approach, the use of neuron firing events to trigger musical performers, by describing the process of the composition of a work written for 24 piece string orchestra and solo violin, ‘Cortical Songs’, which we completed earlier this year and which was commissioned by the University of Plymouth for ‘Voices 2’, a festival of contemporary music which took place in Plymouth, UK in February 2007. The piece was premiered in St. Andrew’s Church, Plymouth by the Ten Tors Orchestra, conducted by Simon Ible, with John Matthias as the solo violinist [Matthias and Ryan 2007].
There were two starting points to the development of the piece, one was the idea of the song, and the other was the idea of generating melodies and textures using neuronal network firing events, which might form rhythmic patterns or ‘cortical songs’. Two early sketches of short song-like pieces gave us an indication of the kinds of harmonies and timbres we wanted to use during the development of the piece: The first piece is played on violin plus string pizzicato accompaniment (The commission was for a piece for string orchestra) and in addition some of the textural background is provided by the neurogranular sampler, briefly mentioned above, which in this case uses a similar sample above, triggering grains of sounds taken from the recording of a microphone dragged across the pages of a book. Other textural accompaniment is provided by a physical modelling synthesizer.
The second sketch recording involved a similarly written song-like melody with an ostinato-like bass-line with a rhythm inspired by, but not taken from some of the neuronal simulations we had performed using the computer and the mathematical modelling techniques:
We decided to make these sketches the basis of the first two movements of the piece and base the process of writing the third and fourth movements directly on the neuronal firing times of groups of artificial cortical neurons. We also decided to use a similar method to link all the movements to provide an underpinning theme.
Beginning with the idea that the rhythmic firings of a spiking group of neurons could initiate rhythm in the context of the previous two sketches, we began a process of the development of melodic content. The most successful experiments began with a series of notes with a performance ‘rule’ that the note should change when one of the neurons in the group fired. In the following example the notes were G C# D G# G D# D –in a particular simulation, the following melody occurred:
This process was repeated on two other occasions with different, but harmonically related notes, which resulted in the following three part harmonic structure, which in the following recording has been repeated three times:
Following this experiment we made two decisions: firstly the structures similar to that in the previous recording made an ideal framework for the overlaying of an improvisational tune -a single instrument (in this case a violin) could make sense of the structurally formed parts and knit them together in an improvisationally composed fashion. Secondly we decided that it would be desirable to have the neural firing visible in some way to performers such that any ‘rules’ pertaining to the neural firing could be instigated in real performance time, meaning that a performer would be able to know when a neuron had fired and act upon that event in a particular preconceived way (i.e a rule might be ‘change the note that you are playing’).
With this latter decision in mind, we constructed an interface between the computer program containing the mathematical/ biological spiking neuron model and a set of LED lights. The idea was to have a single LED light connected to each neuron. As there were 24 players in the string ensemble, we constructed a model with 24 neurons, each connected via a MIDI interface to an LED light. When one of the neurons fired its connected light flashed. In performance, each member of the string orchestra had a musical part to follow in addition to the flashing light, with the score containing both composed notes and instructions about what to do when the lights flashed. In order to make decisions about what the performers should do when their lights flashed, we conducted a series of musical experiments with a string quintet, made up of the desk leaders of the Ten Tors Orchestra conducted by Simon Ible.
Several of the experiments were successful in the context of the previously developed music. These included tapping the instruments on seeing a flashing light, changing notes taken from a theme from the earlier improvised violin part and the following extract taken from the tests with the quintet in which the instrumentalists were asked to play a G note and then move a quarter tone sharp when seeing a light flash for the first time, going back to the original note on seeing a second flash, going a quarter tone flat when seeing a third flash, and returning to the original note on seeing a fourth flash and so on.
The most successful parts of the experiments with the string quintet were then integrated into the full score, which comprised of four movements. The triggering patterns were unknown to the performers, as they depended on what was going on in the network at the time of performance. The recording of the performance is reproduced below, with the kind permission of the orchestra and director, Simon Ible.
The first two movements contain elements of the song structures shown earlier. The third movement is largely constructed through the rule-based score and the light triggers alone and the fourth movement is based on the initial experiment with the improvised violin, described above. A recording of the piece, including several remixes, will be released by NonClassical records in spring 2008. We hope to develop many of the processes initiated here to make new pieces and we are also keen to work with musicians and performers to develop the both software and the compositional, performance and recording processes. The Neurogranular sampler, for example, is being adapted by artist, Jane Grant with research assistant, Tim Hodgson (financially supported by the AHRC), to develop a new work.
References
Roads, C, 2002 Microsound MIT Press
Miranda, E. R. and Matthias, J.R. (2005). "Granular Sampling using a Pulse-Coupled Network of Spiking Neurons", Proceedings of EvoWorkshops 2005, Lecture Notes in Computer Science 3449, pp. 539-544. Berlin: Springer-Verlag.
Murray, J., Miranda, E. R., and Matthias, J. (2006). “Real-time granular synthesis with spiking neurons”, Proceedings of Consciousness Reframed - 8th International Conference, University of Plymouth, Plymouth (UK). (Invited talk)
Matthias J.R. and Ryan E.N. 2007 Cortical Songs -Live Performance Plymouth Feb 24th, St. Andrew’s Church, Plymouth. Ten Tors String Orchestra.
Ryan E.N. Matthias J.R. Prior A. and McCracken J.K. 2007 SEMPRE conference on computer aided composition
Izhikevich E.M., Gally J.A. and Edelman G.M. 2004 ‘Spike-timing dynamics of neuronal groups’ Cerebral Cortex 14 933-944
Ikegaya Y., Aaron G., Cossart R., Aronov D., Lampl L., Ferster D. and Yuste R. (2004), “Synfire chains and cortical songs: Temporal modules of cortical activity”, Science, 304:559-564.
Gerstner W. and Kistler W (2002) ‘Spiking Neuron Models: Single Neurons, populations and plasticity’ Cambridge University Press
John Matthias
John Matthias is a physicist, musician, composer, singer and song-writer. He has collaborated with many recording artists including Radiohead, Matthew Herbert and Coldcut and has performed extensively in Europe including at the Pompidou Centre in Paris and at the Union Chapel in London. His first solo studio album, ‘Smalltown, Shining’, was released by Accidental Records (Lifelike) in 2001 and was described by Time Out (London) as one of the first examples of a new genre of song-writing. He is releasing a second album of songs on the Ninja Tune label in spring 2008. He has also contributed as a violinist to the scores of many albums, television programs and two feature films. He is a lecturer in the Faculty of Arts at the University of Plymouth and is published by Westbury Music Ltd.
Nick Ryan
Nick Ryan is a composer, producer and sound designer. In 2004 he won a BAFTA for ‘The Dark House’, a groundbreaking interactive radio drama, broadcast live on BBC Radio 4 in September 2003. It is the first British Academy Award ever to be awarded for Radio. He has composed extensively for film and Television including Film Four Short, ‘Spin’ (winner at the Edinburgh International Film Festival and released in Curzon Cinemas) and multi-award winning 35mm short, ‘Starched’ (Best Short Film, Milan Film Festival 2003). As a music producer Nick has collaborated with many recording artists including Vanessa Mae and Youth for EMI Records. In 2003, Nick co-authored a British Government report on the future of digital music, touring the US West Coast on a DTI trade mission.