Join us on a journey through the rich, and at times contested, history of computer music, explored by Atau Tanaka, Professor of Media Computing at Goldsmiths to accompany our latest Web3 exhibition, Sonic Alchemy: Exploring the Art of Generative Sound.
“One could say that all music is generative. The melodies in our heads must come from somewhere in our subconscious. A modular synth patch can be generative. But for any music to be compelling, it needs an idea and creative purpose.”
Sound has always preceded the visual in harnessing new technologies. Music is mathematical so lends itself to computational representation. Audio as datastream requires less data than the 2D matrices of visual image, meaning that it was digitised, processed and manipulated decades before video. Is music always a harbinger for a technologized cultural form? Where does music find its place in today’s cultures of code? How might musical form become part of the debates surrounding the blockchain? Today, as the European Parliament debates a new AI Act to regulate discriminatory misuses of artificial intelligence (1), we reflect critically on the concept of the generative, in music, and more broadly in art.
Signal vs. Symbol
Generative music traces its history to the beginning of computer music itself. While Max V. Mathews was inventing the notion of digital audio in 1956 at Bell Laboratories, the composer Lejaren Hiller, in 1957 wrote computer code to compose Illiac Suite for string quartet (2,3). We thus encounter the dual nature of music: sound signal vs. symbolic representation.
Fast forward to 1986: IBM announce a scientist has programmed a mainframe computer to generate chorales in the style of Bach (4). In fact, it was Kemal Ebcioğlu, a music student at SUNY Buffalo (where Hiller was professor), who upon graduation became researcher at IBM’s T. J. Watson Research Center (5). Why would IBM publish Ebcioğlu’s thesis as a technical report? Did the corporate world see potential for generative music, commercial or otherwise?
Around the same time in California, the composer David Cope began creating his EMI (Experimentals in Musical Intelligence) system (6). He used the LISP programming language to model the music of composers such as Bach, Mozart and Rachmaninov and generate music true to their style as MIDI files – a digital symbolic representation. He released CDs, some that were subject to musical Turing Tests (7)– where listeners thought the machine generated version was Bach. We confront the question of imitation and perfection – was the ‘perfect’ Bach-like music musically satisfying for the listener, or are the wrinkles and at times awkward dissonant passages in Bach what fascinate us in his human genius?
In the early 2000s, my colleague at Sony Computer Science Labs, François Pachet developed the Continuator, a system to model jazz improvisation (8). He used Markov Models to create a conversational system that ‘listened’ to human input – a jazz lick or improvisational flourish – then ‘responded’ with a phrase in the same style. Not a copy, same, but different. This recalled the musical traditions of Greek antiphony, of call and response.
After Sony, François became director of Spotify’s Creator Technology Research Lab. The press speculated that Spotify was creating AI to replace human musicians (9), much in the way Uber was working on self-driving cars to obviate its fleet of human drivers. Here we need to critically differentiate the creative potential of any technology from exploitation by platform capitalism. Pachet was the first to say that his algorithms were nowhere near replacing the human musician (despite his releasing the supposed ‘first album composed with AI’, Hello World). He sees AI as a capable songwriting partner, and worked with a range of high profile musicians including Belgian rapper, Stromae. In a recent Tweet, Pachet declared, “Mass production of creative artefacts, with or without AI, is an oxymoron.”
Generative AI x Blockchain
Deep learning techniques use large datasets to create new information that has likeness to another. This was applied to audio data directly in DeepMind’s WaveNet in 2016 where speech synthesis is carried out using a Convolutional Neural Network. With neural synthesis the same technique can be used to directly synthesise audio. At Goldsmiths, colleagues have developed network bending techniques where designers intervene inside the latent space of a generative adversarial network (GAN) to produce new sound (10).
If neural synthesis represents machine learning of sound as signal, where do we find AI generation of symbolic music? With ChatGPT, OpenAI turbocharged a chatbot with a large language model (LLM) to make a formidable and highly contested prompt-based writing system. DALL-E 2 from the same company takes natural language input to a diffusion model, a probabilistic technique to generate a specific image in an information space of all images. Competitor Stability AI has open-sourced its Stable Diffusion neural network, leading to a range of code forks, including one for music. Riffusion modified the model to generate music where a prompt describing music generates an image that is the frequency spectrogram of the imagined sound. Textual representation then creates the picture of signal to be rendered sound. The artist, patten earlier this year released Mirage FM, ‘the first album’ composed using Riffusion. Listening to Mirage FM is cognitively exhausting – while human agency has been exercised in compiling the album, the barrage of sound – imagine music resembling a surreal DALL-E picture – makes us wonder whether this is the sound of the uncanny valley.
Blockchains and NFTs have captured the attention of the art world with the promise of digital scarcity, triggering debates on speculative artistic value and real concerns on the ecological impact of minting tokens. Musicians have experimented with the blockchain as a platform not just for distribution but as a distributed ledger system for sales, merchandise, touring, and production. The singer/songwriter Imogen Heap founded Mycelia for Music, making a Creative Passport on the blockchain for musicians, labels, sound engineers and roadies to have a platform for egalitarian economic exchange. New York trombone player Benny Conn created the Beat Foundry, an NFT platform for on-chain generative music. However, after a public launch in 2017, @mycelia4music has not tweeted since 2020. Beat Foundry hit the scene in 2022, but its website is already offline one year later. What happens to music on the blockchain if perennity of the underlying infrastructure is not assured?
Code & Corps
Creative coding facilitates the procedural generation of sound and visuals (11). The practice enables artists, like the four in Sonic Alchemy, to script interactions and create generative audiovisual works. Alida Sun set herself the challenge to produce a new piece of generative art every day resulting in a series of over 1,533 sketches. Boreta and Aaron Penne’s Rituals on Art Blocks runs code in the browser in a manner where its output will not repeat for over 9 million years. In Sonic Alchemy, a new artist will be highlighted each week, starting with Alida Sun, followed by Boreta, Joëlle Snaith and finally, Elias Jarzombek. Sun and Boreta’s pieces are sold as unique works – there will be one owner – while Snaith’s piece is a series of seven, each in a limited edition of 5. The notion of generative becomes but an echo of the process at the root of each work.
Alida Sun’s Synesthesia opens the exhibition. The piece builds upon her four year practice of generating a new generative work each day and embodied interaction with creative audiovisual systems. We see a series of lines, like those on a musical staff, with glowing colour shapes. We hear delicate, percussive digital tones sonifying the pulsing and movement of the shapes. What we don’t see is the artist’s body that is input to the system, generating the movement of sound and image. The body is mapped to a Cartesian grid, triggering events in the horizontal, vertical space of the screen. The corporeal thus generates the symbolic, recalling important early work: Sergi Jordá’s ‘sonigraphic instrument’, FMOL (1995) and before it, computer music pioneer Laurie Spiegel’s seminal Music Mouse – An Intelligent Instrument (1986). Where Music Mouse opened up X-Y interaction for non-musicians via the then new mouse and FMOL facilitated collective composition along a thin line, Alida Sun’s Synesthesia projects the author’s body into the abstract space of pixel coordinates and sonic frequencies.
Commodity Form or Noise
We arrive at critical questions of form, collaboration, and commodity. The distribution of generative audiovisual art in the form of executable code enables an artwork to retain its original evolutive concept. This may be based on numerical process, as in Rituals, or it may invite the spectator to interact to create their own ‘version’ of a piece. In the latter, we can think of viewer as ‘collaborator’ in a procedural work, and the minting of an NFT as establishing an incipient form of shared ownership between artist and spectator. On the other hand, when rendered, the generative process is captured at a moment in space-time, freezing the procedural work to become an objet d’art. We revert from Umberto Eco’s Open Work (12) to an older conception of fixity. We rewind Baudrillard’s Simulations (13) where the original begot the copy to give rise to the model in an attempt to imbue traditions of originality to model-based explorations. The NFT here serves to reduce the generative to the fixed, imposing on something inherently reproducible (data and process) commodity form.
We close by interrogating the nature of the generative, and the generative potential of any music – be it digital or acoustic, algorithmic or manual. Jacques Attali, in Noise (14), imagined an evolution of the political economy of music across history from ritual sacrifice through représentation (performance) to répétition (recording) arriving at composition (proto-procedural). Is generative music the arrival of Attali’s composition? An algorithm can generate minimal techno ad infinitum, but can it read the energy of the dance floor like a good DJ? How do we engage productively and critically with the generative musical act?
One could say that all music is generative. The melodies in our heads must come from somewhere in our subconscious. A modular synth patch can be generative. But for any music to be compelling, it needs an idea and creative purpose. If a concept is strong, the music makes sense. Generative music is not an exception. What do we put into it? What is the prompt? What is the seed for the generative process? How do we as humans intervene, through editing, mixing, and making apparent machine processes in their intra-actions with our sonic imagination?
Aknowledgements
This piece has come out of conversations with collaborators, conspirators and colleagues: Atay Ilgun, Evan Raskob, Rachel Falconer, Dane Law and RKB Vitesse.
Atau Tanaka
is an electronic music pioneer who studied under Ivan Tcherepnin (brother of modular synth designer Serge), John Chowning (inventor of FM synthesis) and Max Mathews (American pioneer of computer music). With a background in research at IRCAM Centre Pompidou, Apple France, and Sony Computer Science Laboratory Paris, he co-founded Sensorband in the 1990s and has worked with artists including Lillevan, Dane Law and Cécile Babiole. He has released music on labels such as Superpang and Touch, performed at festivals including Ars Electronica and has exhibited at Eyebeam, the Musikinstrumenten-Museum Berlin and SFMOMA. He is professor at Goldsmiths University and works with Bristol Interaction Group and MSH Paris Nord.
Footnotes
1. EU AI Act: First Regulation on Artificial Intelligence, https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence.
2. Nick Collins and Andrew R. Brown, ‘Generative Music’, Contemporary Music Review 28(1). 2009.
3. Roger T. Dean and Alex McLean, The Oxford Handbook of Algorithmic Music (Oxford University Press, 2018).
4. ‘Researcher Uses Bits, Bytes and Bach for Program of Note’, Los Angeles Times, 20 August 1988, https://www.latimes.com/archives/la-xpm-1988-08-20-mn-622-story.html.
5. Kemal Ebcioğlu, ‘An Expert System for Harmonizing Four-Part Chorales’, Computer Music Journal 12(3), 1988.
6. David Cope, Experiments in Musical Intelligence, (A-R Editions, 1996).
7. Christopher Ariza, ‘The Interrogator as Critic: The Turing Test and the Evaluation of Generative Music Systems’, Computer Music Journal 33(2), 2009.
8. François Pachet, ‘The Continuator: Musical Interaction With Style’, Journal of New Music Research 32(3), 2003.
9. ‘Spotify Wants To Hook Users On AI Music Creation Tools’, Forbes, 29 June 2022. https://www.forbes.com/sites/williamhochberg/2022/06/29/spotify-is-developing-ai-tools-to-hook-users-on-music-creation/.
10. Matthew Yee-King and Louis McCallum, ‘Studio Report: Sound Synthesis with DDSP and Network Bending Techniques’, Conference on AI Music Creativity (MuMe + CSMC), 2021.
11. Steve Gibson, Stefan Arisona, Donna Leishmann, and Atau Tanaka, Live Visuals: History, Theory, Practice (Routledge, 2022).
12. Umberto Eco, The Open Work (Harvard University Press, 1989).
13. Jean Baudrillard, Simulations (MIT Press, 1983).
14. Jacques Attali, Noise: The Political Economy of Music, (University of Minnesota Press, 1985).