{"id":1218446,"date":"2022-10-07T15:26:57","date_gmt":"2022-10-07T15:26:57","guid":{"rendered":"https:\/\/www.ghanamma.com\/2022\/10\/07\/ai-music-generators-could-be-a-boon-for-artists-but-also-problematic\/"},"modified":"2022-10-07T15:26:57","modified_gmt":"2022-10-07T15:26:57","slug":"ai-music-generators-could-be-a-boon-for-artists-but-also-problematic","status":"publish","type":"post","link":"https:\/\/www.ghanamma.com\/2022\/10\/07\/ai-music-generators-could-be-a-boon-for-artists-but-also-problematic\/","title":{"rendered":"AI music generators could be a boon for artists – but also problematic \u2022"},"content":{"rendered":"

<\/p>\n

\n

It was only<\/span> five years ago that electronic punk band YACHT entered the recording studio with a daunting task: they would train an AI on fourteen years of their music, then synthesize the results into the album \u201cChain Tripping.\u201d<\/p>\n

\u201cI\u2019m not interested in being a reactionary,\u201d YACHT member and tech writer Claire L. Evans said in a documentary about the album. \u201cI don\u2019t want to return to my roots and play acoustic guitar because I\u2019m so freaked out about the coming robot apocalypse, but I also don\u2019t want to jump into the trenches and welcome our new robot overlords either.\u201d<\/span><\/p>\n

But our new robot overlords are making a whole lot of progress in the space of AI music generation. Even though the Grammy-nominated \u201cChain Tripping\u201d was released in 2019, the technology behind it is already becoming outdated. Now, the startup behind the open source AI image generator Stable Diffusion is pushing us forward again with its next act: making music.<\/span><\/p>\n

Creating harmony<\/h2>\n

Harmonai is an organization with financial backing from Stability AI, the London-based startup behind Stable Diffusion. In late September, Harmonai released Dance Diffusion, an algorithm and set of tools that can generate clips of music by training on hundreds of hours of existing songs.\u00a0<\/span><\/p>\n

\u201cI started my work on audio diffusion around the same time as I started working with Stability AI,\u201d Zach Evans, who heads development of Dance Diffusion, told in an email interview. \u201cI was brought on to the company due to my development work with [the image-generating algorithm] Disco Diffusion and I quickly decided to pivot to audio research. To facilitate my own learning and research, and make a community that focuses on audio AI, I started Harmonai.\u201d<\/span><\/p>\n

Dance Diffusion remains in the testing stages \u2014 at present, the system can only generate clips a few seconds long. But the early results provide a tantalizing glimpse at what could be the future of music creation, while at the same time raising questions about the potential impact on artists.<\/span><\/p>\n

\n

Image Credits:<\/strong> DALL-E 2 \/ OpenAI<\/p>\n<\/div>\n

The emergence of Dance Diffusion comes several years after OpenAI, the San Francisco-based lab behind DALL-E 2, detailed its grand experiment with music generation, dubbed Jukebox. Given a genre, artist and a snippet of lyrics, Jukebox could generate relatively coherent music complete with vocals. But the songs Jukebox produced lacked larger musical structures like choruses that repeat, and often contained<\/span> nonsense lyrics<\/span>.\u00a0<\/span><\/p>\n

Google\u2019s AudioLM, <\/span>detailed<\/span> for the first time earlier this week, shows more promise, with an uncanny ability to generate piano music given a short snippet of playing. But it hasn\u2019t been open sourced.\u00a0<\/span><\/p>\n

Dance Diffusion aims to overcome the limitations of previous open source tools by borrowing technology from image generators such as Stable Diffusion. The system is what\u2019s known as a diffusion model, which generates new data (e.g., songs) by learning how to destroy and recover many existing samples of data. As it\u2019s fed the existing samples \u2014 say, the entire Smashing Pumpkins discography \u2014 the model gets better at recovering all the data it had previously destroyed to create new works.<\/span><\/p>\n

Kyle Worrall, a Ph.D. student at the University of York in the U.K. studying the musical applications of machine learning, explained the nuances of diffusion systems in an interview with :\u00a0\u00a0<\/span><\/p>\n

\u201cIn the training of a diffusion model, training data such as the <\/span>MAESTRO data set<\/span> of piano performances is \u2018destroyed\u2019 and \u2018recovered,\u2019 and the model improves at performing these tasks as it works its way through the training data,\u201d he said via email. \u201cEventually the trained model can take noise and turn that into music similar to the training data (i.e., piano performances in MAESTRO\u2019s case). Users can then use the trained model to do one of three tasks: Generate new audio, regenerate existing audio that the user chooses, or interpolate between two input tracks.\u201d<\/span><\/p>\n

It\u2019s not the most intuitive idea. But as DALL-E 2, Stable Diffusion and other such systems have shown, the results can be remarkably realistic.<\/span><\/p>\n

For example, check out this Disco Diffusion model fine-tuned on Daft Punk music:<\/p>\n


Or this style transfer of the Pirates of the Caribbean theme to flute:<\/p>\n

Or this style transfer of Smash Mouth vocals to the Tetris theme (yes, really):<\/p>\n

Or these models, which were fine-tuned on copyright-free dance music:<\/p>\n

Artist perspective<\/h2>\n

Jona Bechtolt of YACHT was impressed by what Dance Diffusion can create.<\/span><\/p>\n

\u201cOur initial reaction was like, \u2018Okay, this is a leap forward from where we were at before with raw audio,\u2019\u201d Bechtolt told . <\/span><\/p>\n

Unlike popular image-generating systems, Dance Diffusion is somewhat limited in what it can create \u2014 at least for the time being. While it can be fine-tuned on a particular artist, genre or even instrument, the system isn\u2019t as general as Jukebox. The handful of Dance Diffusion models available \u2014 a hodgepodge from Harmonai and early adopters on the official Discord server, including models fine-tuned with clips from Billy Joel, The Beatles, Daft Punk and musician Jonathan Mann\u2019s Song A Day\u00a0project \u2014 stay within their respective lanes. That is to say, the Jonathan Mann model always generates songs in Mann\u2019s musical style.<\/span><\/p>\n

And Dance Diffusion-generated music won\u2019t fool anyone today. While the system can \u201cstyle transfer\u201d songs by applying the style of one artist to a song by another, essentially creating covers, it can\u2019t generate clips longer than a few seconds in length and lyrics that aren\u2019t gibberish (see the below clip). That\u2019s the result of technical hurdles Harmonai has yet to overcome, says Nicolas Martel, a self-taught game developer and member of the Harmonai Discord.<\/span><\/p>\n

\u201cThe model is only trained on short 1.5-second samples at a time so it can\u2019t learn or reason about long-term structure,\u201d Martel told . \u201cThe authors seem to be saying this isn\u2019t a problem, but in my experience \u2014 and logically anyway \u2014 that hasn\u2019t been very true.\u201d<\/span><\/p>\n

YACHT\u2019s Evans and Bechtolt are concerned about the ethical implications of AI \u2013 they are working artists, after all \u2013 but they observe that these \u201cstyle transfers\u201d are already part of the natural creative process.<\/span><\/p>\n

\"Dance<\/p>\n

Image Credits:<\/strong> DALL-E 2 \/ OpenAI<\/p>\n<\/div>\n

\u201cThat\u2019s something that artists are already doing in the studio in a much more informal and sloppy way,\u201d Evans said. \u201cYou sit down to write a song and you\u2019re like, I want a Fall bass line and a B-52\u2019s melody, and I want it to sound like it came from London in 1977.\u201d<\/span><\/p>\n

But Evans isn\u2019t interested in writing the dark, post-punk rendition of \u201cLove Shack.\u201d Rather, she thinks that interesting music comes from experimentation in the studio \u2013 even if you take inspiration from the B-52\u2019s, your final product may not bear the signs of those influences.\u00a0<\/span><\/p>\n

\u201cIn trying to achieve that, you fail,\u201d Evans told . \u201cOne of the things that attracted us to machine learning tools and AI art was the ways in which it was failing, because these models aren\u2019t perfect. They\u2019re just guessing at what we want.\u201d\u00a0<\/span><\/p>\n

Evans describes artists as \u201cthe ultimate beta testers,\u201d using tools outside of the ways in which they were intended to create something new.\u00a0<\/span><\/p>\n

\u201cOftentimes, the output can be really weird and damaged and upsetting, or it can sound really strange and novel, and that failure is delightful,\u201d Evans said.\u00a0<\/span><\/p>\n

Ethical consequences<\/h2>\n

Assuming Dance Diffusion one day reaches the point where it can generate coherent whole songs, it seems inevitable that major ethical and legal issues will come to the fore. They already have, albeit around simpler AI systems. In 2020, Jay-Z \u2018s record label filed copyright strikes against a YouTube channel, Vocal Synthesis, for using AI to create Jay-Z covers of songs like Billy Joel\u2019s \u201cWe Didn\u2019t Start the Fire.\u201d After initially removing the videos, YouTube reinstated them, finding the takedown requests were \u201cincomplete.\u201d But deepfaked music still stands on murky legal ground.<\/span><\/p>\n

Perhaps anticipating legal challenges, OpenAI for its part open-sourced Jukebox under a <\/span>non-commercial license<\/span>, prohibiting users from selling any music created with the system.\u00a0<\/span><\/p>\n

\u201cThere is little work into establishing how original the output of generative algorithms are, so the use of generative music in advertisements and other projects still runs the risk of accidentally infringing on copyright, and as such damaging the property,\u201d Worrall said. \u201cThis area needs to be further researched.\u201d<\/span><\/p>\n

An academic<\/span> paper<\/span> authored by Eric Sunray, now a legal intern at the Music Publishers Association, argues that AI music generators like Dance Diffusion violate music copyright by creating \u201ctapestries of coherent audio from the works they ingest in training, thereby infringing the United States Copyright Act\u2019s reproduction right.\u201d Following the release of Jukebox, critics have also questioned whether training AI models on copyrighted musical material constitutes fair use. Similar concerns have been raised around the training data used in image-, code-, and text-generating AI systems, which is often scraped from the web without creators\u2019 knowledge.<\/span><\/p>\n

Technologists like Mat Dryhurst and Holly Herndon founded Spawning AI, a set of AI tools built for artists, by artists. <\/span>One of their projects, \u201cHave I Been Trained,\u201d allows users to search for their artwork and see if it has been incorporated into an AI training set without their consent.\u00a0<\/span><\/p>\n

\u201cWe are showing people what exists within popular datasets used to train AI image systems, and are initially offering them tools to opt out or opt in to training,\u201d Herndon told via email. \u201cWe are also talking to many of the biggest research organizations to convince them that consensual data is beneficial for everyone.\u201d<\/span><\/p>\n

\"Dance<\/p>\n

Image Credits:<\/strong> DALL-E 2 \/ OpenAI<\/p>\n<\/div>\n

But these standards are \u2014 and will likely remain \u2014 voluntary. Harmonai hasn\u2019t said whether it\u2019ll adopt them.<\/span><\/p>\n

\u201cTo be clear, Dance Diffusion is not a product and it is currently only research,\u201d said Zach Evans of Stability AI. \u201cAll of the models that are officially being released as part of Dance Diffusion are trained on public domain data, Creative Commons-licensed data, and data contributed by artists in the community. The method here is opt-in only and we look forward to working with artists to scale up our data through further opt-in contributions, and I applaud the work of Holly Herndon and Mat Dryhurst and their new Spawning organization.\u201d<\/span><\/p>\n

YACHT\u2019s Evans and Bechtolt see parallels between the emergence of AI generated art and other new technologies.<\/span><\/p>\n

\u201cIt\u2019s especially frustrating when we see the same patterns play out across all disciplines,\u201d Evans told . \u201cWe\u2019ve seen the way that people being lazy about security and privacy on social media can lead to harassment. When tools and platforms are designed by people who aren\u2019t thinking about the long term consequences and social effects of their work like that, things happen.\u201d\u00a0<\/span><\/p>\n

Jonathan Mann \u2014 the same Mann whose music was used to train one of the early Dance Diffusion models \u2014 told that he has mixed feelings about generative AI systems. While he believes that Harmonai has been \u201cthoughtful\u201d about the data they\u2019re using for training, others like OpenAI have not been as conscientius.<\/span><\/p>\n

\u201cJukebox was trained on thousands of artists without their permission \u2014 it\u2019s staggering,\u201d Mann said. \u201cIt feels weird to use Jukebox knowing that a lot of folks\u2019 music was used without their permission. We are in uncharted territory.\u201d<\/span><\/p>\n

From a user perspective, Waxy\u2019s Andy Baio speculates in a <\/span>blog post<\/span> that new music generated by an AI system would be considered a derivative work, in which case only the original elements would be protected by copyright. Of course, it\u2019s unclear what might be considered \u201coriginal\u201d in such music. Using this music commercially is to enter uncharted waters. It\u2019s a simpler matter if generated music is used for purposes protected under fair use, like parody and commentary, but Baio expects that courts would have to make case-by-base judgements.<\/span><\/p>\n

\"Dance<\/p>\n

Image Credits:<\/strong> DALL-E 2 \/ OpenAI<\/p>\n<\/div>\n

According to Herndon, copyright law is <\/span>not structured<\/span><\/a> to adequately regulate AI art-making. Evans also points out that the music industry has been historically more litigious than the visual art world, which is perhaps why Dance Diffusion was explicitly trained on a dataset of copyright-free or voluntarily-submitted material, while DALL-E mini will easily spit out a Pikachu if you input the term \u201cPok\u00e9mon.\u201d<\/span><\/p>\n

\u201cI have no illusion that that\u2019s because they thought that was the best thing to do ethically,\u201d Evans said. \u201cIt\u2019s because copyright law in music is very strict and more aggressively enforced.\u201d<\/span><\/p>\n

Creative potential<\/h2>\n

Gordon Tuomikoski, an arts major at the University of Nebraska-Lincoln who moderates the official Stable Diffusion Discord community, believes that Dance Diffusion has immense artistic potential. He notes that some members of the Harmonai server have created models trained on dubstep \u201cwebs,\u201d kicks and snare drums and backup vocals, which they\u2019ve strung together into original songs.<\/span><\/p>\n

\u201cAs a musician, I definitely see myself using something like Dance Diffusion for samples and loops,\u201d Tuomikoski told via email.<\/span><\/p>\n

Martel sees Dance Diffusion one day replacing VSTs, the digital standard used to connect synthesizers and effect plugins with recording systems and audio editing software. For example, he says, a model trained on \u201970s jazz rock and Canterbury music will intelligently introduce new \u201ctextures\u201d in the drums, like subtle drum rolls and \u201cghost notes,\u201d in the same way that artists like <\/span>John Marshall<\/span> might \u2014 but without the manual engineering work normally required.\u00a0\u00a0\u00a0<\/span><\/p>\n

Take this Dance Diffusion model of Senegalese drumming, for instance:<\/p>\n

And this model of snares:<\/p>\n

And this model of a male choir singing in the key of D across three octaves:<\/p>\n

And this model of Mann\u2019s songs fine-tuned with royalty-free dance music:<\/p>\n

\u201cNormally, you\u2019d have to lay down notes in a MIDI file and sound-design really hard. Achieving a humanized sound this way is not only very time-consuming, but requires a deeply intimate understanding of the instrument you\u2019re sound designing,\u201d Martel said. \u201cWith Dance Diffusion, I look forward to feeding the finest \u201970s prog rock into AI, an infinite unending orchestra of virtuoso musicians playing Pink Floyd, Soft Machine and Genesis, trillions of new albums in different styles, remixed in new ways by injecting some Aphex Twin and Vaporwave, all performing at the peak of human creativity \u2014 all in collaboration with your own personal tastes.\u201d<\/span><\/p>\n

Mann has greater ambitions. He\u2019s currently using a combination of Jukebox and Dance Diffusion to play around with music generation, and plans to release a tool that\u2019ll allow others to do the same. But he hopes to one day use Dance Diffusion \u2014 possibly in conjunction with other systems \u2014 to create a \u201cdigital version\u201d of himself capable of continuing the Song A Day project after he passes away.<\/span><\/p>\n

\u201cThe exact form it\u2019ll take hasn\u2019t quite become clear yet \u2026 [but] thanks to folks at Harmonai and some others I\u2019ve met in the Jukebox Discord, over the last few months I feel like we\u2019ve made bigger strides than any time in the last four years,\u201d Mann said. \u201cI have over 5,000 Song A Day songs, complete with their lyrics as well as rich metadata, with attributes ranging from mood, genre, tempo, key, all the way to location and beard (whether or not I had a beard when I wrote the song). My hope is that given all this data, we can create a model that can reliably create new songs as if I had written them myself. A Song A Day, but forever.\u201d<\/span><\/p>\n

If AI can successfully make new music, where does that leave musicians?<\/span><\/p>\n

YACHT\u2019s Evans and Bechtolt point out that new technology has upended the art scene before, and the results weren\u2019t as catastrophic as expected. In the 1980s, the UK Musicians Union attempted to ban the use of synthesizers, arguing that it would replace musicians and put them out of work.<\/span><\/p>\n

\u201cWith synthesizers, a lot of artists took this new thing and instead of refusing it, they invented techno, hip hop, post punk and new wave music,\u201d Evans said. \u201cIt\u2019s just that right now, the upheavals are happening so quickly that we don\u2019t have time to digest and absorb the impact of these tools and make sense of them.\u201d<\/span><\/p>\n

Still, YACHT worries that AI could eventually challenge work that musicians do in their day jobs, like writing scores for commercials. But like Herndon, they don\u2019t think AI can quite replicate the creative process just yet.<\/span><\/p>\n

\u201cIt is divisive and a fundamental misunderstanding of the function of art to think that AI tools are going to replace the importance of human expression,\u201d Herndon said. \u201cI hope that automated systems will raise important questions about how little we as a society have valued art and journalism on the internet. Rather than speculate about replacement narratives, I prefer to think about this as a fresh opportunity to revalue humans.\u201d<\/span><\/p>\n<\/div>\n