Introduction
In this course, we'll cover the basics of audio synthesis for music, in particular by studying how synthesizers work. We'll also look at Web Audio and Web Midi, two W3C standards that enable computer music applications to be designed and used in Web browsers. Finally, we'll look at recent advances in AIs for the automatic generation of music from text.
First Day: Introduction to music synthesis (using Web Audio Applications), Introduction to Web Audio
Learn about Web Audio and Web Audio Modules (aka VSTs for the Web)
Try WAXML for declarative music using an HTML like language
Try the audio orchestrator by BBC Research (for distributing music on smartphones)
Second Day: Introduction to Machine Learning / Deep Learning
Course documents: see the powerpoint files on this mega link25 (2 gigabytes).
Videos:
Also worth reading: "La revanche des neurones29", an article by Dominique Cardon (in french), an excellent introduction to the 2012 AI learning revolution. The first four pages tell the story of the 18-year-old who programmed classic deep learning algorithms into a GPU and improved the object recognition record by an incredible amount, sparking a new craze for Machine Learning.
Third Day: Using generative AIs for music
To come (2024): multimodal text to XXX generators (video with voice, sound and music):
For installing AI apps locally (Mac M1-M2-M3 or gamer PC recommended) : https://pinokio.computer/50
Fourth Day: project! Compose music with the tools we've seen so far!
Assignment 1: use Web Audio tools (at least 5 different ones) for composing the Music for the Games on Web 202451 challenge (main anthem)
Assignments 2: use IAs for composing music, sound effects, visuals
- Music for a video game developed for this challenge, inspired by the old "Track & Field52" video game from the 80's, by Konami
- Main menu theme
- Level 1 theme (100m), level 2 theme (110m hurdle), level 3 theme etc.
- Game over music
- Sound effects (coin inserted, jump, steps, etc.)
- Cover of the virtual record (CD cover) created by an AI
Constraints:
- Use at least 4 different NON COMMERCIAL IA applications presented in this course (audiocraft plus, musicgen, audiogen, mustango, etc. others from this list53)
- Provide a detailed README explaining your approach (ideas, goal, choosen tool, prompts, how you adapted the prompts etc.)
- Provide all prompts + non definitive attemps in addition to the final results
- Explain how you assembled / fine tuned the results using other classic tools (Logic, etc. -videos expected here)
Due date: 25 Mars 2024, Jordan Sintes will send me the final content (URL)
Online Exam 25 Mars 2024 à 11h: a quizz about IA (second day course), to check what you understood / know from the course. Date to come, I'll send the URL and you will have to pass it all a the same time, with a limited time (40mns). You will have to read the whole course I gave this day.
Notes de bas de page
1 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Introduction
2 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#First_Day.3a_Introduction_to_music_synthesis_(using_Web_Audio_Applications).2c_Introduction_to_Web_Audio
3 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Learn_synth_basics
4 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Learn_about_Web_Audio_and_Web_Audio_Modules_(aka_VSTs_for_the_Web)
5 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Try_WAXML_for_declarative_music_using_an_HTML_like_language
6 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Try_the_audio_orchestrator_by_BBC_Research_(for_distributing_music_on_smartphones)
7 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Second_Day.3a_Introduction_to_Machine_Learning_.2f_Deep_Learning
8 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Third_Day.3a_Using_generative_AIs_for_music
9 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Fourth_Day.3a_project!_Compose_music_with_the_tools_we've_seen_so_far!
10 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Assignment_1.3a_use_Web_Audio_tools_(at_least_5_different_ones)_for_composing_the.c2.a0Music_for_the.c2.a0Games_on_Web_2024.c2.a0challenge_(main_anthem)
11 http://miageprojet2.unice.fr/Intranet_de_Michel_Buffa/MSC_Sound_Design_2023-2024_-_Introduction_to_audio_synthesis%2c_Web_Audio%2c_generative_AI_for_music#Assignments_2.3a_use_IAs_for_composing_music.2c_sound_effects.2c_visuals
12 https://learningsynths.ableton.com/
13 https://learningmusic.ableton.com/
14 https://jackschaedler.github.io/circles-sines-signals/
15 https://webaudiomodules.org/s64GtX3K/private/tutorial/
16 https://aikelab.net/websynthv2/
17 https://wam-studio.i3s.univ-cotedazur.fr/
18 https://nicroto.github.io/viktor/
19 https://codepen.io/w3devcampus/pen/YzgwxMV??editors=1010
20 https://webaudiomodules.org/demos/jariseon/logue/
21 https://docs.google.com/document/d/1HvGznI6xzsSCW-Usmv_IyB8Hqbpz9RZmbLHLW0iO6-Q/edit
22 https://sequencer.party/
23 https://waxml.org/wac24/
24 https://www.bbc.co.uk/makerbox/tools/audio-orchestrator
25 https://mega.nz/folder/j9xSjYoJ#dJ7qDRxboX3vCCe7EaZrSg
26 https://www.youtube.com/watch?v=7ell8KEbhJo
27 https://projector.tensorflow.org/
28 https://www.youtube.com/watch?v=tdelUss-5hY
29 http://miageprojet2.unice.fr/@api/deki/files/2932/=LaRevancheDesNeurones_DominiqueCardonpdf.pdf
30 https://ampedstudio.com/
31 https://sounds.studio/
32 https://www.suno.ai/
33 https://github.com/archinetai/audio-ai-timeline?tab=readme-ov-file
34 https://ai.honu.io/red/musicgen-colab
35 https://colab.research.google.com/github/camenduru/MusicGen-colab/blob/main/MusicGen_ClownOfMadness_plus_colab.ipynb
36 https://app.audiogen.co/
37 https://theresanaiforthat.com/music-creation/?ref=suggest
38 https://magenta.tensorflow.org/piano-transformer
39 https://soundraw.io/
40 https://www.rtbf.be/article/acdc-une-version-incroyable-de-back-in-black-avec-la-voix-de-bon-scott-11250345
41 http://rockmeeting.com/news/13934-ac-dc-back-in-black-avec-bon-scott-au-chant-cree-par-intelligence-artificielle
42 https://github.com/SociallyIneptWeeb/AICoverGen
43 https://github.com/ardha27/AI-Song-Cover-RVC
44 https://github.com/ardha27/AI-Song-Cover-RVC%C2%A0
45 https://www.uberduck.ai/
46 https://www.youtube.com/@ValerioVelardoTheSoundofAI/videos
47 https://aitestkitchen.withgoogle.com/tools/music-fx
48 https://www.youtube.com/watch?v=5vEnRb-KxXQ
49 https://lumiere-video.github.io/#section_text_to_video
50 https://pinokio.computer/
51 https://ds4h.univ-cotedazur.fr/games-on-web-2024-est-lance
52 https://www.youtube.com/watch?v=ozcE8qzTdHo
53 https://github.com/archinetai/audio-ai-timeline?tab=readme-ov-file