Andrew Hainen, Nikhil Mangla, Syed Wajahat Karim
Our project is a machine learning application that learns what proper emotion to associate with a song based on the characteristics of the song.
Our idea was to have a computer react in a realistic fashion to music in a way that a human would react. With a human, if I’m walking down the street listening to my iPod and “Twist and Shout” by the Beatles starts playing, then I’ll hypothetically be happier, more upbeat, and cheerful. Oppositely, if “Smells Like Teen Spirit” by Nirvana comes on my iPod, then I might be more depressed, slow-walking, and down.
For a computer, we wanted to illicit these responses based on the certain types of songs it heard.
To accomplish out concept, we are utilizing:
- A machine learning experience based on jMIR’s jAudio library and the Weka library
- GIF Animations
Primarily, we split a song of our choice into 10 second clips. Then, we run the clips through jAudio, which calculates 78 different numbers based on characteristics of the song. We then take those numbers, which have been compiled into a .arff file for the Weka Machine Learning Library, and use that to compare them. Once the the numbers have been compared, the program spits out “Sad” or “Happy”. From there, Processing.org will play a simple animation that is either “Sad” or “Happy”.
Final Output / Experience
Overall, we saw that the music data was very different in the “happy” and “sad” areas, which is what we were hoping for. There were approximately 78 paramaters the songs were being judged upon, most of which we were unsure of what they were measuring, but relied more so on the technical achievements of the jAudio Application.
Regardless of the data being quite different for each song, our data had mixed results. If a song we “thought of” as happy had a mild moment or didn’t meet all the requirements the program labeled as “happy”, then it would go to sad, even if it wasn’t a sad part. This “black or white” idea of categorizing music wasn’t the best implementation, but had to be relied upon for times’ sake.
Our original code included a “neutral” category of songs, but was skewing our results further than if we just had “happy” or “sad”. Furthermore, the concept of categorizing music into emotional categories is entirely opinion and debatable, and humans have learned how to do so over the course of their lives. We feel that using many more test samples for the program to learn our ideas of “happy”, “sad”, or “neutral” would greatly benefit the outcomes and results.
Our final product can be seen in the areas below:
- The happy song test, based on Huey Lewis and the News “The Power of Love”: http://andrewhainen.com/adaptiveart/happy.mp4
- The sad song test, based on Dave Matthew’s Band’s “Gravedigger”: http://andrewhainen.com/adaptiveart/sad.mp4
- Our source folder: http://www.andrewhainen.com/adaptiveart/emotionMusic.zip (14.7MB and requires the gifAnimation Library for Processing.org and the Wekaizing Library for Processing.org)
- Our source code (PDE file only): http://www.andrewhainen.com/adaptiveart/emotionMusic.pde
Sources / Inspirations
- Bright Side of the Road (Van Morrison)
- Twist and Shout (The Beatles)
- Tighten Up ( The Black Keys)
- The Joker (Steve Miller Band)
- Hurt (Johnny Cash)
- Rape Me (Nirvana)
- Learning How To Smile (Everclear)
- Stan (Eminem)
- Don’t Stop Believing (Journey)
- Behind Blue Eyes (The Who)
- Gravedigger (Dave Matthews’ Band)
- The Power of Love (Huey Lewis & The News)