AI Directing and Composing Video Game Soundtracks

Meet the Lab!

Left to right: Hien Tran, Todd Neller, Veysel Yilmaz, Aaron Dorsey

Hien Tran is a rising senior majoring in computer science. In his free time, he enjoys listening to music, playing games, and reading novels. Hien has always been interested in the field of AI, and he feels like this is the perfect opportunity to expand his knowledge on something that he is not familiar with, which is the relationship between AI and game soundtrack designing.

Todd Neller is a Professor of Computer Science at Gettysburg College, and was the recipient of the 2018 AAAI/EAAI Outstanding Educator Award. A game enthusiast, Neller has enjoyed pursuing game AI challenges, computing optimal play for jeopardy dice games such as Pass the Pigs and bluffing dice games such as Dudo, creating new reasoning algorithms for Clue/Cluedo, analyzing optimal Risk attack and defense policies, and designing games and puzzles.  However, Neller also greatly enjoys music from Baroque to Progressive Rock, and notes that this X-SIG project is special for its real-time aspect and broad scope for possible AI engineering and musical creativity.

Veysel Yilmaz is a double major in computer science and music. His academic interests range from video game development to sound engineering. He is specifically interested in utilizing computers in musical compositions. He most certainly does not think that AI should replace songwriters and composers, although he is interested in trying to achieve that.

Aaron Dorsey is a rising senior majoring in computer science and mathematical economics. Outside the lab, he enjoys playing video games, tabletop games, and writing creative fiction. In the lab, he mostly does data collection and processing, and works on finding the best model for the existing data. AI is very interesting to him, and he is very familiar with dynamic video game soundtracks, so this program is something he is very excited about participating in.

Human-Aware AI in Sound and Music

Artificial intelligence (AI) is a broad and constantly changing field of computer science. For our project, we want to focus on AI applications that can mimic human intelligence processes through the creation and application of algorithms built into a dynamic computing environment. Human-aware AI in Sound and Music (HAAISAM) is a broad category of research work, examples of which are automated accompaniment, sentiment recognition, interactive audio, intelligent performance tutor, and many more. Many present-day applications that use AI interact with people on a daily basis, but the AI systems do not necessarily consider the people in these interactions beyond a source of data and queries. People have motivations, preferences, emotions, and other features that affect the way they act. Sound and music are also ubiquitous for many individuals, ranging from interactions between objects in the physical world, to auditory signals and verbal communication, to means of expression and entertainment. What we hope to achieve at the end of our project is a better understanding of how to map our emotions and emotional association with music onto the user’s gameplay experience, and how AI can use sound and music to better understand and/or interact with the people around it. For this purpose, the specific genre of music that we want to focus on is game soundtracks. We want to study game soundtracks in general because we believe that the music is a crucial part to the immersion of a game. A good soundtrack can create an atmosphere to really captivate the players, whereas a bad soundtrack can be detrimental to the game experience. We believe that the integration of AI and machine learning in soundtrack development could be “game changing”. Throughout the summer, the primary goal of our project is to design a machine learning system that can dynamically create and adapt a game soundtrack to enhance a player’s experience by directing and amplifying their emotions. 

Soundtracks and Players

Although quite niche, video game soundtracks are currently a popular subject of academic research. From 8-bit style music that one could recognize from early Mario games to Hans Zimmer’s orchestral compositions for Call of Duty: Modern Warfare 2, video game soundtracks have a wide expressive range despite their recent history. One of the main points we are focusing on in our research is the feedback loop between the player and the soundtrack in which the player’s status affects and is affected by the soundtrack. Our goal is to use machine learning to reflect the player’s status through the soundtrack which in turn will create an immense gameplay by influencing the player’s status. Simply put, we want AI to influence gamers through reflective dynamic composition. (It may sound like AI controlling people, but not really…)

Bouncy Shots

Bouncy Shots (developed by DaFluffyPotato) is a simple game where the objective is to knock the ball past the AI opponent to score. Over time, though, the game gets more complicated and hectic. If enough time passes by without a point scored, additional balls will be added to the field. Also, the difficulty of the opponent rises as your score increases, and the game ends once the opponent scores.

(Here we have a score of 6, 3 balls present on the screen, and 3 projectiles – two blue, from the player, and one red, from the opponent)

The data we drew from this game were variables such as:

  • Number of balls on the screen
  • Number of projectiles on screen (the blue/red balls that are shot from the bases)
  • Player’s score
  • Average button press frequency
  • Average distance of balls from the player’s goal
  • Ratio of balls on the opponent’s side to balls on the player’s side
  • Average direction of the balls.

We collected data by playing the game on one computer while an observer next to the player monitored the intensity and winning status of their play. We then took the data from the game and the monitoring and synchronized them according to the absolute timestamp (the number of milliseconds that have elapsed since January 1, 1970). After synchronizing them, we trained the AI on the existing inputs (data from the game) and outputs (manual evaluation of the game state), and the AI started predicting future outputs from real-time inputs.

Future Work

Our initial attempt for a dynamic soundtrack was to utilize different instrumentations for different levels of intensity and winning statuses. For instance, while a high level of intensity is reflected with the brass and percussion sections, the winning status of the player is reflected with the choral section. Although the music is not being dynamically composed yet, the dynamic instrumentation is still quite effective in reflecting the player’s status through the soundtrack. With our next approach, we will aim to “compose” the soundtrack in real time by using intervals and soundwaves. To test our next approach, we wanted to experiment on a secondary game, namely Chaosman Overloaded, developed by Cheezesoft.
We wanted to try our approach with different types of games, which is why we picked a platformer game. Chaosman Overloaded is a platformer shooter game. In this game, the player must race through a dungeon filled with dangerous enemies and traps, and get to the safe door. To navigate through the dungeon, the player has the options to jump, run and shoot at the enemies. The most important aspect of this game is the chaos meter, if the meter is full, the player will lose a life. If the player touches an enemy, the meter fills up, and the meter also naturally increases as time goes by. The player has 3 lives, and there are a total of 3 levels in this game. If the player loses all 3 lives, then they will have to start over from the first level. We believe that the features of this game are suitable for dynamic soundtrack development, and we are excited to see what we can achieve in the future.

Here he have a game of Chaosman Overloaded.

We hope that you are excited to learn of the results of our project in the future. 

If you have any questions, feel free to drop a comment below.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s