The lab has two papers at the conference this year:
Chenyu Gao, Federico Reuben, and Tom Collins. Variation Transformer: New datasets, models, and comparative evaluation for symbolic music variation generation.
Rajesh Fotedar and Tom Collins. Analysis of the originality of Gen-AI song audio.
The 2024 AI Song Contest
5th October, 2024. We opened the 2024
AI Song Contest
at Innovation Park Zürich
with a live performance of our song called “Heart not
found” by Error 305.
This lab works on basic science – development and testing of
artificial intelligence for music analysis and generation –
through to studies of application and societal impact – artist
collaborations and contests such as this. Our
$100K Concerts with Humans and Artificial Intelligence (CHAI)
project,
funded by
U-LINK at
University of Miami,
is supporting a lot of this applied work.
L to R:
Ryan Baker,
Eli Yaroch,
Dr. Raina Murnak,
Spencer Soule,
Dr. Tom Collins
L to R:
Amanda Pasler and
Dr. Raina Murnak
“Heart not found” by Amanda Pasler, Spencer Soule, Ryan Baker, Jack Reilly, Eli Yaroch, Raina Murnak, Aditya Jeganath K., and Tom Collins
Latest research projects in lab
9th May, 2024. The following research projects have just got underway or are starting next month. Follow the links to read more about them!
SAMPA (Synchronized Action in Music-ensemble Playing and Athletics)
CHAI (Concerts with Humans and Artificial Intelligence)
FilmGate Fusion at Frost Science
11th April, 2024. Tom was at the Frost Science Museum on Thursday evening for
FilmGate Fusion,
presenting the
VR version
(best experienced with a Meta Quest)
of
VTGO (Vertigo).
There is also a
hackathon
taking place all day Sat 10th Feb with generous cash prizes!
VTGO (Vertigo) on general release today!
29th November, 2023. VTGO (Vertigo), a collaboration with
Kemi Sulola and
Harriet Raynor,
is out today! See
here
to listen to it on your platform of choice.
Chenyu's contribution is about interactive pendular graphs, which
can be explored here.
NoiseBandNet: Controllable, time-varying neural synthesis of sound effects using filterbanks
18th July, 2023. Lab PhD student
Adrián Barahona-Ríos
has been working on sound effect modelling and synthesis. The new
model, indicated in the diagram above and called
NoiseBandNet,
has some exciting creative applications. For example, once trained on a
sound like this metal impact:
we can drive the timbral world of metal impact with the loudness
curve extracted from another sound, such as this beat box clip:
resulting in an interesting hybrid:
In addition to the music-creative possibilities indicated above,
we anticipate applications in game audio and XR, where until now it
has been labour-intensive to generate scene-driven alterations to
sound effects.
10th April, 2020. "It's doing what I hoped it would do, which is
what I believe AI will do for musicians, which is to push us to the
next level of our own creativity"
(Imogen Heap, Grammy Award-winning
musican and technologist on working with music generation algorithms
built in the lab).
12 April, 2021. Here are some fun, educational interfaces for kids
to explore music technology. We have found some of these interfaces
engage kids as young as two years old. For kids aged four and above,
up to you whether to explore side-by-side with them, or say farewell
to your device and let them have at it...
Creating music is about composing sounds. Sometimes the possibilities
are overwhelming, so why not explore the 625 possibilities of the
Sample selector?!
Colouring with keys
(select "Play own stuff", hit start, and use A, S, D..., W, E,... keys
to explore making different colours with different major/minor keys)
Chrome Music Lab
(Spectrogram, Voice spinner, Rhythm, Oscillators, and Song maker
are Tom's four-year-old's favourites).
New to (web) programming?
Learning JavaScript is a good place to start. I've put some demos
below to get you excited about wanting to do this!
Read more...
Demos
Here are some examples of dynamic web-based music interfaces that have
been developed in the lab, using packages built on the
Web Audio API.
For pedagogical purposes, we have used mostly basic JavaScript, left
comments in the code, and avoided optimizations like minifying. If
you make JSFiddles, CodePens, or your own standalone interfaces
based directly or indirectly on what you find below, please feel
free to share them with us to enhance the pedagogical
experience!
New to (web) programming?
To rework/extend the demos, you'll need to understand how to
program in HTML, CSS, JavaScript, PHP, and Node.js, with the most
important of these being JavaScript.
Read more...
Jobs
In the interests of closing the gender pay gap, my salary as an Associate Professor at a private institution is $114K.
We like hearing from people who are interested in contributing to the
work of the lab. At the moment, we're particularly interested in hearing
from people with software engineering expertise who are looking for more
autonomy and to learn some research skills.
Feel free to
get in touch
if you fit any of the categories below.
Full-stack JavaScript developer
Familiarity with Node.js and SQLite or another database solution.
Experience with client-side syntax/library such as Handlebars, React, or Vue.
Ideal candidate has
experience with building single-page web applications from API to
UI/X with authorization and access control.
Grad/postdoc
First-author publications and/or evidence of writing productivity
appropriate to level. Experience as a music scholar, a computer
scientist, a cognitive scientist, or some combination of the three.
Demonstration of willingness to work on and optimize time-consuming
tasks such as data collection and analysis, and music data curation.
Undergrad
Interest in music, computer science, cognitive science, or some
combination of the three. Has looked at the
demos
and attempted to rework/extend at least one of them.
About
Both in the lab and with collaborators across the globe, we apply the
scientific method to explore...
The
Web Audio API
and resultant possibilities for musical creation, consumption, and
collaboration
Machine learning applied to music and game audio, including but not limited to
automatic generation of stylistic compositions, incorporation in
software, and the technology's effect on users and their work
Discovery of repeated patterns and patterns of successful
coordination in music, visual, and sporting domains
NLP
and
NLU
for editing and querying music scores (given a query like
'perfect cadence followed by homophonic texture', retrieve the relevant
events from a digital score)
Musical expectancy and listening choices (for symbolic/audio
input and different listener backgrounds/contexts)
Team
This is a team of researchers that we hope will grow in exciting ways
over the next decades – even beyond the current PI's retirement!
If you are interested in working with us, you are welcome to
get in touch to discuss
opportunities. We're happy to try to support face-to-face research
visits and/or distributed collaborations.
Here's the team, present and past, in pseudo-random order...
Members
See here,
for the CHAI team, which consists of ~15 members.
Tom Collins, principal investigator, with interests including (but
not limited to) the development and impact of Web-based music
software; machine learning applied to music; pattern discovery in
music and other domains; automatic identification of high-level
music-theoretic concepts; modelling musical expectancy.
Navaneeth Suresh Kumar is a Master's student
(Music Engineering)
at the
Frost School of Music,
University of Miami.
Navaneeth is interested in raga transfer – involving
automatic transformation of a given melody into a specifiable raga
from Carnatic music, utilizing a hybrid rule-based and generative
adversarial network approach.
Jemily Rime is a PhD student
(Music)
at
University of York,
working with
XR Stories
and
BBC R&D,
with research interests in the production of personalised and
immersive audio using AI-driven tools for podcasts.
Jem also contributed songwriting and vocals to our entries to the
2022
and
2021AI Song Contests.
Chenyu Gao is a PhD student
(Music)
at
University of York,
with research interests in discovery of repeated patterns in
music, and human-centered music generation.
Mark Hanslip worked with the lab on our entries to the
2022
and
2021AI Song Contests, creating datasets as well as training neural network models for the generation of audio and visuals.
Ashay Dave is a Master's student
(Music Engineering)
at the
Frost School of Music,
University of Miami.
Ashay is interested in automatic object recognition using neural
nets in Unity, integrated with PureData for audio, with aim of
applications in games and XR.
Sourav Pande graduated the Master's program in
(Music Engineering)
at the
Frost School of Music,
University of Miami
in 2024.
Sourav's thesis concerns audio-visual zooming using neural net
approaches. AV zooming is where if one can zoom in on a video, the
sound should adjust appropriately too, foregrounding particular
human speakers or sound sources, while de-emphasizing others.
Zongyu (Alex) Yin graduated in 2022, having been a PhD student
(Computer Science)
at
University of York,
with research interests in music generation with deep learning,
and exploring various generation methods based on music-theoretic
expertise.
After leaving the lab, Alex went on to work for the SAMI (Speech, Audio, and Music Intelligence) team at TikTok.
Luke George, Integrated Master's student
(Electronic Engineering with Music Technology Systems),
joined the team as an intern from the Student Internship Bureau. He has aspirations to work in a field combining his passions for music and technology.
Andreas Katsiavalos,
DMU
PhD student, with research interests in
adaptive, complete music information retrieval systems, and a
focus on the automatic extraction of high-level concepts such as
musical schemata.
Dr. Berit Janssen is a researcher and scientific programmer based
in the
Digital Humanities Lab,
Utrecht University,
the Netherlands. She is interested in expectation and prediction
in music.
Ben Gordon, having graduated from
Lafayette College
(Data Science and Music), continues to work with Tom worked on
web-based interfaces involving natural language understanding and
music.
Annabell Pidduck, former
Music
undergraduate
at
University of York,
with interests in the use of music technology in (high) schools
and its effects on student learning and development.
Jasmine Banful, Lehigh undergraduate (Mechanical Engineering),
with interests in web-based DJ'ing software.
Reggie Lahens, Lehigh undergraduate (Journalism), with interests
in web-based mixing software.
Linda Chen, Lehigh undergraduate (Psychology and Management), worked
on a project that aimed to determine how differing levels of
feedback affect users' ability to lean to read staff notation.
Dr. Thom Corah,
formerly DMU PhD student, worked on a framework for the use of real-time
binaural audio on personal mobile devices. The aim was to create
an audio-based augmented reality, with applications in digital
heritage and assisted living.
Dr. Katrien Foubert visited the group in June 2015 while still a
PhD student. We worked on extracting structural features from
piano improvisations recorded during music therapy sessions, with
a view to predicting diagnoses of borderline personality disorder.
Among other outputs, the collaboration resulted in a
Frontiers in Psychology paper.
Austin Katz, Lehigh undergraduate (Journalism and Psychology),
worked on a project that aimed to shed light on the perception of
repetitive structure in music.
Fahida Miah was the
Nuffield Research Placement
student in summer 2014. Her project involved auto-generation of
Pop music and quantitative evaluation of creative systems.
Ali Nikrang
is a Key Researcher and Artist at the
Ars ElectronicaFuturelab,
Linz, Austria. As a Master's student, he was the main developer of
the
PatternViewer, an
application that plays an audio file of a piece synchronized to
interactive representations of tonal and repetitive structures. Ali's
thesis describes the construction of this application, the
music-psychological research on which it is founded, and the influence
of the application on listeners' music appraisal skills.
Emily Stekl, Lehigh undergraduate (Psychology), assisted with the
investigation of the effect of music artificial intelligence on
creativity. We embedded an AI suggestion button in an interface
and studied how it affects users' compositional processes.
Zhanfan (Jeremy) Yu, Lafayette undergraduate (Computer Science),
helped develop a cloud-based music transcription system.
Contact and credits
I hope you enjoyed visiting this site.
Feel free to get in touch (tom.collins@miami.edu) if you have any
questions or suggestions.
Credits
The code above was written by Tom Collins and others as specified
(e.g., toward the bottom of each demo interface). Reuse of the
code is welcomed, and governed by the
GNU General Public License Version 3
or later.