Editor’s note: This story was originally slated to run in early March 2020, prior to the pandemic shutdown.
A hand-sized robot sits on the head of a drum and taps along with a human player.
It is mobile, moving around the drum, the sound of its motion contributing to the music.
It can improvise, helping the musician explore new directions. It can also work in co-ordination with multiple robot drummers, each responsible for a certain instrument or part of a composition. It even has applications in music therapy.
That’s the dream of two Memorial University researchers, Dr. Andrew Staniland, a professor with the School of Music, and Dr. Andrew Vardy, a jointly appointed professor with the Department of Computer Science, Faculty of Science, and the Department of Electrical and Computer Engineering, Faculty of Engineering and Applied Science.
They’ve been working with Dr. Maxence Blond, their co-supervised post-doctoral fellow, to make that dream a reality.
“The idea is to develop a robotic percussion accompaniment,” said Dr. Vardy. “This could be a tool for serious musicians, as well as for amateur musicians looking to “jam” with a very agreeable musical partner.”
Robot in a piano
The researchers became friends after crossing paths repeatedly, not only at the university, but also at their children’s daycare and in the community through friends.
“The idea is to develop a robotic percussion accompaniment.”
They also share interests in music and technology. Four years ago, they tried their first project together when they put a small round robot, called a Sphero, inside a piano.
“We had it run around on the strings of the piano,” said Dr. Staniland. “It made some interesting sounds.”
“That got the wheels turning in our heads,” added Dr. Vardy.
As founder of the Memorial ElectroAcoustic Research Lab, or MEARL, Dr. Staniland’s creative activities and research focus primarily on the composition of music, but he also has a secondary interest in digital instruments, creating two of them since he joined the university.
In addition to using human input to make meaningful musical gestures on electronic media, he has also used biofeedback to make music.
“Instead of touching something, my JADE project uses EEGs and heart rate as the driver for initiating sound,” said Dr. Staniland. “So, it seemed a natural extension to take my interest in music composition and digital instruments and see what happens when you add robots into the mix.”
Building a prototype
The idea had been bubbling since their “robot-in-a-piano” project.
“We wanted to explore the idea of a robot on a surface having some kind of interaction with a human player,” said Dr. Vardy. “While other people have developed robot musicians, none of them have been mobile and I generally work with mobile robots.”
The prototype robot built by Dr. Blond has a microcontroller base on the bottom, a board that controls a smart servo motor and a stick it can tap on the drum’s surface.
It is also linked to a Raspberry Pi computer that takes audio input, processes it and then sends a signal to the robot to respond.
At the moment it can repeat a simple rhythm and has a few shortcomings the pair hope to address in their future work.
“We’ve gone through three different types of motor and it’s still too slow, so we’re exploring what technology will work,” said Dr. Vardy. “That’s part of the journey, but an important part.”
“Dr. Vardy and Dr. Staniland gave me full autonomy on the project, while being available for any help,” said Dr. Blond.
“It was also a good opportunity for me to work with different technologies, such as the Raspberry Pi controller and different kinds of motors, while designing the prototype of our system. It was a multidisciplinary project, requiring skills like signal processing, programming, and electronics. I found it very innovative.”
See below for a video demonstration of the mobile robot dummer.
Interesting avenues
The next step? Drs. Vardy and Staniland want new students or post-doctoral fellows to join the project.
They are hopeful that they’ll find someone who is very musical and also technically skilled, someone that can “really drive” the research forward.
“I think there are a lot of interesting avenues open to us now that we’ve laid the bedrock.”
They say this type of project necessitates a broad team with various skills.
They’ve also identified a number of potential directions the work could go in, including interactivity with humans, improvisation and working in a group.
“I think there are a lot of interesting avenues open to us now that we’ve laid the bedrock by creating this prototype,” said Dr. Staniland. “We’re only just beginning to explore them, but we believe this wee little seed can grow into a very cool project.”
