Welcome to Nono’s World, a brightly colored landscape of rocks and trees and clouds, occupied by a pink and blue cartoon axolotl with enormous eyes and an impish smile.
Known as a neuro-animation therapeutic experience, Nono’s World was developed to help people who have had strokes regain fine motor movements. Users turn specially designed knobs to make the salamander-like amphibian swim, jump, and fly. The soundtrack constantly changes to reflect those movements.
The concept was developed by Omar Ahmad, director of the Kata Design Studio, a Johns Hopkins Medicine group that creates experiences for patient care and wellness. The design was led and implemented by Arnold David Gomez, a Johns Hopkins neurology faculty member and a neuro-animator and engineer with Kata.
For the soundtrack, Gomez and Ahmad turned to the Peabody Conservatory’s Music for New Media department, which teaches students how to create music for video games, film, video, and other multimedia applications.
“The musical experience that a patient will have playing this game is not an added benefit,” Gomez says.
“It’s actually core to healing. It’s core to adherence to a rehabilitation program. We treat it very, very seriously.”
Students in the class of Thomas Dolby, director of Music for New Media, developed initial music ideas for Nono’s World before his colleague Chris Kennedy took on the challenge of making it interactive, working with Peabody undergraduate students Matthew Flynn, Naomi Biela, and Cade Mullen.
The team used a program called Wwise, an application specifically for developing music and sound for interactive media and games. Working mainly between February 2021 and August 2022, the students built software instruments from samples of acoustic instruments that were in the public domain. After composing music in a digital audio workstation, they used Musical Instrument Digital Interface (MIDI) sequences in Wwise to trigger playback of the sounds.
The task proved more difficult, gratifying, and groundbreaking than Kennedy and the students expected.
“It is common in games for the music to change based on what your character is doing, but not to this specific level,” Kennedy says. “The real challenge was making it so that the music would change in different ways based on movements from controllers in both hands.”
He continues: “The right hand controlled the tempo, which was pretty straightforward. When Nono moved faster, the pace of the music increased. But what about the left hand?”
The team’s solution was to use different instruments and volumes for different movements, layering them so that users create real-time original music as Nono explores.
“Whether the character is crawling, or swimming, or even flying, the sound changes,” Kennedy says. The game also distinguishes between quick events such as jumping, which produces a harp glissando, and longer-term actions like flying, characterized by a celeste.
Gomez says neurological patients now use Nono’s World on a case-by- case basis. “Preliminary observations show that patients enjoy and utilize the game on their own and that they tend to improve their performance during interactive music sessions compared to gaming with non-interactive music,” he says.
Future studies will examine the therapeutic effectiveness of Nono’s World in both hospital and home settings, he says.
Biela, now a 22-year-old Peabody senior double-majoring in Voice and in Music for New Media, says she was particularly interested in the project because some of her family members have experienced strokes.
“It was neat to work on something so grounded in real life,” she says. “One of our goals with the game was to use music to create an emotional connection between the person who’s playing and the character that they’re controlling,” she says. “This game is something that could have a very, very real impact on somebody’s life.”
— Karen Nitkin
