The system, which is still in prototype, works by connecting off-the-shelf noise-canceling headphones to a smartphone app. The microphones embedded in these headphones, which are used to cancel out noise, are repurposed to also detect the sounds in the world around the wearer. These sounds are then played back to a neural network, which is running on the app; then certain sounds are boosted or suppressed in real time, depending on the user’s preferences. It was developed by researchers from the University of Washington, who presented the research at the ACM Symposium on User Interface Software and Technology (UIST) last week.
The team trained the network on thousands of audio samples from online data sets and sounds collected from various noisy environments. Then they taught it to recognize 20 everyday sounds, such as a thunderstorm, a toilet flushing, or glass breaking.
It was tested on nine participants, who wandered around offices, parks, and streets. The researchers found that their system performed well at muffling and boosting sounds, even in situations it hadn’t been trained for. However, it struggled slightly at separating human speech from background music, especially rap music.
Researchers have long tried to solve the “cocktail party problem”—that is, to get a computer to focus on a single voice in a crowded room, as humans are able to do. This new method represents a significant step forward and demonstrates the technology’s potential, says Marc Delcroix, a senior research scientist at NTT Communication Science Laboratories, Kyoto, who studies speech enhancement and recognition and was not involved in the project.
“This kind of achievement is very helpful for the field,” he says. “Similar ideas have been around, especially in the field of speech separation, but they are the first to propose a complete real-time binaural target sound extraction system.”
“Noise-canceling headsets today have this capability where you can still play music even when the noise canceling is turned on,” says Shyam Gollakota, an assistant professor at the University of Washington, who worked on the project. “Instead of playing music, we are playing back the actual sounds of interest from the environment, which we extracted from our machine-learning algorithms.”
Gollakota is excited by the technology’s potential for helping people with hearing loss, as hearing aids can be of limited use in noisy environments. “It’s a unique opportunity to create the future of intelligent hearables through enhanced hearing,” he says.
The ability to be more selective about what we can and can’t hear could also benefit people who require focused listening for their job, such as health-care, military, and engineering professionals, or for factory or construction workers who want to protect their hearing while still being able to communicate.