What started as a way of training classical musicians has turned into an iPhone application for teens to take “musical selfies” and share them with their friends online. Along the way, MuseAmi, a Nassau Street-based software company, has developed some of the most advanced pitch correcting software in the world.
The company’s latest product is Hook’d, a free music app that aims to be the musical equivalent of the photo sharing app Instagram. The software hit the Apple store in July. It allows smartphone users to record themselves singing along to pop tracks, replacing the artist’s voice with their own, and then share the video with their contacts on Hook’d, YouTube, and the usual social networking sites.
Robert Taub, the renowned classical pianist and co-founder of the company, says Hook’d is a way of making music active and social again, in a world where people mostly just listen to their favorite tunes on headphones. “We’re very interested in participatory music,” Taub says. “We are interested in unleashing everyone’s inner creativity, by allowing people to get around any hurdles for entry. What Hook’d does is it takes you and makes you sound good with pitch correction, reverb, and echo, and allows you to interact with a song that you know and love in less than 30 seconds.”
There is a selection of 30-second to one minute-long song choruses available for free. To make a Hook’d video, you plug in headphones and hear the backing track with the option of hearing the singer too, as lyrics scroll across the screen. When you are done, you have the option of posting the video and adding electronic “studio effects” and pitch correction. (The original singer’s voice is gone from your cover version.) To sing an entire song, you buy it for 99 cents, or a themed bundle, such as a “hip-hop pack,” at a discount.
“There’s a fantasy element to it,” said Ed Hynes, chief operating officer of MuseAmi. “When you sit in your car, and you can sing along to a favorite song of yours, you get out of your car and it’s over. With Hook’d, you have a little bit of reverb and you can become god-like for a moment, and make a permanent record of it.”
Hook’d may invite comparisons to that other sing-along fad, Karaoke, but that kind of talk is frowned upon in the MuseAmi offices, where they call it “The K-Word.” There are several major differences that set Hook’d apart from its drunken barroom forerunner. First, MuseAmi has signed deals with major record labels (and many minor “independent” ones, too) that give them access to the real backing tracks that go along with songs. So when a teen is singing along to Christina Perri’s “Jar of Hearts,” she is singing to the real music, not a cover. Second, the app offers the option of adding electronic effects to the music video.
As explained by product manager Jon Sheldrick, each track in the app’s library contains metadata about what key the song is in. If a singer is off key, the program can “fix” the user’s voice subtly so that it hits the nearest on-key note. “It’s not going to make everyone sing the exact same notes,” Sheldrick says. “Then the performances would have no variation. Instead, it nudges you in the right direction.”
The trick works. The most popular Hook’d video as of early August was of a girl in braces and glasses belting out her completely re-imagined take on Iggy Azalea’s “Fancy.” She is riding around in a car, but the app’s effects make her sound as if she is singing in a studio.
The most fundamental difference is that Hook’d aims to make the sing-along experience less intimidating in general. “We encourage activity,” Taub says. “We get rid of any hurdles such as embarrassment.” While someone may not be comfortable belting out “Kiss Me” to a crowded room, the process of recording a Hook’d video is more like singing in the car, or in your bedroom with a hairbrush standing in as a microphone.
Currently, the app is being marketed to teenage users. Its first focus group was culled from Princeton High School’s music program, and many of the 60 or so “seed” users were also teens. Taub says more features and songs are on the way that are designed to appeal to people in their 20s and 30s, who, to Taub’s surprise and delight, are already using the app. There are a handful of older users too, including Taub, who declined to reveal his user name on the service.
Hook’d maintains a YouTube channel of videos its users have made: some good, some bad, some funny, and some creative. In the first few days, people have found ways of using the app that its creators never thought of. One girl blinked her eyes in perfect time with the music. Another drew pictures to go along with a song’s lyrics and panned the camera over them as she sang. In another video, a cat is the star.
One thing the Hook’d team didn’t have to worry about was whether the concept of making sing-along videos would be popular. Sheldrick says there are already “millions” of such videos on YouTube.
Most of those videos were made with laptops or desktops, but the process of doing it took a bit of know-how. Those videos also ran the risk of copyright “takedown” notices if they got popular enough to attract the notice of the music industry’s intellectual property watchdogs.
The Hook’d app just makes the process of recording one dead easy, and since MuseAmi has bought the rights to the music, there is no chance of the video being removed. The company has formed partnerships to share profits with labels and artists in exchange for the right to use the backing tracks.
Taub never intended any of this when he launched MuseAmi in 2007. Back then he was only thinking of how to make the process of learning music easier for his daughter, who was about 12, while he was away on tour. “I was a performing concert pianist and my daughter was striving to learn the violin,” Taub says. “I saw her struggling with the challenges of reading notated music, and I thought it would be helpful if she could pick up a device, take a snapshot of the printed sheet music, and hear the passage in question: what well-trained musicians can do in their brain, she should be able to do in the palm of her hand.”
The scope of the project doubled within five seconds. “After a few seconds thinking about it, I realized that’s only half the battle. You’d also need to be able to see what you hear; that what you are playing is in tune, with the right rhythm and the right notes.”
This idea eventually became the MusicPal app, which has two functions: it can read any sheet music with multiple instruments, and play it over a phone’s speakers or headphones. In reverse, it can listen to a single instrument and transcribe its sound into sheet music.
Taub’s daughter, now in her 20s, learned the violin without the aid of a music app, and is now a violinist in a community orchestra. (His son Ben also has earned musical fame, as a contestant on the singing contest show “The Voice” in 2012.)
In the meantime, Taub has strived to bring his original idea to fruition. MuseAmi’s co-founders are Yann LeCun, a professor of computer science at NYU who is a machine learning expert, and Urs Muller, the inventor of webmail, who was LeCun’s co-worker at Bell labs. He is also director of a new artificial intelligence lab at Facebook.
Taub’s interest in music goes all the way back to his childhood growing up in Metuchen. His father was a research chemist and his mother was also a chemist, both of whom encouraged his musical side from a young age. Taub studied music at Princeton, graduating in 1977 before earning his doctorate at Juilliard. He was a concert pianist for more than 20 years, performing at venues around the world as a featured soloist. For three years, he was artist-in-residence at the Institute for Advanced Study.
He has played with the MET Orchestra in Carnegie Hall, the Boston Symphony Orchestra, the BBC Philharmonic, and many others. His recordings include complete sonatas of Beethoven and Scriabin, and a book he wrote, “Playing the Beethoven Piano Sonatas” is well known in the music world.
Taub reached such lofty heights by being a meticulous, at times obsessive, student of music. He once spent eight years learning how to play Beethoven’s Hammerklavier sonata before playing it in public, telling Piano Education in 2008:
“Although one hears all sorts of performances of this piece which disregard these metronomics, I actually think that they play an important role in reflecting the ways in which Beethoven considered the wide spectrum of character of the movements, from the extraordinary, unbridled enthusiasm of the first movement to the plaintive profundity of the Adagio, to the slow, searching quasi-improvisatory nature of the Largo, to the stern brilliance of the fugue. Internalizing all of this, bringing the work to a level with which I felt I could do it justice and convey and express all that the work can offer, took me longer than any other piece I have performed. But now, I feel as though I have lived with this piece for a long time, and I love it dearly. I also love to perform it.”
It was the dream of helping others learn how to play music that drew Taub away from his career as a pianist and into the world of software development. They may seem like divergent careers, but Taub is joining a long line of musicians who have excelled in other fields. Speaking to the New York Times in October, 2013, Taub described his thought process during performing as a kind of synesthesia where he could “visualize all the notes and their interrelationships,” a skill which carries over into developing a complex computer program where multiple components interact with one another.
‘Hook’d lets you voice your inner selfie!” says the voiceover of a promotional video explaining how the app works. “It’s totes simple!” But the underlying technology is actually totes complicated.
It turned out that making a music translation program was more work than Taub expected. the MuseAmi team is still working on MusicPal, the software designed to fulfill Taub’s original vision of software that could take a picture of sheet music and play it back as sound. It is expected to launch in 2015.
In the music world, Taub is known for his exceptional ability to read sheet music. “He has played some notoriously difficult music,” Hynes says, including pieces by Milton Babbit which stymie even other concert pianists. To a gifted musician like Taub, sight reading is second nature.
Taub figured that reading sheet music would turn out to be little more complicated than reading printed words, which computers have already mastered. Computers are also good at waveform pattern analysis and speech recognition on the other side, so the task of scoring music also seemed within the realm of possibility.
After only a short time, the MuseAmi team discovered sides of the equation have big complications. The problem of reading music is a problem of computer vision. The software must take an image captured with a camera phone, see the symbols of musical notation, and interpret them correctly.
Technical developer George Tourtellot, who worked on the software, explained some of the problems the team had to solve: First, the images are often far from the perfect reproductions captured by flatbed scanners. The software has to account for the curling of the page, blurriness of the image, shadows, and other features introduced by amateur photography.
Then there is the imperfection of sheet music itself.
A lot of sheet music has words running through it. How can the computer tell an “o” from a whole note? It gets worse from there. “Thumb through a child’s music book and you’ll see exactly what I mean,” he says. “Not only are there pictures of frogs everywhere, but maybe there are stickers or pencil markings the teacher or parent have made. That’s not ideal on a sheet of music, but realistically, this is the type of music that people have sitting on their piano.”
Then there is the more fundamental problem of context. “A lot of symbols mean different things depending on where they’re placed,” Tourtellot says. “Take an accidental. It can be made part of a key signature, a modifier to a pitch, and it can also be present in a chord.” Some of the most common “accidentals” — the sharp, flat, and the natural — happen to look like the “#”, “b” and “d” symbols sometimes found in text, which further muddies the waters.
The basic problem is that whereas text is fairly straightforward to scan and turn into data, everything in sheet music is dependent on context. “There are a lot of irregularities that you find in notation in the wild,” Tourtellot explained.
Another obstacle the programmers were up against is that the software had to work on cell phones, which, for all their strengths, are very poor computers, and music software can be a computing hog even on fast desktop computers. “They call it a smartphone, but it is one of the dumbest devices there is,” Hynes says. “There are not a lot of processing cycles to spare.” (Efficient use of computing resources is also what made Hook’d viable. The app uses its Internet connection to download the music and upload completed videos, but all of the audio processing is done on the phone itself.)
Processing efficiency is also what allows MuseAmi’s software to work its magic in real time, where some other music software suites require the user to wait out a processing delay before hearing the results. It’s also what allowed the Improvox suite of digital effects to be used in the video games Guitar Hero 6 and DJ Hero 2.
To solve the music reading problem, the team, led by co-founder and AI expert LeCun, used machine learning techniques and an AI method called “convolutional neural networks” that is widely used in the field of image recognition. Essentially, they created a program that mimicked the structure of the human visual cortex, with small collections of “neurons” piecing together overlapping layers of an image.
Machine learning involves the program looking images many times, trying to interpret them correctly, and improving its algorithm based on its successes and failures over time.
Tourtellot didn’t want to go into too much detail here, for fear of giving valuable information to competitors.
The other function of the application, listening to music and turning it into musical notation, turned out to be more straightforward, but still has its limitations. “If you say, try to analyze any arbitrary recording and turn it into annotated music, that’s an extremely difficult problem,” Tourtellot says. “Recognizing certain specific elements of a limited scope of audio is a simpler problem.” MusicPal tackles the simpler of the two problems. You can’t play a symphony and have MusicPal print out a complete arrangement, but you can pick out a tune on the piano and see if you are playing the right notes.
The software can’t untangle multiple simultaneous notes, so it is currently limited to mono-instrument recording with no chords. It can detect pitch, tempo, and vibrato. (On the other side, it can read complicated sheet music with many instruments and play it back.)
“If you are trying to learn a piece of sheet music, we’re here to tell you how you’re doing,” Hynes says.
So far, it has taken MuseAmi almost as long to develop its music reading software than it took Taub to learn the Hammerklavier Sonata. In the meantime, several competitors have emerged, including a multitude of music apps. There are several sheet music players already on the market that have a similar goal in mind to MusicPal. But MuseAmi hope its technology will beat them all in terms of usability, speed, and accuracy. Hynes called the competitors “underwhelming.”
“We perceive there is a fairly high barrier to entry in this market,” Hynes says. “We are confident about our timing to market, and the competitive edge we’ve been able to build.”
Hynes envisions MusicPal being used in all kinds of music education situations, from individual teachers, to K-12 music programs, to conservatories.
“None of us should underestimate the value of something that useful to everyone learning sheet music,” says Hynes, who has played the guitar for 30 years. “If I were serious about learning sheet music, I’d have this with me at all times. We want to deliver that value to everyone who is interested in learning.”
MuseAmi Inc., 20 Nassau Street, Suite 250 E, Princeton 08542; 609-917-3000; fax, Robert Taub, president & CEO. www.museami.com.