Distinguishing instruments in a music file

Question

Given a music file, is it possible to split out each instrument that is being played? I.e. let's say I have someSong.mp3, and in that song there's vocals, guitar, bass and drums. I'd want to get 4 "tracks" - one for each distinct instrument.

I'm guessing that it's almost impossible to do this, given that instruments can overlap, and it's notoriously difficult to distinguish overlapping voices let alone instruments.

However is there is a library, or an algorithm, or SOME way of doing this, I'd be curious to hear how.

Answer 1

The easiest way to do this is to maintain the instruments separately in the first place, which is why many intermediate musical processing applications use MIDI to store instrument messages in an abstract form on separate tracks.

Answer 2

Every instrument has a characteristic spurious oscillation, so if you isolate single notes (with a DFT/FFT) and compare the other frequencies at that point of time, you might distinguish at least different types of instruments.

Answer 3

mp3 is a lossy format. It works by modeling the acoustic response of a person and throwing away information that it considers that a person can't hear. In essence louder instruments that overshadowing soft instruments - the softer one gets ditched. This means that you never will be able to extract what originally went into the mp3 let alone separate out the instruments.

I have a friend who is a sound engineer and he always has to say "No" to people who ask him to re-engineer a track recorded as an mp3.

A few weeks ago I saw a study that suggested that the younger generation actually preferred the sound of mp3's over more complete formats as that is what they had grown up with.

Answer 4

Long story short: You can't except in the case that your four instruments are synthesizers playing pure sine-waves.

Answer 5

My undergraduate project dealt with transcribing notes from a WAV file to a MIDI file. We handled only the simple case of one instrument, possibly playing more than one note at a time (a piano, for instance). Our research into the subject before we started showed that even this (i.e. only one instrument) is considered non-trivial. Basically, the problem is:

• find what frequencies are playing at any given time. This can be done by a DFT/FFT of small windows one at a time.
• Use some heuristic to guess which frequencies are harmonies of the same note, and which belong to different notes. This may be easy if you know what instrument is playing, but it's hard in the general case, because the magnitudes of each harmony differ by instrument. For instance, you might have two Cs one octave apart from one instrument, or you might have one C but from a different instrument.
• after you know what notes are playing at each time, you have to guess when you have breaks in the notes. You could have one long note or a series of short notes. Depending on the size of the windows you used for the initial DFT, you could have different results here.

Now, if you have more than one instrument at a time, and no two are playing the same notes or harmonies thereof at one time, you might be able to tell the instruments apart using some heuristic on the magnitudes of the harmonies or on the sequences of notes they're playing. Most likely there will be times when two instruments are playing the same note. Then you don't really have any way to decide if there is (a) one instrument playing the note, (b) two instruments playing at the same volume, (c) one playing soft and the other playing loud or (d) any combination thereof.

Anyway, that's the short list of problems to solve. I don't know of any algorithm that solves this in the general case. I don't think this problem has been solved yet.

Edit: My project presentation can be found at http://www-sipl.technion.ac.il/new/Archive/Special_Events/sipl2004/Projects_PowerPoint/WAV-to-MIDI.pdf

Answer 6

basically what you want to do is very difficult to do programmatically although no longer impossible..

http://www.youtube.com/watch?v=jFCjv4_jqAY

Answer 7

I have actually bumped into a very interesting algorithm called ICA (Independent Component Analysis). The concept behind this algorithm doesn't come from the signal processing world, but from probabilistic theories. We used it to separate two songs that were mixed into single mp3 file. You can find an implementation library in Matlab \ C++ \ Python called FastICA here. Give it a shot it's really nice.