Per your comment I think you've got a good handle on Music and Math. I'm totally naive but, briefly, what background should one have if he intends to start working on extracting a melody from a song?
I will appreciate your reply (didn't find your email in your profile so posting it here, thanks).
OK, then you want to get into the world of Digital Signal Programming, or DSP. Before you do so, be aware that this is a Hard Problem if you want to achieve more than the most basic results. The basic tool of DSP is the Fourier Transform, which allows you to convert a 1-dimensional signal in the time domain (such as an audio file) to a 2-dimensional signal in the frequency domain (such as a spectrogram aka graphic equalizer display). Many problems that look knotty or impossible in the Time domain are soluble with simple math in the Frequency domain. So you do an FFT, modify or analyze your signal, and then do another FFT if you want to convert it back to an audio stream.
This is a really excellent starter book that you can also download for free: http://www.dspguide.com/ It's far better written than most other books on the field and will help you to develop an intuitive understanding of the fundamental math. Many books just say 'here's the math,' without discussing why it works or why you would want to do it one way rather than another. Many more cover DSP from the point of view of radio or wireless communication - although the same principles apply here as for audio, it's somewhat confusing. This book is very audio-friendly.
The state of the art in pitch extraction from usic recordings is Celemony's Melodyne: http://www.celemony.com/cms/ The company was started in the mid-90s by a German audio geek named Peter Neubäcker with his wife and a programmer. He says in interviews that he's using a different approach based on the shape of sounds, but has never published his methods. I've met him a couple of times at conferences and trade shows, but he knows how to keep a secret! However, you'd be well advised to try out the demo version of his software. How he does is it is a mystery, but he's way, way ahead of any commercial or academic methods.
If you like Matlab, this is the best academic work on the task so far: http://isophonics.net/content/reverse-engineering-mix and you should also grab a copy of Sonic Visualizer, which is a slow-but-flexible analysis tool: http://isophonics.net/SonicVisualiser Be sure to follow up the links on the Isophonics site, which will lead you to a rich variety of libraries and tools for audio programming.
