Just about every language has bindings to C, so you'll probably want to get the applicable C libraries for encoding/decoding mp3's and FLAC files. This list might include
libFLAC http://flac.sourceforge.net/api/index.html FLAC encoding/decoding
LAME http://lame.sourceforge.net/index.php MP3 encoding
MAD http://www.underbit.com/products/mad/ MP3 decoding
The rest of your signal processing needs could be gathered around a single popular API such as LADSPA http://www.ladspa.org/.
Here's a stretching / pitch shifting library: http://www.breakfastquay.com/rubberband/
Most audio processing programs have a certain internal format they use. That keeps things simple. Everything coming in gets converted to the same format. Once you've standardized the internal format, cutting and splicing audio data is about as difficult as cutting and splicing strings. You don't really need a library for that.