Yeah, MP3 files are very different from WAV files. WAV files contain raw audio data in the form of samples from beginning to end to paint the waveform of the output, the same way a bitmap file contains raw data about pixels from left to right, top to bottom. You can think of a WAV file as a bitmap picture of sound waves -- but rather than pixel colors, it stores audio intensities, typically 44,100 of them per second, for two channels if it's stereo, and 2 bytes per channel.
(Knowing this you can actually calculate the file size of a WAV file -- to store 1 minute of audio, you'd need 60 seconds * 44100 samples * 2 channels * 2 bytes = 10.09MB.)
MP3 files contain a mathematically modified version of this image and discards audio that humans can't hear. It works similarly to how jpeg images work to compress images.
Just as video cards ultimately need bitmaps to work with, sound cards ultimately need WAV data to work with -- so yes, you need a decoder.
At the beginning of Mp3 files is a block of data called an ID3 tag, which contains a bunch of basic information about the file -- artist names, track length, album names, stuff like that. You can use something like C# ID3 to read/write ID3 tags in C#.
As for audio itself, I'm not sure there are Mp3 decoders written entirely in C#. Technically there's no reason that it can't be done (it should be fine performance wise too), but the standard is pretty loose and the math is intense so people tend to just use things like FFMpeg to decode. Some ideas in this Google search.
If you don't need to do any special processing and you just want to play the audio, you can use the WPF/Silverlight Media element.
You can probably get some hints out of Josh Smith's Podder app.