For WMP, what is the codec locating
policy.
Until Windows Media Player 7.0, it was just a pure DirectShow player. Since 7.0, Microsoft has started handling different file types differently. This was mostly the case with WMV files. WMP now will use MediaFoundation media pipeline in Vista and 7 for some file types.
How do I differantiate my custom file
format from mpeg-3 format for example.
What is the standarts and programming
structs for this. (The real part I'm
messed up is this. Container formats,
video formats, fourcc codes...etc
drive me to the death)
You could do this (lol, sorry for the MSDN link), if you are creating your own custom container format, but if you want to make life easier on yourself, I'd choose an existing and/or open-source container, such as AVI or MKV. When you create a stream in either of these formats, you'd provide a codec code, such as a fourcc for video or a format tag for audio. Since you are doing your own custom format, you should just make up your own fourcc or format tag values, but careful not to pick one people already use.
Lets say you were using an AVI. The built in avi demux would read the AVI headers and look at the meta data for each stream within the file. It might find a video stream with the fourcc DX50 and an audio stream with the format tag of WAVE_FORMAT_PCM. DirectShow will enum registered filters and query their pins to see if they support taking a DX50 video or a WAVE_FORMAT_PCM audio. So if you AVI had a fourcc of XXXX, DirectShow would enum filters looking for types that take XXXX. You would then make a transform filter that will accept fourcc XXXX. Your transformation filter will decode the video and output an uncompressed format. Maybe RGB24. DirectShow will detect that your output pin outputs that media type and will probably connect it directly to the renderer. The process is the same for audio, except we deal with format tags instead of fourccs (and you deal with WAVEFORMATEX and VIDEOINFOHEADER).
For this work on Windows what method do you suggest to write codec(DMO or Filter)
I've never written an DMO personally, but I would highly recommend looking into it. I've heard there is less boiler plate COM stuff and IIRC, MediaFoundation has a DMO wrapper, so you may get automatic MediaFoundation support.
Besides the DShow SDK samples, you may want to check out this. May help you get started. Also this website has great samples also, but might not be exactly what you want.