Where in the mp4 file structure is the duration of it?
MP4 is a "container" format, which basically means it can contain a number of different audio or video streams. And each stream could have it's own duration value...
To dig out what you need, you're going to want some more reference files. I might suggest looking here and here... but you'll probably have to go searching beyond that for the different types of A/V streams you want to support.
This may not be the answer to your problem but it was to mine: http://mediainfo.sourceforge.net/
(It has a library and it's open source so you can just check for the part(s) you need)
As far as i know - "mp4" container is derived from the QuickTime atom structure. You can read the description of QuickTime File Format.
Parsing quicktime atoms is not a big deal (look at atomicParsley project). I'm not sure for MP4, but as for MOV-files - there's a "duration" field in "mvhd" (movie header) atom and also in "tkhd" (track header) atom. This duration is usually a number of frames multiplied by the "time scale" attribute. Time scale can be found in the same atoms.