If you do not require massive performance, I would use some Exif library and let it try to get the Exif data (if present) for you. (pyexif, perl Image::exif, c# MetaDataExtractor etc)
Otherwise,
take a look at http://en.wikipedia.org/wiki/JPEG#Syntax%5Fand%5Fstructure
You need to create a simple binary parser do dig out the "segment codes", and to find the segment called APP1 (if I understand it correctly). The data should contain the letters "Exif"
e.g. in a random JPEG file on my PC, the bytes 7-10 said "Exif". I do not know if the location is the same in all JPEG files. The segments may be of variable length.