What are correctly content-types for this documents ?
I need to write a simple crawler, that only fetches this kind of files.
Nowadays http://somedomain.com/index.html can serve for example an JPEG file due to mod_rewrite, so I need to check the content-type from the response header and compare it with a list of allowed content-types.
From where I can get such list ?
HTML: text/html, full-stop.
XHTML: application/xhtml+xml, or only if following HTML compatbility guidelines, text/html. See the W3 Media Types Note.
XML: text/xml, application/xml (RFC 2376).
There are also many other media types based around XML, for example application/rss+xml or image/svg+xml. It's a safe bet that any unrecognised but registered ending in +xml is XML-based. See the IANA list for registered media types ending in +xml.
(For unregistered x- types, all bets are off, but you'd hope +xml would be respected.)