Frameset is for documents that define framesets (non-frameset documents are then loaded into the frames). Frames are usually a poor design choice that cause more problems then they solve.
Transitional is, more or less, Strict plus things you shouldn't use (mostly because they have been replaced with CSS). There are a couple of exceptions, such as the start
attribute which has some good arguments to support using it (but isn't often useful).
Strict is the core of HTML and is usually the best bet.
HTML 4.01 is the latest, finished version of HTML.
XHTML 1.0 is HTML 4.01 expressed in XML. It doesn't work in Internet Explorer unless you pretend it is HTML (which the text/html specification does allow).
XHTML 1.1 is XHTML 1.0 Strict plus the target attribute plus Ruby (not the programming language) plus a few other minor tweaks. There is no specification giving the OK to pretend XHTML 1.1 is HTML.
HTML 5 is a draft specification, subject to change, and with many of the interesting bits not working without JavaScript backing them up in most browsers in use today. It lacks a mature validation tool.
HTML 4.01 is generally the best bet, but XHTML 1.0 is worth considering if you have an XML toolchain in your publishing system, and HTML 5 is worth considering if you have a need for something added in it and feel the risk of living on the bleeding edge is worth it.
In short: Use HTML 4.01 Strict unless you know why you need to use something else.