I need to check if remote host is Windows or Unix/Linux. I can't assume that it has web server configured. All I can do is to try to connect to several TCP or UDP services.
Which TCP services (TCP port numbers) usually will be opened on Windows and not on Unix/Linux and vise versa?
The other way is to try to ssh to it, and if it fails assume that it Windows host. The problem is, that I need this in order to choose the remote access method ssh or something Windows friendly like psexec.
Why not just try to connect one way, and if that fails, connect the other way, and if neither work, tell the user?
If that's all you're trying to do, there's no need to actually check the OS.
You can read the output of nmap to detect which OS a remote host is running. It has a whole module dedicated to this. Here is a guide to using it.
This is not an easy thing to answer with any degree of certainty as there are very few ports that will always be open on one OS but not on another.
You could try some/all of the following
I would suggest that scanning ranges of ports may lead you into trouble particularly if these are not your machines. You may find your IP address logged as a possible source of "Port Scanners"
There are some fairly extensive lists of ports available on the web. e.g. http://keir.net/portlist.html