Similar to iTunes' 'Link Maker' (http://www.apple.com/itunes/linkmaker/faq/). Through web browser, iTunes webpages is capable of finding out if itms:// has been registered on client workstation, hence displaying correct message ('View in iTunes' button or 'Download iTunes' promotion banner).
How is this conducted?
Unfortunately, there is really no easy way to do this. iTunes actually uses the following script to do the detection. From the code you can see they use a browser plug-in within each web browser to accomplish this.
In Internet Explorer, there is a feature via the protocolLong property, which can be used, but obviously it's browser specific.
There may be other hacks you available, but I think some sort on browser plug-in would be the surest solution.