Looks like you need to 'chain' the installs http://objectmix.com/xml-soap/84668-installing-sqlxml-net-app.html
You can get the redist here http://www.microsoft.com/downloads/details.aspx?FamilyID=51D4A154-8E23-47D2-A033-764259CFB53B&displaylang=en
CAn you add this as a pre-req for your install?
What are you using to build the create the install?
Edit:
I had a look to see how you can check of the SQLXML is installed and come across this:
http://www.tech-archive.net/Archive/SQL-Server/microsoft.public.sqlserver.xml/2005-04/msg00110.html
The system I am on just now has the following key HKEY_CLASSES_ROOT \ SQLXMLX (note the X at the end), so you might need to do a bit more investigation in to what the actual key is.
I'm not familer with Visual Studio install authoring but if you can add an entry to the AppSearch and RegLocator tables you should be able to check for the existance of the registry key when the install starts. See here
http://msdn.microsoft.com/en-us/library/aa371564(VS.85).aspx
The Reglocator table gives you the option to set a property with a value from the registry if found. You can then use this in the condition on a custom action.
A lot to put together, but I hope it move you in the right direction.