There is no 100% foolproof process for determining how to validate an arbitrary XML document.
For example, this version 2.4 web application deployment descriptor specifies a W3 schema to validate the document:
<?xml version="1.0" encoding="UTF-8"?>
<web-app id="WebApp_ID" version="2.4"
xmlns="http://java.sun.com/xml/ns/j2ee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
However, this is an equally valid way of expressing the same thing:
<?xml version="1.0" encoding="UTF-8"?>
<web-app id="WebApp_ID" version="2.4"
xmlns="http://java.sun.com/xml/ns/j2ee">
RELAX NG doesn't seem to have a mechanism that offers any hints in the document that you should use it. Validation mechanisms are determined by document consumers, not producers. If I'm not mistaken, this was one of the impetuses driving the switch from DTD to more modern validation mechanisms.
In my opinion, your best bet is to tailor the mechanism detector to the set of document types you are processing, reading header information and interpreting it as appropriate. The StAX parser is good for this - because it is a pull mechanism, you can just read the start of the file and then quit parsing on the first element.
Link to more of the same and sample code and whatnot.