Problem loading remote html source code using JDOM

Question

I am trying to use JDOM to load the source code of a remote HTML file (a Blogger's profile). I have this code:

public Document getDoc(URL url) throws JDOMException, IOException{
    SAXBuilder saxBuilder = new SAXBuilder();
    saxBuilder.setFeature("http://xml.org/sax/features/validation", false);
    saxBuilder.setFeature("http://apache.org/xml/features/nonvalidating/load-dtd-grammar", false);
    saxBuilder.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);
    saxBuilder.setValidation(false);
    Document doc = saxBuilder.build(url.openStream());
    return doc;
}

When I try to run something like this:

public static void main(String[] args) throws BadLocationException, JDOMException, IOException{
        linkExtractor(new URL("http://www.blogger.com/profile/07059093309718767384"));
}

I get this exception:

run:
Exception in thread "main" org.jdom.input.JDOMParseException: Error on line 1: The entity name must immediately follow the '&' in the entity reference.
    at org.jdom.input.SAXBuilder.build(SAXBuilder.java:468)
    at org.jdom.input.SAXBuilder.build(SAXBuilder.java:770)
    at tc.Crawler.linkExtractor(Crawler.java:60)
    at tc.Crawler.main(Crawler.java:44)
Caused by: org.xml.sax.SAXParseException: The entity name must immediately follow the '&' in the entity reference.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:195)
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:174)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:388)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1414)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1838)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3024)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
    at org.jdom.input.SAXBuilder.build(SAXBuilder.java:453)
    ... 3 more
Caused by: org.xml.sax.SAXParseException: The entity name must immediately follow the '&' in the entity reference.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:195)
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:174)
    at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:388)
    at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1414)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1838)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3024)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
    at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:140)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)
    at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
    at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
    at org.jdom.input.SAXBuilder.build(SAXBuilder.java:453)
    at org.jdom.input.SAXBuilder.build(SAXBuilder.java:770)
    at tc.Crawler.linkExtractor(Crawler.java:60)
    at tc.Crawler.main(Crawler.java:44)

Note that I had to add this lines:

saxBuilder.setFeature("http://xml.org/sax/features/validation", false);
    saxBuilder.setFeature("http://apache.org/xml/features/nonvalidating/load-dtd-grammar", false);
    saxBuilder.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);
    saxBuilder.setValidation(false);

because at first, I was receiving a 503 error when loading URL: http://www.w3.org/TR/html4/strict.dtd.

Thanks.

Answer 1

It's not the best idea to use XML parsers to parse HTML. Consider using something like NekoHTML first.