Error message while connecting to URL using HTMLUnit

Question

Note : I have included the stack trace also now.

I am using HTMLUnit in java to connect to a remote URL and derive some information from the web page obtained. I am using the following code...

final WebClient webClient = new WebClient(
BrowserVersion.INTERNET_EXPLORER_6_0, "companyproxy.server", 8080); final DefaultCredentialsProvider scp = new DefaultCredentialsProvider();
scp.addProxyCredentials("username", "password","companyproxy.server",8080);
webClient.setCredentialsProvider(scp);

      final URL url = new URL("http://htmlunit.sourceforge.net");<br>
      final HtmlPage page = (HtmlPage)webClient.getPage(url);<br>
      System.out.println(page.asXml());<br><br>

After providing details for the proxy server i am getting this error message

SEVERE: Credentials cannot be used for NTLM authentication:
org.apache.commons.httpclient.UsernamePasswordCredentials
org.apache.commons.httpclient.auth.InvalidCredentialsException: Credentials cannot be used for NTLM authentication: org.apache.commons.httpclient.UsernamePasswordCredentials
at org.apache.commons.httpclient.auth.NTLMScheme.authenticate(NTLMScheme.java:332)
at org.apache.commons.httpclient.HttpMethodDirector.authenticateProxy(HttpMethodDirector.java:320)
at org.apache.commons.httpclient.HttpMethodDirector.authenticate(HttpMethodDirector.java:232)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at com.gargoylesoftware.htmlunit.HttpWebConnection.getResponse(HttpWebConnection.java:97)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1477)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1435)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:327)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:388)
at com.test.Test.main(Test.java:25)
Jun 5, 2009 9:28:35 AM org.apache.commons.httpclient.HttpMethodDirector processProxyAuthChallenge
INFO: Failure authenticating with NTLM @companyproxy.server:8080
Jun 5, 2009 9:28:35 AM com.gargoylesoftware.htmlunit.WebClient printContentIfNecessary INFO: statusCode=[407] contentType=[text/html]
Jun 5, 2009 9:28:35 AM com.gargoylesoftware.htmlunit.WebClient printContentIfNecessary INFO:
Access Denied

....

Exception in thread "main" com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException:

407 Proxy Authentication Required for http://htmlunit.sourceforge.net/
at
com.gargoylesoftware.htmlunit.WebClient.throwFailingHttpStatusCodeExceptionIfNecessary(WebClient.java:535)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:332)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:388)
at com.test.Test.main(Test.java:25)

Can you please provide some info on this ?

Answer 1

Although you have not put the full stack trace in, I am guessing that the error is being thrown on the line:

final HtmlPage page = (HtmlPage)webClient.getPage(url);

This is because the getPage call is returning an UnexpectedPage rather than an HtmlPage. Looking at the documentation for UnexpectedPage, it appears the page request is coming back with a Content-type that is not text/html so htmlunit is not sure what to do with it. You should turn up the debugging and see what is actually coming back to figure out the error.

Answer 2

Hi,

As Rob said, HtmlUnit is not able to detect it is an HTML page.

Please provide sample to the user-list, so we can investigate further

Answer 3

I am not able to use HtmlUnit to do NTLM authentication on the proxy server. When i used HttpClient (HtmlUnit is built on top of this) and set the proxy setting with NTLM authentication.It worked. Here is the code for the same. HttpClient client = new HttpClient();
client.getHostConfiguration().setProxy("companyproxy.server", 8080);
List authPrefs = new ArrayList();
authPrefs.add(AuthPolicy.NTLM);

client.getState().setProxyCredentials(<br>
    new AuthScope(null, 8080, null),<br>
    new NTCredentials("username", "pwd", "", "DOMAIN"));<br>

client.getParams().setParameter(AuthPolicy.AUTH_SCHEME_PRIORITY, authPrefs);<br>

GetMethod method = new GetMethod(url);<br>

method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, 
     new DefaultHttpMethodRetryHandler(3, false));<br>