need help with bizarre java.net.HttpURLConnection behavior

Question

I am attempting to download a jpg using HttpURLConnection and am encountering a very peculiar bug.

Here's the url: http://www.vh1.com/sitewide/promoimages/shows/m/my%5Fantonio/video/supertrailer/seg%5F1/320x240.jpg

if you open it in a browser you will see the image.

However, when I use HttpURLConnection I don't get the image... What I get is a 301 which, quite strangely, redirects to http://wap.vh1.com

so

    con.setInstanceFollowRedirects(true);
//additional stream code here to go and get the stuff found in con

proceeds to go ahead and download the text from wap.vh1.com, rather than the jpg that you see in the browser.

I'm guessing that there is some header wackiness that's causing this, but I haven't the faintest idea what the host is expecting to see in order to redirect me to the same place as where it's redirecting the browser (and curl and wget and everything else I can think to point at it).

I'm just about ready to shoot myself, so, if you help me you will be preventing my 6 year old daughter from going fatherless.

Answer 1

The site redirects you based on user-agent. Add this before you open the connection,

conn.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.15) Gecko/2009101600 Firefox/3.0.15");

Answer 2

While I can't help you with your specific problem, here's what I would do:

• Download wireshark, sniff the HTTP request sent by your java application

• Copy/Paste the request, and run it with telnet (or a tool such as WFetch)

• Fiddle with the request headers and see if behavior changes.

(I'd suspect the site screens the request based on the User agent header or something similar)

Answer 3

For more flexibility you can utilize the http commons libraries, which have great debugging support for the wire through log4j ...

Also, user agents and more request parameters can be set easily.

For more information, see their tutorial.

Answer 4

The java.net package doesn't support lot of the needed features out of the box (like automatically saving and sending cookies). Use Apache's httpClient instead

Answer 5

It seems like the server interprets your request as coming from a mobile device, possibly based on the User-Agent header. That's why your redirected to the mobile site. Try setting the User-Agent explicitly.