How to check if a URL exists or returns 404 with Java?

Question

String urlString = "http://www.nbc.com/Heroes/novels/downloads/Heroes_novel_001.pdf";
URL url = new URL(urlString);
if(/* Url does not return 404 */) {
    System.out.println("exists");
} else {
    System.out.println("does not exists");
}
urlString = "http://www.nbc.com/Heroes/novels/downloads/Heroes_novel_190.pdf";
url = new URL(urlString);
if(/* Url does not return 404 */) {
    System.out.println("exists");
} else {
    System.out.println("does not exists");
}

This should print

exists
does not exists

TEST

public static String URL = "http://www.nbc.com/Heroes/novels/downloads/";

public static int getResponseCode(String urlString) throws MalformedURLException, IOException {
    URL u = new URL(urlString); 
    HttpURLConnection huc =  (HttpURLConnection)  u.openConnection(); 
    huc.setRequestMethod("GET"); 
    huc.connect(); 
    return huc.getResponseCode();
}

System.out.println(getResponseCode(URL + "Heroes_novel_001.pdf")); 
System.out.println(getResponseCode(URL + "Heroes_novel_190.pdf"));   
System.out.println(getResponseCode("http://www.example.com")); 
System.out.println(getResponseCode("http://www.example.com/junk"));

Output

200
200
200
404

SOLUTION

Add the next line before .connect() and the output would be 200, 404, 200, 404

huc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)");

Answer 1

Use HttpUrlConnection by calling openConnection() on your URL object.

getResponseCode() will give you the HTTP response once you've read from the connection.

e.g.

   URL u = new URL ( "http://www.example.com/" ); 
   HttpURLConnection huc =  ( HttpURLConnection )  u.openConnection (); 
   huc.setRequestMethod ("GET"); 
   huc.connect () ; 
   OutputStream os = huc.getOutputStream (  ) ; 
   int code = huc.getResponseCode (  ) ;

(not tested)

Answer 2

There is nothing wrong with your code. It's the NBC.com doing tricks on you. When NBC.com decides that your browser is not capable of displaying PDF, it simply sends back a webpage regardless what you are requesting, even if it doesn't exist.

You need to trick it back by telling it your browser is capable, something like,

conn.setRequestProperty("User-Agent",
    "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.13) Gecko/2009073021 Firefox/3.0.13");

Answer 3

You may want to add

HttpURLConnection.setFollowRedirects(false);
// note : or
//        huc.setInstanceFollowRedirects(false)

if you don't want to follow redirection (3XX)

Instead of doing a "GET", a "HEAD" is all you need.

huc.setRequestMethod("HEAD");
return (huc.getResponseCode() == HttpURLConnection.HTTP_OK);

Answer 4

I have following pb

on my server IP xxx.yyy.zzz.ttt, there are: 1 webapps/app1 with url www.app1.com 2 webapps/app2 with url www.app2.com

when I try download one file file.zip by www.app1.com/temp/file.zip => I hit error 404 but it's OK with IP: xxx.yyy.zzz.ttt/app1/temp/file.zip

what is wrong here ? thanks a lot dsea