Encountered java.io.FileNotFoundException while parsing a rss feed in java

Question

hi
I couldn't figure out the reason why i get this error while i try to parse an feed using sax parser.The code is simple and it has been an working code for many other url's.

    try{
         String myurl="http://news.google.com/news?ned=us&topic=n&output=rss"

        SAXParserFactory spf = SAXParserFactory.newInstance();
        spf.setValidating(false);
        SAXParser sp = spf.newSAXParser();
        XMLReader xr = sp.getXMLReader();
        xr.setContentHandler(this);

        URL com = new URL(urls);
        URLConnection con = com.openConnection();
        con.setConnectTimeout(20000);
        is = new InputSource(new InputStreamReader(con.getInputStream()));
        xr.parse(is);
    } catch (Exception e) {... }

The error i am getting is

  07-30 18:15:28.713: WARN/System.err(596): java.io.FileNotFoundException: http://news.google.com/news?ned=us&topic=n&output=rss
  07-30 18:15:28.763: WARN/System.err(596): at org.apache.harmony.luni.internal.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1064)
  07-30 18:15:28.763: WARN/System.err(596): at com.mycityway.commonparsers.CommonParser.createFeed(CommonParser.java:311)    
  ...

please suggest me the correct way of doing this

Answer 1

They do plain and simple browser spoofing by checking the User-Agent. If they don't like it, they send a 403 (forbidden). Simply add one generic and it works. The following line is the first thing I try whenever I encounter this problem and it works close to always:

URLConnection urlc = url.openConnection();
urlc.addRequestProperty("User-Agent", "firefox");