Downloading a web page with Android

Question

I'm downloading a web page then extracting some data out of it, using regex (don't yell at me, I know a proper parser would be better, but this is a very simple machine generated page). This works fine in the emulator, and on my phone when connected by wi-fi, but not on 3G - the string returned is not the same, and I don't get a match. I can imagine it has something to do with packet size or latency, but I can't figure it out.

My code:

public static String getPage(URL url) throws IOException {
    final URLConnection connection = url.openConnection();
    HttpGet httpRequest = null;

    try {
        httpRequest = new HttpGet(url.toURI());
    } catch (URISyntaxException e) {
        e.printStackTrace();
    }

    HttpClient httpclient = new DefaultHttpClient();
    HttpResponse response = (HttpResponse) httpclient.execute(httpRequest);

    HttpEntity entity = response.getEntity();
    BufferedHttpEntity bufHttpEntity = new BufferedHttpEntity(entity); 
    InputStream stream = bufHttpEntity.getContent();

    String ct = connection.getContentType();

    final BufferedReader reader;

    if (ct.indexOf("charset=") != -1) {
        ct = ct.substring(ct.indexOf("charset=") + 8);
        reader = new BufferedReader(new InputStreamReader(stream, ct));
    }else {
         reader = new BufferedReader(new InputStreamReader(stream));
    }

    final StringBuilder sb = new StringBuilder();

    String line;
    while ((line = reader.readLine()) != null) {
        sb.append(line);
    }

    stream.close();
    return sb.toString();
}

Is it my poor connection causing this, or is there a bug in there? Either way, how do I solve it?

---

Update: The file downloaded over 3G is 201 bytes smaller than the one over wi-fi. While they are obviously both downloading the correct page, the 3G one is missing a whole bunch of whitespace, and also some HTML comments that are present in the original page which I find a little strange. Does Android fetch pages differently on 3G as to reduce file size?

Answer 1

Here you go some hints, some of them silly hints, but just in case:

1. Review your mobile connection, try to open web browser, surf the web, and make sure it actually works
2. I don't know which is the web page your are trying to access but take into account that depending on your phone User Agent (UA), the rendered content might be different (web pages specially designed for mobile phones), or even no content rendered at all. Is it a web page on your own.
3. Try to access that same web page from Firefox, changing the UA (Use the User Agent Switcher for Firefox), and review the code returned.

That will be a good start point to figure out what's your problem

Ger

Answer 2

You may want to check if your provider has a transparent proxy in place with 3G.