ansaurus

Question

Using wget to pull csv from Google Trends

Answer 1

+2 A:

You are getting a redirect message. The URL in the location.replace bit and you get a valid index.html from Google is you that URL in a second call to wget.

Methinks you simply don't have the proper URL from where the csv data is downloaded. For a working example of how to 'hit' a CGI interface with a downloader, look at R's get.hist.quote() in the tseries package.

Edit: Here is what get.hist.quote() does:

R> IBM <- get.hist.quote("IBM")
trying URL 'http://chart.yahoo.com/table.csv?s=IBM&amp;a=0&amp;b=02&amp;c=1991&amp;d=9&amp;e=08&amp;f=2009&amp;g=d&amp;q=q&amp;y=0&amp;z=IBM&amp;x=.csv'
Content type 'text/csv' length unknown
opened URL
.......... .......... .......... .......... ..........
.......... .......... .......... .......... ..........
.......... .......... .......... .......... ..........
.......... .......... .......... .......... ..........
.......... .......... .......... ......
downloaded 236 Kb

R>

You could hit that same URL directly as shown in the code you could study. If you need cookies you may need to look at Duncan TL's code to hit Google Docs etc.

Dirk Eddelbuettel 2009-10-09 16:50:11

Dirk: get.hist.quote() uses download.file(), which has three options (wget, lynx, and internal). I tried each of them, but ran into errors. The url is valid in my web browser, so I still think I need to be able to authenticate one of these downloaders (wget, lynx, etc). Thoughts?

Christopher DuBois 2009-10-09 17:13:40

a) Yes, it worked for me. I get a 124 kB file index.html, else I would not have suggested the answer I offered above.b) If tseries::get.hist.quote does not work for you, then something else is amiss, maybe a proxy or something else. You probably need to sort that our first.c) I only offered get.hist.quote() to show you that may be able to all this from within R. That said, wget has a gazillion options and a detailed manual. Maybe time for RTFineM?

Dirk Eddelbuettel 2009-10-09 17:33:59

Hmm. I get a 124 kb index.html as well, but the original file I'm trying to download is a 23 kb csv file. (And rest assured, once I figure out the issue with wget, I will be bringing this back into R with download.file() or something.)

Christopher DuBois 2009-10-09 18:01:31

Same here. I get the csv if I do it from the browser is a session where I have Google cookies, but not from the command-line. You need to hit the wget docs / web to learn how to tell wget about the cookies used by your browser ... and the wget will masquerade as the browser. The csv is still ugly with the header / footer lines but we can tell R how to cope with that. So back to fixing cookies...

Dirk Eddelbuettel 2009-10-09 18:12:50

Christopher DuBois 2009-10-09 18:32:59

--load-cookies was the answer.

Christopher DuBois 2009-10-09 19:01:08

I had actually tried that too but my cookies.txt must have been to old old (as I switched to Chromium instead of Firefox and I had no time to chase the cookies.txt for Chromium...) Good to know you have it sorted out.

Dirk Eddelbuettel 2009-10-09 23:23:17

ansaurus

tags:

views:

answers:

Using wget to pull csv from Google Trends

related questions