I am trying to get yahoo options data into mysql using perl. I found modules to download stock prices, but not options data. Anyone know what would be the easiest way to do that? If I have to roll my own, what modules should I use? Thanks, CP
+3
A:
I don't know of any specific modules for Yahoo! Finance options, but if you want to roll your own you can usw LWP::UserAgent to download the page in question and something like HTML::TreeBuilder to parse it. If a specific option is the one you need data for, Yahoo! may even offer an RSS feed of its data in which case it's probably more robust to use something like XML::RSS to parse the feed.
Do you have a link to a page of the kind of data you'd like to get?
jasonmp85
2010-05-30 01:33:13
No they don't have a rss feed for options, but here is an example link: http://finance.yahoo.com/q/op?s=IBM+OptionsI saw Template::Extract and the idea sounds great, but I am unable to get it to work correctly.
CptanPanic
2010-05-30 01:46:13
jasonmp85
2010-05-30 01:51:12
Don't use `split` to parse CSVs, you never know when you'll run into quoted values with embedded commas, and other gotchas. Test::CSV_XS http://search.cpan.org/~hmbrand/Text-CSV_XS/CSV_XS.pm is much better.
friedo
2010-05-30 03:55:04
Well sure the caveat about quoted values with commas applies. I'm sure a sufficiently ugly regex could deal with it. But it's important to get something up and running and then iterate on it. CptanPanic can probably get something decent running with just LWP::UserAgent and regexes, and then add robustness to flavor :-D.
jasonmp85
2010-05-30 06:07:21
Ok I go the route where I follow each link to download the csv file, what would be the easiest way to get all the links on the main page that point to each option listed? For example get all the links that look like <a href="/q?s=IBM100619C00100000">IBM100619C00100000</a>
CptanPanic
2010-05-30 18:03:06
A:
Perl is not always the most straightforward choice. Is there a reason why you aren't looking into using Yahoo! Pipes to pull in the data you're after and log it to file? There's nothing stopping you from using Perl to populate your MySQL database from that file at a later date.
If you still feel that Perl is the way to go, WWW::Mechanize is a very good starting point for your screen-scraping needs.
Zaid
2010-05-30 06:40:02
There is a pipe made to download options data, but I don't see how I can use the data? For example http://pipes.yahoo.com/protovist/options, it doesn't show any rss feed, so how can I use this?
CptanPanic
2010-05-30 18:05:27
I thought that Pipes was all about customizing data-collection to suit your needs. Don't expect a cut-and-dried solution to exist out there, you'll have to tailor it to suit your needs.
Zaid
2010-05-30 18:29:32
+1
A:
The Finance::Quote module implements the lookup on Yahoo and a number of other sources.
Gene Vincent
2010-05-30 11:09:34