Can YQL parse web sites requiring cookie-based authentication?

Question

First, my use case:

I'm trying to use YQL's built in XPATH capabilities to scrape content from Yahoo! Fantasy Sports. It uses some sort of cookie-based authentication scheme. Basically, the sequence is:

1) Do an HTTP GET on the Yahoo! Login page 2) Parse the hidden inputs from the response and do an HTTP PUT with your Yahoo! Login on the form URL 3) Use the cookies returned from step 2 to GET any of the Fantasy Sports! websites that you have access to

My question is, does YQL support doing this to scrape data? The only authentication based examples I've seen use OAuth, but I haven't seen any examples of using YQL to parse websites with cookie-based authentication schemes

Answer 1

You will have to use an open table with an execute block.

You basically write some JavaScript that will be run on YQL's servers to do all the logic you need.