what is the best method using ruby/mechanize/nokogiri to go/click through all pages in case there is more than 1 page I need to access/click on? For example here Page 1 of 34 Should I click the page number or next? Or is out there any better solution?
A:
You should try out scrubyt. It's described as "mechanize on steroids". It has dedicated method for clicking through pages.
Eimantas
2010-01-16 11:40:31
thank you for that suggestion. So can I use scrubyt with mechanize?
Radek
2010-01-17 10:16:51
Scrubyt itself uses mechanize so you don't have to. It concentrates on crawling and scraping.
Eimantas
2010-01-17 10:51:03
+2
A:
It looks like the link ">" takes you to the next page, and it does not appear if you are on the last page. So:
page = ... # fetch the first page
loop do
# process the page
break unless link = page.link_with(:text=>'>')
page = link.click
end
Wayne Conrad
2010-01-17 05:00:57
@Wayne. As usual thank you so much. I will test/try and post the result here.But I already know that it will work.Everything you post works...
Radek
2010-01-17 10:15:38
You're welcome. I don't know about _everything_. I didn't test this fragment, so who knows?
Wayne Conrad
2010-01-17 10:30:01
@Wayne: I added bit of your code ad the end of a procedure that processes the page and then it the procedure calls itself in case the is new page. Lovely.
Radek
2010-01-27 19:02:21