ruby mechanize in Facebook

Question

I'm trying to click the Settings button on the home page, but when I do I get this page back:

#<WWW::Mechanize::Page
 {url
  #<URI::HTTP:0x1023c5fc0 URL:http://www.facebook.com/editaccount.php?ref=mb&amp;drop&gt;}
 {meta}
{title nil}
{iframes}
{frames}
{links}
{forms}>

which is.. kinda empty! Is there some problems with these iframes and frames stuff maybe?

Answer 1

Facebook redirects me to: https://register.facebook.com/editaccount.php which I assume is the final destination. Assuming that WWW::Mechanize is set up to follow https redirects you should end up there too.

Much of facebook like most modern websites is generated by javascript which I think that WWW::Mechanize is unable to cope with, this could be the source of your problem. I recommend trying to scrape while appending "?_fb_noscript=1" to the url's you visit. This turns off much of facebooks javascript system and should enable a smoother ride for your little bot.

(Do remember this is only an idea and doubtless whatever you do is against facebooks usage policy and this makes you a "baddy." I don't condone such badness and beleve that baddies should be forced to go to bed early etc... ad nauseum)