Automate Post Of Login Details User & Password Into Safari For Scraping

Question

Hi To All,

I am wanting to automate the input of post variables on a login page for the purpose of webscraping. It would improve the process no end if I can get past the login page.

Then I can schedule some functions to run on cycle automatically. (Had a go with some CURL commands but could not get the result)

Thanks for any help,

lance

Answer 1

Try looking at this article for some suggestions.

The magic part is:

tell application "Safari"
  activate
  tell (make new document) to set URL to "https://www.fedex.com/ship/shipEntryAction.do?method=doInitialEntry&origincountry=us&locallang=en&urlparams=us&sType=&programIndicator=0"
  delay 2

  set doc to document "FedEx | Ship Manager | Shipping"
  log (doc's name)
  do JavaScript "document.forms['domesticShipmentActionForm']['recipient.companyName'].value = 'add'" in doc
  do JavaScript "submitForm('doSelectRecipientCompanyName')" in doc
end tell

Answer 2

You can use TestPlan to do this. You can fully automate your tests, or you could just leave the browser open at the end and continue on your own manually.

For example, a login might look like:

GotoURL http://somesite.com/
SubmitForm with
  %Params% with
    %name% mylogin
    %pass% mypassword
  end
end

It works with both a browserless mode or with a backing browser. I've only tested that with explorer and firefox, but I'd be happy to help you get it running with Safari if you want.