log in with browser and then ruby/mechanize takes it over?

Question

is that even possible? what I need to pass to mechanize? With what url I can start then?

I cannot manage (so far) to log into one website using mechanize so I was thinking if I can do this little workaround. I believe I can capture all cookies and everything else and then pass it to ruby/mechanize to do the rest ...

screenshots below are made using firebug ( Firebug logs the POST or GET request, the response headers )

login that works = just one line

and html for login that works

User Name Password Save?

login that doesn't work for me

and it's html

<form class="login" method="post"> <fieldset>
<legend>Members Login</legend> 

<div>
<label for="auth_username">Username</label> <input id="auth_username" name="auth_username">
</div>

<div>
<label for="auth_password">Password</label> <input id="auth_password" name="auth_password" type="password">
</div>

</fieldset>
<div class="buttons">
<input name="auth_login" type="submit" value="Login"><p class="note"><a href="/forgotpassword">Forgot your password?</a></p>

</div>

</form>

my script is almost the same in both cases.

require 'rubygems'
require 'mechanize'
#agent = WWW::Mechanize.new

agent = WWW::Mechanize.new 

page = agent.get("http://www.vbulletin.org/forum/index.php")

login_form = page.form_with(:action => 'login.php?do=login')


puts 
login_form.fields.each { |f| puts "#{f.name} : #{f.value}" }    
login_form['vb_login_username'] = 'user name'
login_form['vb_login_password'] = ''

page = agent.submit login_form

output = File.open("login.html", "w") {|f| f.write(page.parser.to_html) }

mechanize log for the login that doesn't work

 INFO -- : Net::HTTP::Get: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 400 bytes
DEBUG -- : Read 1424 bytes
DEBUG -- : Read 2448 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:12 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: set-cookie => PHPSESSID=7cfilg86ju2ldcgso22246hpu4; path=/, WebStats:visitorId=lSMkcwuSWEE%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/, WebStats:sessionId=%2B2HHK296t%2BQ%3D; expires=Mon, 27-Jan-2020 23:43:12 GMT; path=/
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
DEBUG -- : saved cookie: PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : saved cookie: WebStats:visitorId=lSMkcwuSWEE%3D
DEBUG -- : saved cookie: WebStats:sessionId=%2B2HHK296t%2BQ%3D
 INFO -- : status: 200
DEBUG -- : query: "auth_username=radek&auth_password=mypassword"
 INFO -- : Net::HTTP::Post: /login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-language => en-us,en;q=0.5
DEBUG -- : request-header: connection => keep-alive
DEBUG -- : request-header: accept => */*
DEBUG -- : request-header: accept-encoding => gzip,identity
DEBUG -- : request-header: content-type => application/x-www-form-urlencoded
DEBUG -- : request-header: user-agent => WWW-Mechanize/0.9.3 (http://rubyforge.org/projects/mechanize/)
DEBUG -- : request-header: cookie => WebStats:sessionId=%2B2HHK296t%2BQ%3D; WebStats:visitorId=lSMkcwuSWEE%3D; PHPSESSID=7cfilg86ju2ldcgso22246hpu4
DEBUG -- : request-header: referer => http://www.somedomain.com/login?auth_successurl=http://www.somedomain.com/forum/yota?baz_r=1
DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
DEBUG -- : request-header: content-length => 43
DEBUG -- : request-header: host => www.somedomain.com
DEBUG -- : request-header: keep-alive => 300
DEBUG -- : Read 650 bytes
DEBUG -- : Read 1674 bytes
DEBUG -- : Read 2698 bytes
DEBUG -- : Read 3211 bytes
DEBUG -- : response-header: vary => Accept-Encoding
DEBUG -- : response-header: cache-control => no-store, no-cache, must-revalidate, post-check=0, pre-check=0
DEBUG -- : response-header: connection => close
DEBUG -- : response-header: expires => Thu, 19 Nov 1981 08:52:00 GMT
DEBUG -- : response-header: content-type => text/html; charset=utf-8
DEBUG -- : response-header: date => Fri, 29 Jan 2010 23:43:13 GMT
DEBUG -- : response-header: content-encoding => gzip
DEBUG -- : response-header: server => Apache/2.2.3 (CentOS)
DEBUG -- : response-header: content-length => 3211
DEBUG -- : response-header: accept-ranges => bytes
DEBUG -- : response-header: pragma => no-cache
DEBUG -- : gunzip body
 INFO -- : status: 200

Answer 1

Yes, capturing cookies (eg, through FireCookies Plugin in Firefox), and manually passing it to mechanize could work for most cases.

Answer 2

You're having trouble using Mechanize to log into a web site that issues cookies? I think Mechanize manages that for you:

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

I wonder if cookies are really the problem.