trying to wget my stock portfolio

Question

I'm trying to read my stock portfolio into a script. The following works with NAB Online Trading but not Bell Direct.

• install the Export Domain Cookies Firefox addon
• log in to my online broker with Firefox
• save the domain cookies to a file (eg cookies.txt)
• wget --no-check-certificate --load-cookies=cookies.txt -O folio.htm https://...(portfolio URL)

-- The idea being to reuse the browser's login session. When I try it with Bell Direct, wget is redirected to the login page. I get the same results with curl. What am I missing? Is there some state that is stored in the browser besides in the cookies? Bell isn't using "basic authentication" because the login page is a form for username / password - it doesn't pop up the browser's built-in login dialog.

Here is what happens (under Windows XP with Cygwin):

$ wget --server-response --no-check-certificate --load-cookies=cookies-bell.txt -O folio-bell.htm https://www.belldirect.com.au/trade/portfoliomanager/
--2009-12-14 10:52:08-- https://www.belldirect.com.au/trade/portfoliomanager/
Resolving www.belldirect.com.au... 202.164.26.80
Connecting to www.belldirect.com.au|202.164.26.80|:443... connected.
WARNING: cannot verify www.belldirect.com.au's certificate, issued by '/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/CN=Thawte Server CA/[email protected]':
Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response...
HTTP/1.1 302 Found
Connection: keep-alive
Date: Sun, 13 Dec 2009 23:52:16 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Location: /account/login.html?redirect=https://www.belldirect.com.au/trade/portfoliomanager/index.html
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 229
Location: /account/login.html?redirect=https://www.belldirect.com.au/trade/portfoliomanager/index.html [following]
...

Answer 1

You need to POST the login form variables, then, with those cookies, goto the inner page.

http://www.trap17.com/index.php/automatic-login-curl%5Ft38162.html for some example code.

Answer 2

The login is encrypted over the HTTPS protocol, and you do not provide a certificate. Perhaps belldirect requires a valid certificate for client authentication.

You can export a certificate in Firefox by clicking the highlighted blue portion of the URL > More Information > Security Tab > View Certificate > Details > Export. Then, you can use the --certificate=filename option to specify the exported certificate in your wget command.

Answer 3

Maybe you need to set the referrer too.

Answer 4

Perhaps the server is validating the session based on User-Agent as well as the cookie. Check what user-agent your Firefox install is using (perhaps use WhatIsMyUserAgent.com if you don't know it), and try using that exact same user agent in your Wget call (via the --user-agent="..." parameter).