Automate downloads from password protected website

Question

I need some help with a work project I have been assigned. At the moment we manually go to the site, logon and then download 2 excel files from a supplier's website every month. The files are then loaded into SQL.

We want to automate this process. Now the loading of the files into SQL I can do, but I am not sure how I can automate logging onto the website entering my user details and collecting the files. I mostly deal with SQL and have very little .NET experience, so any code samples would be most appreciated.

Just to confirm. The logon form is on a aspx page. just a basic form with a table containing the username & password fields, the forgotten password link and the logon button

Answer 1

You can either use webclient or httpwebrequest.

http://stackoverflow.com/questions/450380/login-to-the-page-with-httpwebrequest

http://stackoverflow.com/questions/611092/how-do-you-login-to-a-webpage-and-retrieve-its-content-in-c

Httpwebrequest example:

    HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create("http://sso.bhmobile.ba/sso/login"); 
    req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)"; 
    req.Method = "POST";
    req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
    req.Headers.Add("Accept-Language: en-us,en;q=0.5");
    req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7");
    req.KeepAlive = true;
    req.Headers.Add("Keep-Alive: 300");
    req.Referer ="http://sso.bhmobile.ba/sso/login";

    req.ContentType = "application/x-www-form-urlencoded"; 

    String Username = "username";
    String PassWord = "Password";

    StreamWriter sw = new StreamWriter(req.GetRequestStream());
    sw.Write("application=portal&url=http%3A%2F%2Fwww.bhmobile.ba%2Fportal%2Fredirect%3Bjsessionid%3D1C568AAA1FB8B5C757CF5F68BE6ECE65%3Ftype%3Dssologin%26url%3D%2Fportal%2Fshow%3Bjsessionid%3D1C568AAA1FB8B5C757CF5F68BE6ECE65%3Fidc%3D1023278&realm=sso&userid=" + Username + "&password=" + password + "&x=16&y=11");
    sw.Close();

    HttpWebResponse response = (HttpWebResponse)req.GetResponse();


    StreamReader reader = new StreamReader(response.GetResponseStream());
    string tmp = reader.ReadToEnd();

    foreach (Cookie cook in response.Cookies)
    {
        tmp += "\n" + cook.Name + ": " + cook.Value;
    }


    Response.Write(tmp);
    Response.End();

Webclient example:

WebClient wc = new WebClient();
wc.Credentials = new NetworkCredential("username", "password");
string url = "http://foo.com";                  
try
{
        using (Stream stream = wc.OpenRead(new Uri(url)))
        {
                using (StreamReader reader = new StreamReader(stream))
            {
                return reader.ReadToEnd();
             }
        }
}
catch (WebException e)
{
        //Error handeling
}