tags:

views:

42

answers:

0

I am trying to implement the webrequest get and post methods. I am accessing a secure site that requires login authentication and was told that instead of sending a login request all the time; login to the site and capture the cookie and use this to send it in the header to get results from the page. In other words let us use google for example. Lets say google required u to login before searching for something. Now I login through my browser and leave the site logged in. Now I send a httwebrequest from my program that includes the cookie authentication details in the request header and get results for say ?param=sports. Now when I increment the page number like &page=3, I am still only getting page 1 results. Any ideas?

Here is some code:

Public Function WebsRequest(ByVal method__1 As String, ByVal url As String, ByRef postData As String) As String

        Dim webRequest As HttpWebRequest = Nothing
        Dim requestWriter As StreamWriter = Nothing
        Dim responseData As String = Nothing
        Dim proxylist As New ArrayList
        Dim IP As String = String.Empty
        Dim cookiedata As String = "something"


        webRequest = TryCast(System.Net.WebRequest.Create(url), HttpWebRequest)

        Thread.Sleep(New TimeSpan(0, 0, 10))

        webRequest.Headers.Add("Cookie", cookiedata)
        webRequest.Method = method__1.ToString()
        webRequest.ServicePoint.Expect100Continue = True
        webRequest.ContentType = "application/x-www-form-urlencoded"
        webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 GTB7.1 ( .NET CLR 3.5.30729; .NET4.0E)"
        webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
        webRequest.KeepAlive = True

        Thread.Sleep(30000)

        If webRequest Is Nothing Then

            Throw New NullReferenceException("request is not a http request")

        End If
        Try
            Dim response As HttpWebResponse = DirectCast(webRequest.GetResponse(), HttpWebResponse)

            Using stream As Stream = response.GetResponseStream()
                Using reader As New StreamReader(stream)
                    'now we can scrape the contents of the secure page as needed
                    'since the page contents is now stored in our pagedata string
                    responseData = reader.ReadToEnd()
                End Using
                stream.Dispose()
            End Using
            response.Close()

        Catch ex As Exception
            ex.Message.ToString(ex)
        End Try

        webRequest = Nothing


        Return responseData

    End Function

Now I am using the request method GET. This is in the actual request header from the original request.

Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Thu, 15 Jul 2010 19:56:03 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 2404
Connection: keep-alive


Host: www.
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 GTB7.1 ( .NET CLR 3.5.30729; .NET4.0E)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: something

I am confused with this issue and would appreciate some hekp/