I am trying to implement the webrequest get and post methods. I am accessing a secure site that requires login authentication and was told that instead of sending a login request all the time; login to the site and capture the cookie and use this to send it in the header to get results from the page. In other words let us use google for example. Lets say google required u to login before searching for something. Now I login through my browser and leave the site logged in. Now I send a httwebrequest from my program that includes the cookie authentication details in the request header and get results for say ?param=sports. Now when I increment the page number like &page=3, I am still only getting page 1 results. Any ideas?
Here is some code:
Public Function WebsRequest(ByVal method__1 As String, ByVal url As String, ByRef postData As String) As String
Dim webRequest As HttpWebRequest = Nothing
Dim requestWriter As StreamWriter = Nothing
Dim responseData As String = Nothing
Dim proxylist As New ArrayList
Dim IP As String = String.Empty
Dim cookiedata As String = "something"
webRequest = TryCast(System.Net.WebRequest.Create(url), HttpWebRequest)
Thread.Sleep(New TimeSpan(0, 0, 10))
webRequest.Headers.Add("Cookie", cookiedata)
webRequest.Method = method__1.ToString()
webRequest.ServicePoint.Expect100Continue = True
webRequest.ContentType = "application/x-www-form-urlencoded"
webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 GTB7.1 ( .NET CLR 3.5.30729; .NET4.0E)"
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
webRequest.KeepAlive = True
Thread.Sleep(30000)
If webRequest Is Nothing Then
Throw New NullReferenceException("request is not a http request")
End If
Try
Dim response As HttpWebResponse = DirectCast(webRequest.GetResponse(), HttpWebResponse)
Using stream As Stream = response.GetResponseStream()
Using reader As New StreamReader(stream)
'now we can scrape the contents of the secure page as needed
'since the page contents is now stored in our pagedata string
responseData = reader.ReadToEnd()
End Using
stream.Dispose()
End Using
response.Close()
Catch ex As Exception
ex.Message.ToString(ex)
End Try
webRequest = Nothing
Return responseData
End Function
Now I am using the request method GET. This is in the actual request header from the original request.
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Thu, 15 Jul 2010 19:56:03 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 2404
Connection: keep-alive
Host: www.
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 GTB7.1 ( .NET CLR 3.5.30729; .NET4.0E)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: something
I am confused with this issue and would appreciate some hekp/