I am sending a GET request to a site and would like to know what would be the proper way to do this based on the following parameters.
Host: www.somesite User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 GTB7.1 ( .NET CLR 3.5.30729; .NET4.0E) Accept: text/javascript, text/html, application/xml, text/xml, / Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive X-Requested-With: XMLHttpRequest X-Prototype-Version: 1.6.1 Content-Type: application/x-www-form-urlencoded; charset=UTF-8 Referer: http://www.somewebsite.com/search/ Content-Length: 19 Cookie: __some cookie Pragma: no-cache Cache-Control: no-cache
I did use firebug to get this and now am trying to create my own request header as follows:
webRequest = TryCast(System.Net.WebRequest.Create(url), HttpWebRequest)
Thread.Sleep(New TimeSpan(0, 0, 10))
'webRequest.Credentials = credentials
webRequest.Headers.Add("Cookie", cookielogin)
webRequest.Method = method__1.ToString()
webRequest.ServicePoint.Expect100Continue = True
webRequest.ContentType = "application/x-www-form-urlencoded"
webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 GTB7.1 ( .NET CLR 3.5.30729; .NET4.0E)"
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
webRequest.KeepAlive = True
my url that I have shows in firebug post header the following parameters:
ajax 1
page 2
q item
Now I have included this in my get request since I need to retrieve multiple pages but I only get page 1 back. Am I missing something