Hello Everyone, Is it possible to crawl user-authenticated websites using c#? Thnx
+1
A:
Yes, it is possible assuming you have the credentials. Depending on the authentication mechanism used by the site different methods exist. WebClient could be used to send HTTP requests to the remote sites.
Darin Dimitrov
2010-07-31 08:04:05
Thanks, but is it possible by using sockets?
Jojo
2010-07-31 08:16:03
Yes but that would be insane if the site uses HTTP protocol.
Darin Dimitrov
2010-07-31 08:19:45
A:
To crawl user-authenticated websites, your crawler should be able to handle cookies and POST-requests.
The question is: do you want your crawler to sign up, validate itself by email and then sign in to crawl all contents?
floatless
2010-07-31 08:10:52
i already have the credentials, i think its using form authentication. Can i use c# sockets to do this? What commands should I use? thanks...:)
Jojo
2010-07-31 08:17:32
I suppose you could use **WebClient** or **HttpWebRequest**, as @Darin Dimitrov suggests.
floatless
2010-07-31 08:24:03
Yes, sockets are a bad idea, lots of work for no reason (unless you have one of course?). If you're also asking 'what commands should I use' then it sounds like you haven't read up on the HTTP protocol; in which case using the .NET web classes is an even better idea
Kieren Johnstone
2010-07-31 08:26:29