I am trying to scrape a webpage when it throws a "potentially dangerous script" error. Everytime I do this, I get a server 500 but I can scrape a normal, operating page. Is there a way I can scrape a webpage when it throws an error?
Thanks
+1
A:
Just a guess, but maybe the error thrown, triggers a server error (500)? That would make sense. You should be still able to get the output from the page.
Generally though, and I hope you don't mind, I'd have to object to your error handling. First off, you could put the error in a log and use a tool to monitor it for errors. Secondly, if your script throws an exception, you could hand it off to a service like Exceptional, which in turn notifies you.
Till
2009-04-29 09:34:58
Hi, I don't mind your comments :) I was hoping for a way to do the above without using any error logging. My code is as follows: HttpWebRequest request = (HttpWebRequest)WebRequest.Create(currentUrl); HttpWebResponse response = (HttpWebResponse) request.GetResponse();At the response, it is null and the exception comes up. I can't get any details about it. I get the same problem with webclient etc. The url is the url of the search page with the querystring (what user entered in text box).
dotnetdev
2009-04-29 09:42:54
I'm not enough familiar with ASP to be of real help. Aside from the 'error', do you get an error message though? If so, what is it.
Till
2009-04-29 09:47:23
Just the standard "A potentially dangerous Request.QueryString value was detected from the client (culprit script here)."
dotnetdev
2009-04-29 09:49:08
+1
A:
Why not use ASP.NET health monitoring? There's a mail provider that will automatically send the error details to you without any manual scraping and you can configure it all with just some config file entries. Easy!
Troy Hunt
2009-04-29 09:52:21
A:
Consider using ELMAH to log all errors. You'll get lots of useful additional information also.
More details at:
http://msdn.microsoft.com/en-us/library/aa479332.aspx http://www.hanselman.com/blog/ELMAHErrorLoggingModulesAndHandlersForASPNETAndMVCToo.aspx
Matt Lacey
2009-04-29 09:58:13