PHP curl posting to aspx page causing 500 server error

Question

Hello,

This is a little complex question and a little hard to explain.

We have build a script to auto login and collect the information from one asp.net based website, the script is in PHP using curl. We have been testing the script and it works 20-30% of the time which means the script is complete and works. But rest of the time it fails, as the server gives 500 Internal Server error when we post the login details.

The same script and the same details work sometimes and does not work sometimes, can any one tell me what could be the cause of this kind of problems?

What should we check? We have set all the cookies, viewstate, event validation etc. Also tried to encode the special characters in post fields.

Thanks in Advance.

-happyhardik

---

Additions: - We don't have access to the .Net server - We have been giving the same input (login details) to the .Net server everytime we tried - The .Net website works fine in the browser every time (almost 100%)

Answer 1

Either the .NET application is broken, or your code is.

First thing to do is figure out what need's fixin':

Strip your PHP code down to some "bare essentials" test. The same (valid) username/password, every time. Run it 20 times or so...

How often does it succeed?

If you get 20-30% success rate, then it's a problem on the .NET side. (The same inputs to that system are producing different results).

If you get 100% or 0% success, it's probably something on your side (in some cases, you're sending bad requests)

If you have access to the .NET server (or the people who maintain it), get a look at what's causing the 500 error. There's bound to be some unhandled exception somewhere in their logs.

Answer 2

from my experience if you send requests too fast , some sites will try to slow u down with different kind of errors.. try to put a sleep between ur requests and see if that helps.

Answer 3

We have set all the cookies, viewstate, event validation etc. Also tried to encode the special characters in post fields.

How do you know that your cookies and viewstate are correct for the machine you are sending the requests from? Or that they are correct for the server you are sending the requests to?

ViewState is an encoded representation of the ASP.NET page fields at the time it was sent to a browser, and is encoded based on various things, including the MachineKey of the server, a hash of the page code, and potentially the MAC address of the server - get any one of those wrong for what appears to be a postback and the server will throw an HttpException with a message of "Invalid Viewstate".

There are any number of other reasons why a server may throw an exception back out at you, ranging from other errors on the site code, through to it being too busy to handle requests, to as Sab suggests active rejecting of your requests. Your best bet would be to try and work with the site owner to come up with a solution that both parties can work with - however I'm inclined to assume that you're doing some sort of semi-illicit screen scraping that the sites owners wouldn't be two happy with, otherwise they would have already provided you with an API.