How to analyze IE activity when opening a specific web page

Question

I'd like to retrieve data from a specific webpage by using urllib library. The problem is that in order to open this page some data should be sent to the server before. If I do it with IE, i need to update first some checkboxes and then press "display data" button, which opens the desired page. Looking into the source code, I see that pressing "display data" submits some kind of form - there is no specific url address there. I cannot figure out by looking at the code what paramaters are sent to the server... I think that maybe the simpler way to do that would be to analyze the communication between the IE and the webserver after pressing the "display data" button. If I could see explicitly what IE does, I could mimic it with urllib.

What is the easiest way to do that?

Answer 1

You can use a web debugging proxy (e.g. Fiddler, Charles) or a browser addon (e.g. HttpFox, TamperData) or a packet sniffer (e.g. Wireshark).

Answer 2

An HTML debugging proxy would be the best tool to use in this situation. As you're using IE, I recommend Fiddler, as it is developed by Microsoft and automatically integrates with Internet Explorer through a plugin. I personally use Fiddler all the time, and it is a really helpful tool, as I'm building an app that mimics a user's browsing session with a website. Fiddler has really good debugging of request parameters, responses, and can even decode encrypted packets.