Is there an efficient way to figure out the headers, cookies, and get/post data being passed to a site?

Question

More specifically I'm looking for something, perhaps an add-on for firefox, once enabled it logs all of this information as it's passed to and from the server. I'm doing some web scripting and this would be really handy.

If anyone is wondering specifically what I'm doing currently I'm trying to make a script to repost my craigslist ad every 2 days since I handle a few things on there. Might even go so far as to make a simple gui to manage the submissions.

I do suspect this goes against the ToS, for that reason I don't plan to release the code. Besides cl is already bad enough with spam, I'm not trying to contribute further to it, figured I'd say what I'm doing for the sake of being honest though. I don't have any bad intentions with this, just some things I've been trying to sell and an ad for my pc repair business. I've been reposting some things for months now and so often I just forget to do it.

Answer 1

The Network module of the Firefox Web Developer Toolbar lets you look at the HTTP headers in the request and the response, so it's a good starting point. It won't log everything for you, though, so you will have to copy everything to a text editor if you want to inspect it later.

Since you also tagged your post with Python, I presume you are planning to implement your script in Python. Take a look at the mechanize module which gives you a simple virtual browser object in Python that "remembers" cookies and stuff.

Answer 2

I've used the Firefox extension LiveHTTPHeaders and it has worked well for me.

Answer 3

Wireshark is great for logging and reading packets passed to and from your computer.

Answer 4

I find the Firebug net panel really useful for this.