Is there a good tutorial for figuring out what a website is doing so your program can do the same thing?

Is there a good guide or tutorial for people who need to programmatically interact with dynamic websites? There's been a rash of Perl questions about that lately, and I haven't found a good resource to point people toward. I'm asking not because I need one but because I don't want to waste my time writing it if it already exists. Although I'm most interested in Perl, the extra tools and techniques are mostly the same.

Typically, I see see these problems in people's questions:

Handling, setting, and saving cookies
Finding and interacting with forms
Handling JavaScript inside your user-agent
- especially things like onLoad, onSumbit, and Ajax
Using HTTP sniffer tools
Using Web developer plugins in interactive browsers
Interacting with DOM, screen scraping, etc.

If there's no good tutorial, I'll add it to my list of things to do (unless someone else wants to do it). Along the way, if you don't have a suggestion for an existing tutorial, please suggest the things that you think should be in a new one, including links, your favorite tools, and your own user-agent development experiences. I don't care about the particular language you use.

I'm looking specifically for tutorials, not code. I already know how to do all this stuff. I'm looking to see if anyone has put something together to explain it to newbies.

brian d foy 2010-08-24 12:32:02

Furthermore, HTML::DOM has nothing to do with this problem.

brian d foy 2010-08-30 21:49:56

"Pragmatic programmer" - nice book read, and learn what means pragmatic.

przemo_li 2010-09-06 11:24:49

"Interacting with DOM", "HTML::DOM has nothing to do with this problem". ? ? ?

przemo_li 2010-09-06 11:28:52

Find a library that solve at least one problem you've specified, find tutorials, docs about it. It just may happened that someone put together some docs about those libs that solve more than just one topic. Or you will know what libs are used together so you will be able to repeat procedure.

przemo_li 2010-09-06 11:31:26

And no, I do not know any tuts. That simply fulfill all your needs. (and I do not know if I understand them properly - look up)

przemo_li 2010-09-06 11:32:44

Just curious why this is voted down....book contains many examples of code WITH explanations. not sure how someone would "programmatically interact" without code examples.

JiminyCricket 2010-08-30 14:32:07

I voted it down because it has nothing to do with the question I asked. Not only that, remember who's asking the question. I think I know about Perl already. :)

brian d foy 2010-08-30 16:04:33

@brian, your question is asking for tutorials on how to do different things in Perl for beginners and people who DONT know how to do them. I answered with a book that I used to learn Perl and was full of examples. Does that not answer your question?As you have surmised, I didn't look at your profile and see that you are the author of several Perl books. Clearly you are familiar with Perl. That doesn't change the fact that you asked about tutorials for beginners. Next time if you feel that everyone should reference your background before they answer the question, then you should state that.

JiminyCricket 2010-08-30 16:34:23

@Jiminy: brian d foy is kind of a big deal. You know who Jon Skeet is, right?

Matt Ball 2010-08-30 17:25:27

(In case the humor was missed, http://www.youtube.com/watch?v=gtKT0lwHNKo)

Matt Ball 2010-08-30 17:34:58

@bears I see your humor. will ferrell takes his character to the extremes of toolbaggery. I hope lessons can be learned for some people from that movie. there is such thing as humility is all I'm saying. if the authors professional history is part of the question, the author should say such for those who aren't familiar with them and not down-vote because they weren't recognized for their acclaim.

JiminyCricket 2010-08-30 19:09:16

I'm not asking a Perl question. If so, I would have tagged it with Perl. This question is about the process of interacting with websites and figuring out how to automate them. The answer will likely be the same regardless of the language used. However, you should also realize the Programming Perl covers none of these issues, so even if this were a Perl question you would still have an extremely poor answer.

brian d foy 2010-08-30 21:38:48

I didn't downvote you because you didn't recognize me. I downvoted you because your answer merited a downvote. However, when I answer questions, I also tend to look at a person's profile to see what other sorts of questions they ask or answer so I can judge their skill level. I also use my real name so you don't have to guess who's hiding behind the cartoon name. :)

brian d foy 2010-08-30 21:49:29

ok man. you said you didnt care what language you use, not dont include examples in a particular languages. not a big deal. i guess a matter of interpretation. will check out your new book when it drops. I am an active developer for a company that employs lots of scraping techniques in python. My tutorials are found here on Stack Overflow and other forums. I guess when I encounter the problem, I look for it here and other forums. would be nice if there was a "scraping central" to find out the best way to do things. I guess thats why people write books =)

JiminyCricket 2010-08-30 22:53:48

Well, you didn't even include examples, no matter the language. When you're in a hole, stop digging. I'm not sure what book you think I have coming out, but it's not going to be a web book.

brian d foy 2010-08-31 00:50:46

ok. btw there is a typo in your question. if i had more rep id fix it for you: in your bulleted list you have "onSumbit"

JiminyCricket 2010-08-31 14:13:19

"how to do it *all*" is probably a bit of an overstatement.

mrduclaw 2010-10-06 00:51:37

Interesting video. He also wrote _Webbots, Spiders and Screen Scrapers_ which probably has everything that I hoped someone had already put in a book. He also has [a website for the book](http://www.schrenk.com/nostarch/webbots/).

brian d foy 2010-08-30 21:51:57

ansaurus

tags:

views:

answers:

Is there a good tutorial for figuring out what a website is doing so your program can do the same thing?

related questions