Where do I start with a web bot?

Question

I simply want to create an automatic script that can run (preferably) on a web-server, and simply 'clicks' on an object of a web page. I am new to Python or whatever language this would be used for so I thought I would go here to ask where to start! This may seem like I want the script to scam advertisements or do something illegal, but it's simply to interact with another website.

Sorry for asking, I hope you guys can provide me with some ground to start on!

Answer 1

I'd recommend the Python mechanize library. It's designed to act as a simulated browser. I've used it to drive several web interfaces from script.

Answer 2

I would recommend using the WebBrowser control of the .NET package. You can access all the DOM elements and fully interact with any website. Here is a brief article

If you still prefer python, mechanize might be a good way of doing that.

Answer 3

I'd probably start with Twill -- you can use its scripting language or Python API.

Answer 4

It doesn't have to be Python, I've seen it done in PHP and Perl, and you can probably do it in many other languages. The general approach is:

1) You give your app a URL and it makes an HTTP request to that URL. I think I have seen this done with php/wget. Probably many other ways to do it.

2) Scan the HTTP response for other URLs that you want to "click" (really, sending HTTP requests to them), and then send requests to those. Parsing the links usually requires some understanding of regular expressions (if you are not familiar with regular expressions, brush up on it - it's important stuff ;)).

Answer 5

Keep your eyes on Wade Cybertech (http://www.wadecybertech.com) that site is providing training and resources on webbot, spider, scraper development, mainly with Python but some PHP/CURL also.