Make a Web Crawler/ Spider

views:

139

answers:

+3 Q:

Make a Web Crawler/ Spider

Hi,

I'm looking into making a web crawler/ spider but I need someone to point me in the right direction to get started.

Basically, my spider is going to search for audio files and index them.

I'm just wondering if anyone has any ideas for how I should do it. I've heard having it done in php would be extremely slow. I know vb.net so could that come in handy?

I was thinking about using googles filetype search to get links to crawl, would that be ok?

Thanks Guys

+1 A:

Here is a link on a tutorial on how to write a web crawler in java. http://java.sun.com/developer/technicalArticles/ThirdParty/WebCrawler/ I'm sure if you google it you can find ones for other languages.

qw3n 2010-07-09 03:03:02

If you make a web crawler in java does it have to be server side? Right now I'm on shared hosting which doesn't allow java and I currently cannot afford to get a dedicated or vps.

Belgin Fish 2010-07-09 03:04:24

No this could be run on your home computer if you wanted to.

qw3n 2010-07-09 03:46:44

+2 A:

In VB.NET you will need to firstly get the HTML, use the WebClient class or HttpWebRequest and HttpWebResponse classes. There is plenty of info on how to use these on the interweb. Then you will need to parse the HTML, i recommend using regular expressions for this.

Your idea of using google with filetype search is a good one, i did a similar thing a few years ago to gather PDF's to test PDF indexing in SharePoint, it worked really well.

Chris Diver 2010-07-09 03:07:38

Thanks, any idea how I could insert data into my database form a desktop vb app?

Belgin Fish 2010-07-09 03:08:49

Depends on the flavor of database. There is the `System.Data.SqlClient` namespace for SQL Server. For anything else you will need to look at `System.Data.OleDb` namespace. It's better to use a console VB app if you want this to run unattended

Chris Diver 2010-07-09 03:14:16

In regards to parsing HTML with regex....http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tagsProbably the single best SO answer ever.

rfusca 2010-07-09 03:40:17

Thanks for the link, i suppose 'parse' was the wrong word to choose, he will just extract all relevant hyperlinks from the page, doesn't matter about the structure of the HTML.

Chris Diver 2010-07-09 04:03:23

ansaurus

tags:

views:

answers:

Make a Web Crawler/ Spider

related questions