tags:

views:

93

answers:

2
+1  Q: 

Good .NET based open source web crawler?

Hi,

For my new project I need to implement a .NET based web crawler. I searched for an open source option and found an entry here at SO that mentioned Arachnode.net as an open source solution. I visited arachnode.net and for my surprise, the project is fully commercial and there is no even a free community edition (if it's really an open source project).

Am I missing something about Arachnode.net? If no, is there a good open source alternative?

Thanks in advance :-)

+1  A: 

I don't know of any Open Source projects for .NET, but if your just looking to do something pretty basic you can easily use HttpWebRequest and HttpWebResponse to create your own web crawler. I've written many integration solutions that log in to remote website to process/scrape data using these objects. If you decide to go this route, just an FYI but I almost always need a CookieContainer to hold my session information and a few well tested RegularExpressions to do my scourging (e.g. Find all HREF, Regex("<a href.*?>")).

Good Luck!

Zachary  2010-09-18 07:32:28

related questions

Displaying Flash content in a C# WinForms application
How to get the value of built, encoded ViewState?
Unhandled Exception Handler in .NET 1.1
How do I connect to a database and loop over a recordset in C#?
How do I most elegantly express left join with aggregate SQL as LINQ query
Get a new object instance from a Type in C#
.NET Testing Framework Advice
Automatically update version number
What is the difference between an int and an Integer in Java/C#?
How to write to Web.Config in Medium Trust ?
WinForms ComboBox data binding gotcha
How do you sort a C# dictionary by value?
Adding Scripting functionality to .NET applications
Floating Point Number parsing: Is there a Catch All algorithm?
How do I print an HTML document from a web service?
Decoding T-SQL CAST in C#/VB.net
Anatomy of a "Memory Leak"
How do I get a distinct, ordered list of names from a DataTable using Linq
Reliable Timer in a Console Application
How do I fill a DataSet or a DataTable from a LINQ query resultset ?
What's the difference between Math.Floor() and Math.Truncate() in .NET?
How do I calculate relative time?
How do I calculate someone's age in C#?
Are there any conversion tools for porting Visual J# code to C#?
When setting a form's opacity should I use a decimal or double?