tags:

views:

114

answers:

2
Q: 

How do I data mine various news sources?

I'm working on a free web application that will analyze top news stories throughout the day and provide stats. Most news websites offer RSS feeds, which works fine for knowing which stories to retrieve. However, the problems arise when attempting to get the full news story from the news website itself. At the moment, I have separate NewsSource classes for each source (CNN, NY Times, etc) that read the appropriate RSS feed(s), follows each link, and strips out the body. This seems tedious and very unmanageable when a news website decides to change the HTML structure of their articles.

Is there a service (preferably free) that already aggregates multiple news sources with the full article content (not just a summary)? If not, do you have any suggestions for handling multiple sources with different HTML structures that may change without notice?

A: 

I know this isn't a great answer, but I forget the name of the startup here in Colorado that can take unstructured/semistructured data and parse it into a structured format. I think if you search the coloradostartups blog for 'data' you might find it.

ybakos  2009-09-09 20:40:06
A: 

Spinn3r is a paid service that does what you want. If you are from academia they might give you free access to their data.

MrAnonymous  2010-09-13 03:49:12

related questions

Displaying Flash content in a C# WinForms application
How to get the value of built, encoded ViewState?
Unhandled Exception Handler in .NET 1.1
How do I connect to a database and loop over a recordset in C#?
How do I most elegantly express left join with aggregate SQL as LINQ query
Get a new object instance from a Type in C#
.NET Testing Framework Advice
Automatically update version number
What is the difference between an int and an Integer in Java/C#?
How to write to Web.Config in Medium Trust ?
WinForms ComboBox data binding gotcha
How do you sort a C# dictionary by value?
Adding Scripting functionality to .NET applications
Floating Point Number parsing: Is there a Catch All algorithm?
How do I print an HTML document from a web service?
Decoding T-SQL CAST in C#/VB.net
Anatomy of a "Memory Leak"
How do I get a distinct, ordered list of names from a DataTable using Linq
Reliable Timer in a Console Application
How do I fill a DataSet or a DataTable from a LINQ query resultset ?
What's the difference between Math.Floor() and Math.Truncate() in .NET?
How do I calculate relative time?
How do I calculate someone's age in C#?
Are there any conversion tools for porting Visual J# code to C#?
When setting a form's opacity should I use a decimal or double?