DB design for importing data from multiple providers?

Question

Hi,

I have to build a news database that will potentially take news from many different providers and display them on our site. Right now I am thinking that I should just come up with how it will look on our site, the fields we would display, and then when a new provider comes along, right a script that will parse their data into our format. Is this the best or should I copy their data exactly and then display more or less data based on the provider (logic checks)? I am, of course, leaning toward the first option, but wanted to make sure I'm not crazy.

Answer 1

I'd make very sure that the code and structure of the data for my site is my own.

Where, when and how you convert data from each provider to your own logic / data structure is a matter of choice. You could store it in provider specific tables and have an interface or service that dynamically converts it to what you want, or you could have an importer that does the conversion once and stores the converted information in your own structure.

Either way, I would code the conversion / import logic for each provider in its own class, and have each descend from a common ancestor or have each implement a common interface. That way you ensure that your own logic and data structures are shielded from the intricacies of the various news providers.

The factory and/or builder patterns will come in useful in this scenario as well. They will help separate your own generic import/conversion logic from the choice of which class to instantiate and all that is needed to set that up correctly for a specific news feed.