How can I efficiently and easily import/export large business data in .NET ?

Question

Hi,

What would be the best data format (XML, Database, etc.) for large business data when we want to efficiently and easily load all of it (import) into business objects in .NET ?

I want to load all the data in one go and create all business objects. The data must have a kind of a "schema" to enable other people to create it.

Also note that I want to ease the process of creating the business classes from the specific schema of the data (could be XSD schema, Database schema, etc.). I have tried two approaches, neither of which proved entirely satisfactory :

1. XML, xsd.exe or equivalent tool (like Xsd2Code) to generate the business classes and the mapping. For large data, size of the XML store becomes an issue.

2. Database (SQL Server CE), leveraging the Entity Framework to create the business classes and the mapping : during the first tests, for increasing database size, EF was in serious trouble (performance problems and out of memory exceptions).

What I want to achieve (loading large business data into memory in one go) seems "standard" (or is it really ?), yet I can't find an efficient and clean way to do it.

Answer 1

Take a look at XML Serialization in the .NET framework. I'm using this with a data interchange project and .NET supports serializing and deserializing the information into and from objects with very little code. It may be the path you want to take.

Edit: I see you've played with xsd.exe as well, you could use a db to store the XML in a column in a table for easy retrieval. We're actually doing this as well in the current project. We're receiving some XML from a 3rd party source, so we store it in a table, validate it against the schema, then read it out into .NET objects generated from a .xsd file using the xsd.exe approach. I'm not sure how large your data is, but this approach works well for us.

Answer 2

Your question is actually a bit vague, so I can only answer with my gut feeling. Storing data is for databases. You are talking about business objects, so using an OR/M tool is a logical pick. Since you're in .NET, Entity Framework is a logical pick.

It doesn’t seem to me that Entity Framework should cause any problems around memory that other solutions won't give you. I think your OOM problems are caused by improper use of the tool or perhaps other problems. If you're keep having problems with it, please ask a specific question about this problem, here at SO.

Good luck.

Answer 3

If you have XML and SQLDatabases, you can try LINQ. Use SQLMetal program to automatically generate Classes that correspond to your SQLDatabase tables. If you have relationships in you DB they are also preserved in object model generated by SQLMetal.