Create custom driver for System.Data.Common

Question

Background:

Our C# application generates and executes queries to several types of databases (Oracle, SQL Server, MySQL), but a requirement came up to also apply them to a proprietary file format.

1. The namespace used until now is System.Data.Common.
2. The queries we need to apply are non-trivial (nested SELECTs, aliases in FROM, substring methods and string concatenations)

We initially converted the contents of the proprietary file to CSV, for which there exists the Driver {Microsoft Text Driver (*.txt; *.csv)}. However, the client required that no temp files are generated, and everything should happen in-memory.

Creating an ODBC driver to query the file directly seems too time-demanding. So, we are considering creating a driver (possibly ODBC), in which we will "embed" the SQLITE ODBC driver. Inside that driver, we can load the contents of the CSV files in the "in-memory" database, and then forward the query to the inner ODBC driver.

My questions:

1. Does this solution seem feasible?
2. Where should we start in order to create an ODBC driver from scratch?

Thanks

Answer 1

I think your solution is time consuming. You can find existing solutions for what you are trying to do. Here are few alternates.

Why not try Linq to text/csv file?

• http://www.codeproject.com/KB/linq/LINQtoCSV.aspx
• http://blogs.msdn.com/ericwhite/archive/2008/09/30/linq-to-text-and-linq-to-csv.aspx

Both solutions are in memory.

Another idea is that, you can export file in xml instead of csv or text (I always prefer export into xml when I have to process in my code). Than you use System.Xml or Linq to Xml to perform operations.

Answer 2

If you only are restricted not to create temporary files, you may try to use in-memory mode of SQLite. Here's a sample of connection string for it (source):

Data Source=:memory:;Version=3;New=True;

To my mind, this is simpler than building full-blown provider.$

Answer 3

If using SQLite as a backend is a possibility, I cannot really see why you could not use an ADO .NET provider to your SQLite database like System.Data.SQLite.

---

From your comment, I think you are in effect using the ODBC ADO .NET provider for all database connections (System.Data.Odbc). If you need to keep the same scheme, then a custom ODBC provider is the way to go (but then it is plain C native development, and rather painful I believe).

Another way to go would be to add a third parameter to your DB (the first two being the SQL and the connection string) : the ADO .NET provider to be used (like it is supposed to be done in configuration files, see the providerName attribute). This way you could use any ADO .NET provider available.

Then you could wrap the the SQLite provider into your own, custom, ADO .NET provider, that could include the generation and population of the SQLite DB. Advantage of this solution : pure managed .NET.

Answer 4

The client request is odd. There will always be files involved, you are reading from a file.

If they want their stuff in memory (again a weird customer) put the CSVs on to a RAM Disk: http://members.fortunecity.com/ramdisk/RAMDisk/ramdriv001.htm

Building and ODBC drive is a non-trival undertaking if you care about performance and stability.