What is the fastest way to write to a text file from a database table?

Question

Hi,

I have data analysis application and I need to be able to export database tables to a delimited text file using c#. Because of the application architecture, that data must be brought to the c# application. No database exporting functionality can be used. The tables size can range from a few columns and a few hundred rows to ~100 columns to over a million rows.

Further clarification based on comments --

I have a Windows Service acting as the data access layer that will be getting the request for the export from the presentation layer. Once the export is complete, the service will then need to pass the export back to the presentation layer, which would either be a WPF app or a Silverlight app, as a stream object. The user will then be given an option to save or open the export.

What is the fastest way to do this?

Thanks

Answer 1

for SQL Server: use BCP

http://www.simple-talk.com/sql/database-administration/creating-csv-files-using-bcp-and-stored-procedures/

Answer 2

If you are using SQL Server 2008 (or maybe 2005), you can right-click the database and choose "Tasks->Export Data". Choose your database as input, and choose the "Flat file destination" as output. Specify the file name, specify double-quote as the text qualifier, click "next" a few times and you're done. You can even save the task as an SSIS package that you can run again.

Doing it this way uses SSIS under the covers. It has very high performance, as it uses multiple threads in a pipeline.

Answer 3

hmm, first of all, if its not a must to use c#, the sql managment console is capable of such a task.

To achieve best perfrormance i would you a consumer-producer 2 thread concept,

• One thread will be the reader, responsible for reading items from the DB - in which case i highly recommand using the IReader to read the values, and put them in a cuncurrent queue.
• The other will be the writer who will simply use a fileStream to write the data from the queue.

you can also achieve much greater performance by reading the information via a paged manner, thats is, if you know you'll have 100000 records, devide it to chunks of 1000, have a reader reading those chunks from the DB and putting them in a queue.

Although the later solution is more complicated he'll allow you to utilize your CPU in the best way possibble and avoid latency.

Answer 4

If you really need to use C#, the fastest way would be to use ADO.NET's DataReader, it is read-only and forward-only, may suit you well. Just be careful with null fields, it doesn't handle very well, if you need to deal with them, maybe other ADO.NET resources will be more interesting for you.

Answer 5

If you need to just query the data quickly you can use 'Firehose' cursors in one or more threads and just read straight from the database.

Answer 6

var sqlConnection = new SqlConnection(ConfigurationManager.ConnectionStrings["connstr"].ToString());
var sqlDataAdapter = new SqlDataAdapter("select * from tnm_story_status", sqlConnection); sqlConnection.Open();
var dataSet = new DataSet();
sqlDataAdapter.Fill(dataSet); sqlConnection.Close();
var dataTable = dataSet.Tables[0];
var streamWriter = new StreamWriter(@"C:\db.txt", false);
var sb = new StringBuilder();
for (var col = 0; col < dataTable.Columns.Count; col++)
{
if (sb.ToString() != "") sb.Append(",");
sb.Append(dataTable.Columns[col].ColumnName);
}
streamWriter.WriteLine(sb.ToString());
sb.Remove(0, sb.ToString().Length);
for (var row = 0; row < dataTable.Rows.Count; row++ )
{
for (var col = 0; col < dataTable.Columns.Count; col++)
{
if (sb.ToString() != "") sb.Append(",");
sb.Append(dataTable.Rows[row][col].ToString());
}
streamWriter.WriteLine(sb.ToString());
sb.Remove(0, sb.ToString().Length);
}
streamWriter.Close();


Answer 7

I would look at using the SQLBulkCopy object.