Faster way to create tab deliminated text files?

Question

Hello everyone,

Many of my programs output huge volumes of data for me to review on Excel. The best way to view all these files is to use a tab deliminated text format. Currently i use this chunk of code to get it done:

ofstream output (fileName.c_str());
for (int j = 0; j < dim; j++)
{
    for (int i = 0; i < dim; i++)
        output << arrayPointer[j * dim + i] << " ";
    output << endl;
}

This seems to be a very slow operation, is a more efficient way of outputting text files like this to the hard drive?

Update:

Taking the two suggestions into mind, the new code is this:

ofstream output (fileName.c_str());
for (int j = 0; j < dim; j++)
{
    for (int i = 0; i < dim; i++)
        output << arrayPointer[j * dim + i] << "\t";
    output << "\n";
}
output.close();

writes to HD at 500KB/s

But this writes to HD at 50MB/s

{
    output.open(fileName.c_str(), std::ios::binary | std::ios::out);
    output.write(reinterpret_cast<char*>(arrayPointer), std::streamsize(dim * dim * sizeof(double)));
    output.close();
}

Answer 1

Don't use endl. It will be flushing the stream buffers, which is potentially very inefficient. Instead:

output << '\n';

Answer 2

ofstream output (fileName.c_str());
for (int j = 0; j < dim; j++)
{
    for (int i = 0; i < dim; i++)
        output << arrayPointer[j * dim + i] << '\t';
    output << endl;
}

Use '\t' instead of " "

Answer 3

does it have to be written in C? if not, there are many tools already written in C, eg (g)awk (can be used in unix/windows) that does the job of file parsing really well, also on big files.

awk '{$1=$1}1' OFS="\t" file

Answer 4

It may be faster to do it this way:

ofstream output (fileName.c_str());
for (int j = 0; j < dim; j++)
{
    for (int i = 0; i < dim; i++)
        output << arrayPointer[j * dim + i] << '\t';
    output << '\n';
}

Answer 5

Use C IO, it's a lot faster than C++ IO. I've heard of people in programming contests timing out purely because they used C++ IO and not C IO.

#include <cstdio>

FILE* fout = fopen(fileName.c_str(), "w");

for (int j = 0; j < dim; j++) 
{ 
    for (int i = 0; i < dim; i++) 
        fprintf(fout, "%d\t, arrayPointer[j * dim + i]); 
    fprintf(fout, "\n");
} 
fclose(fout);

Just change %d to be the correct type.