tags:

views:

172

answers:

6

When trying to copy a text file A to another file B, there may have several methods: 1) byte by byte 2) word by word 3) line by line

which one is more efficient?

+4  A: 

Just "buffer by buffer", copy files in binary mode and read/write X bytes long parts. I think that fastest solution is to just use copy function of C language itself or system call.

Largest buffer will provide you less HDD find for data operations (faster copying) but more RAM usage.

killer_PL
A: 

If done well, Byte by byte is more efficient. Of course that's not the whole story: it depends on how many bytes you copy at once. If you literally copy byte for byte you'll do an I/O call for every byte and end up being slower than the string libraries. Most people just guess at a good buffer size (generally 2048 or bigger, in multiples of 2) and use that.

wds
Can you explain what's the mechanism underlying the guessing at a good buffer size?
Depending on what kind of medium your text files are stored on, you will most likely either have 512-byte sectors (many traditional hard drives) or 2048-byte sectors (optical discs, many solid-state memory devices, some newer hard drives, etc). To minimize the work that your drive has to do, you want to copy in multiples of a sector. Therefore, you can copy 2048 bytes at a time, with operations aligned to 2048-byte boundaries (or, substitute any multiple of 2K for 2048).
bta
Generally, the best option is to use a multiple of the memory pagesize. In linux, I believe the default is 4K. The virtual memory manager is very good at optimizing the rest away for you, so reading/writing one page at a time can be quite fast. Using bigger buffer sizes might be better because of less syscalls needed (all syscalls have overhead) and perhaps hard disk buffering (there's no guarantee though that your file will be in consecutive sectors on the drive anyway). It depends.
wds
+7  A: 

Using buffers:

#include <fstream>

int main()
{
    std::ifstream    inFile("In.txt");
    std::ofstream    outFile("Out.txt");

    outFile << inFile.rdbuf();
} 

The C++ fstreams are buffered internally. They use an efficient buffer size (despite what people say about the efficiency of stream :-). So just copy one stream buffer to a stream and hey presto the internal magic will do an efficient copy of one stream to the other.

But learning to do it char by char using std::copy() is so much more fun.

Martin York
Excellent one. Thanks.
Iostreams' reputation for inefficiency comes from formatted I/O; for shovelling raw bytes around like this, it's about as good as anything could be.
Mike Seymour
rdbuf() not rdBuf()
wrang-wrang
A: 

I actually had to do the same thing myself once, so I timed it with various sizes. What I found was that, given a large file, the time taken was almost entirely a function of how many I/O's I performed (regardless of their size).

Thus your best bet is to do as few I/O's as possible. Preferably two (one to read and the other to write).

T.E.D.
A: 

If you do word-by-word or line-by line you can hardly reconstruct the original file since the are many forms of line breaks (\r, \n, \r\n) and spaces (\p, \f, 0x32) embedded in text files which you are risking to loose this way.

The most efficient way to copy files is to use byte-buffers. The larger the buffer, the more efficient the copying will be, as long as your buffer size isn't bigger than hard disk internal buffer size (today mostly ~8mb).

codymanix
A: 

Try using the C++ iostreams and the STL. Below is an example:

ifstream infile("to_copy.txt");
if (infile)
{
    istreambuf_iterator<char> ifit(infile);
    ofstream outfile("the_copy.txt");
    ostreambuf_iterator<char> ofit(outfile);
    if (outfile)
    {
        copy(ifit, istreambuf_iterator<char>(), ofit);
        outfile.close();
    }
    else
    {
        cerr << "Could not open output file" << "\n";
    }
    infile.close();
}
else
{
    cerr << "Could not open input file" << "\n";
}

Note: this might not be suitable in all situations. Use/tailor this depending on your specific requirements (e.g. ordinary or humongous files).

celavek