Fast file copy with progress

Question

Hello,

I'm writing an SDL application for Linux, that works from the console (no X server). One function I have is a file copy mechanism, that copies specific files from HDD to USB Flash device, and showing progress of this copy in the UI. To do this, I'm using simple while loop and copying file by 8kB chunks to get copy progress. The problem is, that it's slow. I get to copy a 100 MB file in nearly 10 minutes, which is unacceptable.

How can I implement faster file copy? I was thinking about some asynchronous API that would read file from HDD to a buffer and store the data to USB in separate thread, but I don't know if I should implement this myself, because it doesn't look like an easy task. Maybe you know some C++ API/library that can that for me? Or maybe some other, better method?

Answer 1

Let the OS do all the work:

1. Map the file to memory: mmap, will drastically speed up the reading process.
2. Save it to a file using msync.

Answer 2

Don't synchronously update your UI with the copy progress, that will slow things down considerably. You should run the file copy on a separate thread from the main UI thread so that the file copy can proceed as fast as possible without impeding the responsiveness of your application. Then, the UI can update itself at the natural rate (e.g. at the refresh rate of your monitor).

You should also use a larger buffer size than 8 KB. Experiment around, but I think you'll get faster results with larger buffer sizes (e.g. in the 64-128 KB range).

So, it might look something like this:

#define BUFSIZE (64*1024)

volatile off_t progress, max_progress;

void *thread_proc(void *arg)
{
    // Error checking omitted for expository purposes
    char buffer[BUFSIZE];
    int in = open("source_file", O_RDONLY);
    int out = open("destination_file", O_WRONLY | O_CREAT | O_TRUNC);

    // Get the input file size
    struct stat st;
    fstat(in, &st);

    progress = 0;
    max_progress = st.st_size;

    ssize_t bytes_read;
    while((bytes_read = read(in, buffer, BUFSIZE)) > 0)
    {
        write(out, buffer, BUFSIZE
        progress += bytes_read;
    }

    // copy is done, or an error occurred
    close(in);
    close(out);

    return 0;
}

void start_file_copy()
{
    pthread_t t;
    pthread_create(&t, NULL, &thread_proc, 0);
}

// In your UI thread's repaint handler, use the values of progress and
// max_progress

Note that if you are sending a file to a socket instead of another file, you should instead use the sendfile(2) system call, which copies the file directly in kernel space without round tripping into user space. Of course, if you do that, you can't get any progress information, so that may not always be ideal.

For Windows systems, you should use CopyFileEx, which is both efficient and provides you a progress callback routine.