I want to copy a large a ram-based file (located at /dev/shm direcotry) to local disk, is there some way for an efficient copy instead of read char one by one or create another piece memory? I can use only C language here. Is there anyway that I can put the memory file directly to disk? Thanks!
A:
/dev/shm is shared memory, so one way to copy it would be to open it as shared memory, but frankly I don't think you will gain anything.
when writing your memory file to disk, the bottleneck will be the disk. just be sure to write data in big chunks, and you should be fine.
Omry
2010-04-03 19:20:10
A:
You can just copy it like any other file:
cp /dev/shm/tmp ~/tmp
So, a quick, simple way is to issue a cp command via system().
Justin Ethier
2010-04-03 19:22:23
-1 because this is a low performance solution and it utilizes an external program (which causes various problems in itself) via system() (which runs the command using a shell, causing various other problems). Implementing an identical copy operation in C takes only a few lines of code.
Tronic
2010-04-03 20:27:20
Agree with Tronic.
Siu Ching Pong - Asuka Kenji
2010-04-04 02:48:08
@Tronic - Fair enough; care to post some code?
Justin Ethier
2010-04-04 03:42:18
FILE* input = fopen("foo", "rb"); FILE* output = fopen("bar", "wb"); for (int ch; (ch = fgetc(input)) != EOF; fputc(ch, output)) {} fclose(output); fclose(input); // A simple implementation, far from perfect, but something to begin with.
Tronic
2010-04-04 07:19:13
I doubt this offering good performance either. It still does the copying on CPU, and in benchmarks MMAP tends to be slower for linear access than other methods.
Tronic
2010-04-03 20:29:02
This doesn't force the CPU to touch the memory. If the data is already set and just needs to be written, I think this is a very direct way of accomplishing the task.
Potatoswatter
2010-04-05 00:39:16
`memcpy()` will certainly involve the CPU touching the memory. It has no idea that the destination is a memory mapped file.
caf
2010-04-06 09:52:36
A:
You could try to see if the splice system call works for this. I'm not sure if it will since it has some restrictions about the types of files that it can work with, but if it did work you would call it repeatedly with memory page sized (or some multiple page size) requests repeatedly until it finished, and the kernel would handle it very efficently.
If this doesn't work you'll need to do either mmap or do plain old read/write.
Reading and Writing in memory page sized chunks makes things much more efficient. It can be even more efficient if your buffers are memory page size aligned since it opens up the oppurtunity for the kernel to just move the data to/from your process's memory via memory managment trickery rather than actually copying the data around.
nategoose
2010-04-04 04:27:29
A:
The only thing you can do is read() in page size aligned chunks. I'm assuming you need to guarantee the data as written, which is going to mean bypassing buffers via posix_fadvise() or using O_DIRECT (I typically use posix_fadvise(), but O_DIRECT is appropriate here).
In this case, the speed of the media being written to alone dictates how quickly this will happen.
If you don't need to bypass buffers, the operation will complete faster, but there's no guarantee that the data will actually be written in the event of a reboot / power outage / etc. Since the source of the data is in shared memory, I'm (again) guessing you want the write to be guaranteed.
The only thing you can optimize is how long it takes read() to get data from shared memory into your own address space, which page size aligned chunks will improve.
Tim Post
2010-04-04 04:40:52
+1
A:
Thanks you guys for the help! I made it by mmap the ram-based file and write the entire block directly to the destination. memcopy was not used because I am actually writing to a parallel file system (pvfs), which does not support mmap operation.
Hui Jin
2010-04-05 00:03:56