Reading binary files, Linux Buffer Cache

Question

I am busy writing something to test the read speeds for disk IO on Linux.

At the moment I have something like this to read the files:

Edited to change code to this:

  const int segsize = 1048576;
  char buffer[segsize];
  ifstream file;
  file.open(sFile.c_str());
  while(file.readsome(buffer,segsize)) {}

For foo.dat, which is 150GB, the first time I read it in, it takes around 2 minutes. However if I run it within 60 seconds of the first run, it will then take around 3 seconds to run. How is that possible? Surely the only place that could be read from that fast is the buffer cache in RAM, and the file is too big to fit in RAM.

The machine has 50GB of ram, and the drive is a NFS mount with all the default settings. Please let me know where I could look to confirm that this file is actually being read at this speed? Is my code wrong? It appears to take a correct amount of time the first time the file is read.

Edited to Add: Found out that my files were only reading up to a random point. I've managed to fix this by changing segsize down to 1024 from 1048576. I have no idea why changing this allows the ifstream to read the whole file instead of stopping at a random point.

Thanks for the answers.

Answer 1

in_avail returns the lower bound of how much is available to read in the streams read buffer, not the size of the file. To read the whole file via the stream, just keep calling the stream's readsome() method and checking how much was read with the gcount() method - when that returns zero, you have read everthing.

Answer 2

• in_avail doesn't give the length of the file, but a lower bound of what is available (especially if the buffer has already been used, it return the size available in the buffer). Its goal is to know what can be read without blocking.

• unsigned int is most probably unable to hold a length of more than 4GB, so what is read can very well be in the cache.

• C++0x Stream Positioning may be interesting to you if you are using large files

Answer 3

On Linux, you can do this for a quick troughput test:

$ dd if=/dev/md0 of=/dev/null bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.863904 s, 243 MB/s

$ dd if=/dev/md0 of=/dev/null bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.0748273 s, 2.8 GB/s

$ sync && echo 3 > /proc/sys/vm/drop_caches

$ dd if=/dev/md0 of=/dev/null bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 0.919688 s, 228 MB/s

echo 3 > /proc/sys/vm/drop_caches will flush the cache properly

Answer 4

It appears to take a correct amount of time the first time the file is read.

On that first read, you're reading 150GB in about 2 minutes. That works out to about 10 gigabits per second. Is that what you're expecting (based on the network to your NFS mount)?

Answer 5

One possibility is that the file could be at least in part sparse. A sparse file has regions that are truly empty - they don't even have disk space allocated to them. These sparse regions also don't consume much cache space, and so reading the sparse regions will essentially only require time to zero out the userspace pages they're being read into.

You can check with ls -lsh. The first column will be the on-disk size - if it's less than the file size, the file is indeed sparse. To de-sparse the file, just write to every page of it.

If you would like to test for true disk speeds, one option would be to use the O_DIRECT flag to open(2) to bypass the cache. Note that all IO using O_DIRECT must be page-aligned, and some filesystems do not support it (in particular, it won't work over NFS). Also, it's a bad idea for anything other than benchmarking. See some of Linus's rants in this thread.

Finally, to drop all caches on a linux system for testing, you can do:

echo 3 > /proc/sys/vm/drop_caches

If you do this on both client and server, you will force the file out of memory. Of course, this will have a negative performance impact on anything else running at the time.