Determining cache misses for various filesystems

Question

Hello, I've got a project for school where I have to find out how many cache misses a filesystem will have under heavy and light loads and on a multiple processor machine. After discussing this with my professor, I came up with a basic plan of execution:

1. Create a program which will bog down the filesystem and fill up the buffer cache.
2. Use a system benchmarking tool to record the number of cache misses.
3. Rinse and repeat with a new conditions.

But being new to operating system design, I am unsure of how to proceed. So here are some points where I need some help:

1. What actions would an ideal program perform to fill up the buffer cache? Currently, the program that I've written reads and writes to several different files, x amount of times.
2. What tools are there that record the number of cache misses? I have looked into oprofile but I don't think it monitors the filesystem's buffer cache. But I have found this list which looks promising.
3. Will other running processes affect these benchmarks?

Thanks for your help!

Answer 1

1) If you are trying to test your filesystem performance, throw in several threads that are manipulating large amounts of file metadata alongside your I/O threads. Also, when doing I/O in several parallel threads, mix threads doing large-sized transfers and threads doing small-sized transfers. Many filesystems will coalesce small I/O operations together into larger requests that the physical drive can handle in a more time-efficient manner, and mixing I/O of various sized may help fill up the cache faster (since it has to buffer the coalesced I/O).

2) Be careful with that list of tools, many look like they are designed to operate on raw devices and not through the filesystem layer (so the results you'd get might not represent what you think they do). If you are looking for a tool to benchmark a particular filesystem, your best bet may be to check with the development team for that filesystem. They can most likely point you to the tool that they used to benchmark their FS during development, even if it is a custom tool developed internally.

3) Yes, anything else that is running and might access the filesystem under test can potentially impact your results. You may want to create a separate filesystem to use only for this test and turn off any background scans that might try to access it while you are running your tests.

Answer 2

That is an interesting question. May be I can give you a partial answer.

You should be aware that Linux has multiple caches related to file systems that may have different tools

• Inode cache
• Dentry cache
• Block cache

One way is to calculate (guess?) how much block level traffic your operations should generate, and then measure the real block operations (reads, writes, seeks) with blktrace.

I am not aware of any way to read the cache miss state of the inode and dentry cache. I would really like to be told that I am wrong here.

The hard way is to annotate the inode cache and dentry cache with own counters, but these caches are pretty hard kernel code.