SD card write performance

Question

I am writing a little application, which is writing jpeg images at a constant rate on a SD card. I choose an EXT3 filesystem, but the same behaviour was observed with an EXT2 filesystem.

My writing loop looks like this :

get_image()
fwrite()
fsync()

Or like this :

get_image()
fopen() 
fwrite()
fsync()
fclose()

I also display some timing statistics, and I can see my program is sometime blocked for several seconds. The average rate is still good, because if I keep the incoming images into a fifo, then I will write many image in a short period of time after such a stall. Do you know if it is a problem with the OS or if it is related to the SD card itself ? How could I move closer to realtime ? I don't need strong realtime, but being stalled for several seconds is not acceptable.

Some precision : Yes it is necessary to fsync after every file, because I want the image to be on disk, not in some user or kernel buffer. Without fsyncing, I have much better throughoutput, but still unacceptable stall. I don't think it is a buffer problem, since the first stall happens after 50 Mbytes have been written. And according to the man page, fsync is here precisely to ensure there is no data buffered.

Precision regarding the average write rate : I am writing at a rate that is sustainable by the card I am using. If I pile incoming image while waiting for an fsync to complete, then after this stall the write transfer rate will increase and I will quickly go back to the average rate. The average transfer rate is around 1.4 MBytes /s.

The systeme is a modern laptop running ubuntu 8.04 with stock kee (2.6.24.19)

Answer 1

I'm not very knowledgeable in this area, but the symptoms you describe sound an awful lot like filling up a buffer. You may be filling a buffer in the file writer or in the I/O device communicating with the SD card itself. You then have to wait until it actually writes the data to the card (thus emptying the buffer) before you can write more. SD cards are not particularly fast writers. If you can find a way to check if data is actually being written to the card during these pauses, that would verify my theory. Some card readers have an LED that blinks when data is being accessed -- that would probably be a good indicator.

Just a hunch... take it with some salt :)

Answer 2

Is it necessary to fsync() after every file? You may have better luck letting the OS decide when a good time is to write out all enqueued images to the SD card (amortizing the startup cost of manipulating the SD card filesystem over many images, rather than incurring it for every image).

Can you provide some more details about your platform? Slow I/O times may have to do with other processes on the system, a slow I/O controller, etc..

You might also consider using a filesystem more suited to how flash memory works. FAT32 is more common than extN, but a filesystem specifically built for SD may be in order as well. JFFS is a good example of this. You will probably get better performance with a filesystem designed for flash (as opposed to spinning magnetic media), and you get better wear-leveling (and thus device lifetime/reliability) properties as well.

Answer 3

AFAIK some flash disks have really bad write performance (esp. cheap brands). So if you measure the write speed of your application (including the time required for fsync), what do you get? It might easily be in the order of very few megabytes per second - just because the hardware doesn't do better.

Also, apparently writing can be much slower if you write many small blocks instead of one big block (the flash disk might only get about 10 writes per second done, in bad cases). This is probably something that can be mitigated by the kernel buffer, so using fsync frequently might slow down the writing...

Btw. did you measure write performance on FAT32? I would guess it is about the same, but if not, maybe there's some optimization still available?

Answer 4

May be this will help - Benchmarking Filesystems:

...I was quite surprised how slow ext3 was overall, as many distributions use this file system as their default file system...

And "ext3 fsync batching":

...This patch measures the time it takes to commit a transaction to the disk, and sleeps based on the speed of the underlying disk.

Answer 5

Might also consider the SD Card, is it NOR or NAND? This page shows an order of magnitude between sd cards (2M/s vs 20M/s).
http://www.robgalbraith.com/bins/camera_multi_page.asp?cid=6007-9597
I think ZFS is optimized for flash memory.

Answer 6

I have same problem. Tried several sd card brands. No matter if i am writing on windows or ubuntu, 2mb/s was the top, on my best sd card. Reading, in other hand, is pretty fast. Looking for a solution too.

Answer 7

Try to open the file with O_DIRECT and do the caching in application level.

We met the similar issue when we were implementing a PVR (Personal Video Record) feature in STB Box. The O_DIRECT trick satisfied our need finaly.(*)

Without O_DIRECT. The data of wite() will firstly be cached in the kernel buffer and then be flushed to the media when you call fsync or when the kernel cache buffer is full.(**).

With O_DIRECT.Th kernel will do DMA directly to the physical memory pointed by the userspace buffer passed as parameter to the write syscalls. So there will be no CPU and mem bandwidth spent in the copies between userspace memory and kernel cache, and there will be no CPU time spent in kernel in the management of the cache (like cache lookups, per-page locks etc..).( copied from here )

Not sure it can also solve your problem, but you might want to have a try.

*Despite of Linus's critize of O_DIRECT ,it did solve our problems.

** suppose you did not open the file with O_DSYNC or O_SYNC