Question about fwrite API

views:

134

answers:

+3 Q:

Question about fwrite API

Hi,

In C++, there is a fwrite() which writes a buffer to a file on disk:

http://www.cplusplus.com/reference/clibrary/cstdio/fwrite/

Can you please tell me if there is any buffer inside that fwrite implementation?

i.e. if I call fwrite() multiple times (say 10 times), does it actually invoke file i/o 10 different times? I am asking this for Ubuntu 10.04 environment.

Thank you.

+1 A:

Yes, it is buffered. The size of the buffer is defined by BUFSIZ. (Thanks @ladenedge.) Ten sufficiently large operations will result in ten system calls.

You are also allowed to set your own buffer using setbuf and setvbuf.

The functions in cstdio are inherited from C. The preferred interface in C++ is fstream and filebuf.

Potatoswatter 2010-05-10 20:17:59

I believe the default buffer size is `BUFSIZ`, as defined in stdio.h, though of course callers shouldn't need that information. `setbuf` may be used to control the size on some systems.

ladenedge 2010-05-10 20:22:04

Lol, you posted that literally at the exact second I made an edit: "ladenedge 1 sec ago"

Potatoswatter 2010-05-10 20:22:41

+4 A:

It depends. On most platforms, file IO is cached hence the fflush() call exists to write cached data to disk - in that respect, the answer to your first question is (usually) "yes", and your second "no". That said, it's not guaranteed to be that way by any stretch of the imagination for any particular platform - generally, the standards only specify the interface for such functions, not their implementation. Also, its quite possible that calling fwrite() will cause an implicit flush if the cache becomes "full", in which case calling fwrite() may in fact trigger file IO - especially if calling fwrite() with large amounts of data.

Mac 2010-05-10 20:19:44

The library IO functions are buffered, but also the system IO functions usually are; not only that, the disk controller too has a cache. Actually, usually the problem with disks is not getting sure that your output gets cached, but getting sure it's *really* written to disk when you need it. :)

Matteo Italia 2010-05-10 20:43:44

@Matteo, I know what you mean - for example, using fwrite to write an error log can be interesting if your program crashes *before* the caches are written to disk. Luckily there's fflush for that...

Mac 2010-05-10 20:47:50

"standards only specify the interface for such functions, not their implementation" is not accurate. While the standard does not specify implementation, it does specify behavior. And in this case, the standard does specify that FILE* is buffered (C++ incorporates the C standard in 17.3.1.4/1 and the C standard define that FILE* is buffered in 7.19.5.6/2).

R Samuel Klatchko 2010-05-10 21:00:57

Worst case, assume 1:1 write operations with `fwrite()`. Profile your code to see if there is only 1 file write operation. There is generally a difference in execution speed between 1 file write operation and 10. I prefer to perform buffering in my code rather than rely on the OS or Run-Time Library.

Thomas Matthews 2010-05-10 21:03:15

Profile your code to see what the bottleneck is, rather than worrying about what you **think** it is.

WhirlWind 2010-05-10 22:02:47

@R Samuel Klatchko: while I might concede that the sentence you highlight could be a little more accurate, I still stand by it - note the use of "generally" preceding the portion you quoted. One *might* also consider behaviour of an API part of the interface, in that a programmer *may* need to know about it to successfully use said API. That said, thanks for taking the time to find out what the standard actually *does* say - I've certainly learnt something I didn't know before.

Mac 2010-05-10 22:24:50

@Mac - okay, I could go with the definition that behavior is part of the interface and not the implementation. But your answer is still incorrect in that it says buffering is not guaranteed even though it is required behavior.

R Samuel Klatchko 2010-05-11 00:12:42

@R Samuel Klatchko: Sure, on any *standards-compliant* platform you'll be guaranteed buffering. That wont stop a non-compliant platform from not buffering output though. I don't know of any such platforms offhand, but I can easily imagine (say) a microcontroller-based platform using immediate file IO rather than buffered IO due to memory constraints. You're absolutely right - that would not be standard behaviour, but that doesn't mean it can't happen.

Mac 2010-05-11 02:07:14

It depends on the library implementation. You might take a look at something like dtruss or strace to see what system calls are actually invoked by the implementation.

Why do you care about this?

WhirlWind 2010-05-10 20:20:17

My program is doing a lot of fwrite(). And it is running on Ubuntu 10.04. I would like to know if I need to buffering up the IO myself before calling fwrite() for performance reason.

michael 2010-05-10 20:34:19

@michael: Yes, buffer up the I/O. Reducing the number of calls to `fwrite` will at the minimum reduce the number of branches in your code. Branches tend to slow down performance as many processors reload their instruction cache after a branch. Also, less code to execute makes a program faster.

Thomas Matthews 2010-05-10 21:05:18

Why not just profile your code to find bottlenecks?

WhirlWind 2010-05-10 22:02:00

I don't believe the standards says anything about this but in general: no the operating system will decide how many I/O operations to do. It could be 1 it could be 10. It could even be more if the data size is large. Most operating systems allow you to control how I/o should behave but AFAIK there's no portable way to do it. (and often you don't want to anyway)

Isak Savo 2010-05-10 20:20:26

FILE* I/O can be buffered but it does not have to be (it can be different for each stream). Furthermore, there are multiple ways to do buffering (fully buffered where the buffer is not flushed until it is full or an explicit call to fflush is made or line buffered where the buffer is not flushed until it sees an EOL). It is also possible to turn off buffering altogether.

You can change the type of buffering with the setvbuf call.

R Samuel Klatchko 2010-05-10 20:48:16

ansaurus

tags:

views:

answers:

Question about fwrite API

related questions