C++ program runs slow in VS2008

Question

I have a program written in C++, that opens a binary file(test.bin), reads it object by object, and puts each object into a new file (it opens the new file, writes into it(append), and closes it). I use fopen/fclose, fread and fwrite. test.bin contains 20,000 objects.

This program runs under linux with g++ in 1 sec but in VS2008 in debug/release mode in 1min!

There are reasons why I don't do them in batches or don't keep them in memory or any other kind of optimizations.

I just wonder why it is that much slow under windows.

Thanks,

Answer 1

I believe that when you close a file in Windows, it flushes the contents to disk each time. In Linux, I don't think that is the case. The flush on each operation would be very expensive.

Answer 2

I think it is not matter of VS 2008. It is matter of Linux and Windows file system differences. And how C++ works with files in both systems.

Answer 3

So, what do you suggest? Let's say I have a main file and 100000 other files, and I have to read the main file and distribute data into each of these 100000 files. The whole data is so big and I can not store it on the main memory. Also, it is not possible to have all the files open because of max limit of open handles associated with a process. Right now I open and close any of them that I need at a time and keep the main file open...

Answer 4

Unfortunately file access on Windows isn't renowned for its brilliant speed, particularly if you're opening lots of files and only reading and writing small amounts of data. For better results, the (not particularly helpful) solution would be to read large amounts of data from a small number of files. (Or switch to Linux entirely for this program?!)

Other random suggestions to try:

• turn off the virus checker if you have one (I've got Kaspersky on my PC, and writing 20,000 files quickly drove it bananas)
• use an NTFS disk if you have one (FAT32 will be even worse)
• make sure you're not accidentally using text mode with fopen (easily done)
• use setvbuf to increase the buffer size for each FILE
• try CreateFile/ReadFile/etc. instead of fopen and friends, which won't solve your problem but may shave a few seconds off the running time (since the stdio functions do a bit of extra work that you probably don't need)

Answer 5

I'm seeing a lot of guessing here.

You're running under VS2008 IDE. You can always use the "poor man's profiler" and find out exactly what's going on.

In that minute, hit the "pause" button and look at what it's doing, including the call stack. Do this several times. Every single pause is almost certain (Prob = 59/60) to catch it doing precisely what it doesn't do under Linux.

Answer 6

Thanks Everybody, especially BRONE :) I got it solved! It was the damn McAfee anti-virus!!! I turned it off and now it works perfect.