Need some help writing my results to a file

Question

Hello guys,

My application continuously calculates strings and outputs them into a file. This is being run for almost an entire day. But writing to the file is slowing my application. Is there a way I can improve the speed ? Also I want to extend the application so that I can send the results to an another system after some particular amount of time.

Thanks & Regards,

Mousey

Answer 1

How if you write the results to a socket, instead of file. Another program Y, will read the socket, open a file, write on it and close it, and after the specified time will transfer the results to another system.
I mean the process of file handling is handled by other program. Original program X just sends the output to the socket. It does not concern it self with flushing the file stream.

Also I want to extend the application so that I can send the results to an another system after some particular amount of time.

If you just want to transfer the file to other system, then I think a simple script will be enough for that.

Answer 2

Use more than one file for the logging. Say, after your file reaches size of 1 MB, change its name to something contains the date and the time and start to write to a new one, named as the original file name. then you have:

results.txt

results2010-1-2-1-12-30.txt (January 2 2010, 1:12:30)

and so on.

Answer 3

You can buffer the result of different computations in memory and only write to the file when buffer is full. For example, your can design your application in such a way that, it computes result for 100 calculations and writes all those 100 results at once in a file. Then computes another 100 and so on.

Answer 4

There are several things that may or may not help you, depending on your scenario:

1. Consider using asynchronous I/O, for instance by using Boost.Asio. This way your application does not have to wait for expensive I/O-operations to finish. However, you will have to buffer your generated data in memory, so make sure there is enough available.
2. Consider buffering your strings to a certain size, and then write them to disk (or the network) in big batches. Few big writes are usually faster than many small ones.
3. If you want to make it really good C++, meaning STL-comliant, make your algorithm a template-function that takes and output-iterator as argument. This way you can easily have it write to files, the network, memory or the console by providing appropriate iterators.

Answer 5

Writing file is obviously slow, but you can buffered data and initiate the separate thread for writhing on file. This can improve speed of your application.

Secondly you can use ftp for transfer files to other system.

Answer 6

I think there are some red herrings here.

On an older computer system, I would recommend caching the strings and doing a small number of large writes instead of a large number of small writes. On modern systems, the default disk-caching is more than adequate and doing additional buffering is unlikely to help.

I presume that you aren't disabling caching or opening the file for every write.

It is possible that there is some issue with writing very large files, but that would not be my first guess.

How big is the output file when you finish?

What causes you to think that the file is the bottleneck? Do you have profiling data?

Is it possible that there is a memory leak?

Any code or statistics you can post would help in the diagnosis.