How to perform fast formatted input from a stream in C++?

Question

The situation is: there is a file with 14 294 508 unsigned integers and 13 994 397 floating-point numbers (need to read doubles). Total file size is ~250 MB.

Using std::istream takes ~30sec. Reading the data from file to memory (just copying bytes, without formatted input) is much faster. Is there any way to improve reading speed without changing file format?

Answer 1

Do you need to use STL style i/o? You must check out this excellent piece of work from one of the experts. It's a specialized iostream by Dietmar Kuhl.

I hate to suggest this but take a look at the C formatted i/o routines. Also, are you reading in the whole file in one go?

Answer 2

You might also want to look at Matthew Wilson's FastFormat library:

• http://www.fastformat.org/

I haven't used it, but he makes some pretty impressive claims and I've found a lot of his other work to be worth studying and using (and stealing on occasion).

Answer 3

You haven't specified the format. It's possible that you could memory map it, or could read in very large chunks and process in a batch algorithm.

Also, you haven't said whether you know for sure that the file and process that will read it will be on the same platform. If a big-endian process writes it and a little-endian process reads it, or vice versa, it won;t work.

Answer 4

Parsing input by yourself (atoi & atof), usually boosts speed at least twice, compared to "universal" read methods.

Answer 5

Something quick and dirty is to just dump the file into a standard C++ string, and then use a stringstream on it:

#include <sstream>
// Load file into string file_string
std::stringstream s( file_string );
int x; float y;
s >> x >> y;

This may not give you much of a performance improvement (you will get a larger speed-up by avoiding iostreams), but it's very easy to try, and it may be faster enough.