Positioning an ifstream in very large files

Question

Hi,

I have to process very large log files (hundreds of Gigabytes) and in order to speed things up I want to split that processing on all the cores I have available. Using seekg and tellg I'm able to estimate the block sizes in relatively small files and position each thread on the beginning of these blocks but when they grow big the indexes overflow.

How can I position and index in very big files when using C++ ifstreams and Linux?

Best regards.

Answer 1

The easiest way would be to do the processing on a 64-bit OS, and write the code using a 64-bit compiler. This will (at least normally) give you a 64-bit type for file offsets, so the overflow no longer happens, and life is good.

Answer 2

You have two options:

1. Use 64-bit OS.
2. Use OS specific functions.