Large file support in C++

Question

64bit file API is different on each platform.
in windows:

_fseeki64

in linux:

fseeko

in freebsd: some another sh.t

How to make it convinient and portable? Any examples?

Answer 1

My best suggestion would be to use a library for handling this which already exists.

A good candidate might be using the CPL file library from GDAL. This provides cross-platform large-file support for read+write, in ascii and binary access. Most of the GDAL file formats have been implemented using it, and are widely used.

Answer 2

STLport have native support for 64bits file size. On the other hand if you're doing intensive disk accesses, you might want to use file mapping : mmap on unix and CreateFileMapping on Windows.

Answer 3

Most POSIX-based platforms support the "_FILE_OFFSET_BITS" preprocessor symbol. Setting it to 64 will cause the off_t type to be 64 bits instead of 32, and file manipulation functions like lseek() will automatically support the 64 bit offset through some preprocessor magic. From a compile-time point of view adding 64 bit file offset support in this manner is fairly transparent, assuming you are correctly using the relevant typedefs. Naturally, your ABI will change if you're exposing interfaces that use the off_t type. Ideally you should define it on the command line, e.g.:

cxx -D_FILE_OFFSET_BITS=64

to make sure it applies to all OS headers included by your code.

Unfortunately, Windows doesn't support this preprocessor symbol so you'll either have to handle it yourself, or rely on a library that provides cross-platform large file support. ACE is one such library (both POSIX based and Windows platforms - just define _FILE_OFFSET_BITS=64 in both cases). I know that Boost.filesystem also supports large files on POSIX based platforms, but I'm not sure about Windows. Other cross-platform libraries likely provide similar support.

Answer 4

I wrote a library once called 'mfile', which stands for 'multi-file'. The problem was that I had a system that didn't support large files, so this library would either use the largefile support available on some systems, or handle being passed multiple files that it would virtually concat.

At any rate, it provided a 'FILE'-like interface, and worked fairly well. When I had to port to Windows, it was easy to put in the appropriate 64 bit support. On Unix, _FILE_OFFSET_BITS provide enough magic that I didn't have to play too many games for it to work.

Answer 5

Write a class that encapsulates file-handling, and use conditional compilation in the class to handle each platform.

Then just use the class instead of the native file-handling when you need to access a file from your program.

As far as I can tell, there's no set of universally portable file I/O functions when it comes large file handling.