casting from binary to multi-dimensional array on heap

Question

i'm currently using binary files for quick access to custom objects stored in multidimensional arrays which are saved to file as object arrays. so far, reading from file hasn't been too much of a problem since i've been reading the array into an identical object array on the stack. it looks something like this:

Foo foo[5][10];
ifstream infile("foofile.dat",ios::in|ios::binary);
infile.read((char*)&foo, sizeof Foo);

the problem i'm currently running into is that i'm increasingly storing larger amounts of data, and am overflowing my stack when creating the local object array. the solution then, would seem to create the local object array on the heap, and i'm not sure how to format it so that i can cast it directly from the char*. here's how i would allocate on the heap:

Foo (*foo)[1000] = new Foo [500][1000];

however, this doesn't seem to be the proper format if i want to cast to the whole array. for instance, if i try to do the following, it doesn't work:

ifstream infile("foofile.dat",ios::in|ios::binary);
infile.read((char*)(*foo), (sizeof Foo * 500 * 1000));  //or below
infile.read((char*)(&foo), (sizeof Foo * 500 * 1000));  //or below
infile.read((char*)(foo), (sizeof Foo * 500 * 1000));   //or below

essentially, i have no idea how to structure this using an array i've allocated on the heap. can anyone give any pointers (heh)?

thanks so much, jim

Answer 1

I don't think the solution you're using will function in the general case; modern operating systems make use of heap randomization, so it's not always possible to claim the same virtual address space across multiple runs of a program.

If any of your stored objects contain pointers, they will dangle when you load into a different virtual topology.

Have you tried serializing your objects into a proper on-disk format instead of just writing a binary blob?

Answer 2

It's been so long since I wrote unmanaged code that I can't even remember the syntax now. There ws a way to allocate a contiguous block of memory and be able to cast to a multidimensional array. If you allocate all you need in one go, then make each element point to its relevant location in that block. (Apologies for broken syntax/code).

void* ptr = malloc (void*)(sizeof(Foo) * 500 * 1000);
Foo** foo = (Foo**)ptr;
for (int i=0;i<500;i++) {
    foo[i] = (Foo*)(ptr+500*i*sizeof(Foo))
}

Answer 3

To my opinion here it makes not much sense to use C++ mechanisms to allocate / deallocate, since you overwrite the data completely by your reads. So don't do new / delete stuff. Your cast (char*) is in fact a reinterpret_cast< char* >. Why not just use a malloc with the correct size?

typedef Foo largeFoo[500][1000];
largeFoo* data = reinterpret_cast< largeFoo* >(malloc(sizeof largeFoo));

Then do your read into that (again with a reinterpret_cast). For convenience you could then alias your pointer to a reference.

largeFoo& foo = *data;

so all of your remaining code could remain the same as in the stack version, with only a free of your pointer at the end.