Memory-Mapped Files & Transparent Persistence of Java Objects

Question

Greeting All,

I want to achieve transparent persistence of Java objects through memory-mapped files (utilize the OS paging/swapping mechanism).

My problem is: how can I move a Java object to my memory-mapped block ? Plus, how can I force a new object instance to reside in such blocks ?

As you all know, a memory-mapped block can be seen as a byte array, and what I am really asking here is how to overlap the address space of Java objects with the one of such arrays ? So that we can still manipulate the data via objects while OS handles persistence transparently (writes dirty pages).

If Java does not allow me for this, what cross-platform & garbage-collecting OO language would you advise me to use ?

Thank you all in advance.

Answer 1

This is afaik not possible. And there are several good reasons why the runtime doesn't allow this.

• The memory layout of your object is part of the JVM internals. Each JVM can make it own decisions how to layout objects. This layout can change over releases. When you memory-map a object to a file, you would run into trouble. What happens when the object-layout has changed? Crash, update the file, do magic? In the end memory-mapping objects to files would require to specify a fixed object-layout.
• What happens if you add/remove a field? Then the object-layout has changed for sure. You cannot memory-map a file with the old memory-layout.
• A object has also additional information, like the VTable for virtual function-calls, fields for lock-states etc. Do you want the map them also? Probably not, because thats internal runtime-state. So you would require to alway ignore a few bytes or split-up objects. That would make the memory-layout extremely complex.
• The garbage collector compacts the memory to minimize fragmentation by moving objects around. Therefore there needs to be a 'fixation'-mechanism for objects. (that's possible in .NET, for interop-reasons)
• The garbage collector works in generations. A object if first allocated on a young generation. As it survives it is moved to other generation. A memory-mapped object would require an exception.

For all these reasons you cannot memory-map Java/.NET-objects. Supporting such a feature would make the JVM/CLR extremely complex.

The JVM/CLR still give you access to Memory-Mapped files as an array-like abstraction, where you can write/read bytes. On top of this you can implement your persistence-mechanism. From simple serialisation to complex databases. There are object-databases which come quite close to provide transparent persistence. Then objects behave like persistent datastructures / memory-mapped objects.

Answer 2

You can't.

Java, by design, enforces type safety and referential integrity. That is, if I have a field of reference type T, I know it points to a instance of T. (Barring heap-pollution introduced by type erasure). This is in contrast to "unsafe" languages such a C++ where it is quite possible for a reference to point to an invalid location, where that reference can cause "memory corruption".

If Java allowed to treat a byte[] as Object, it could not guarantee such referential integrity.

Answer 3

The only way you can do this is by using your own Java VM and adding this functionality. Of course the code that you would write would not be in Java but in the language the VM is implemented in. Quite some time ago Gemstone used this approach for their object database engine.

Todays object databases (I work on one.) don't do things this way. It's much more straightforward to enhance bytecode to track field access and to use reflection or injected methods to turn objects into some kind of serialized form. This performs quite good. If you want to support querying, you have to work with individual field values anyway to index them.

For us it would just not be doable to maintain a VM for all the platforms that we want to run on. We also couldn't possibly convince serious customers to rely their entire (banking) application on a VM that we customize.

If you are seriously interested in producing a solution based on a Java VM: There used to be an interesting Java research project for orthogonal transparent persistence called "Forest". You may be able to find old papers or even source code.

If you are looking for a different language to take "objects" from memory directly: C++ will let you do that. There are some old object databases written in C++ that use this approach. ...but hey, that's crazy stuff, using page faults to load objects. These kind of object databases produced the bad image. I hope we will turn it around again soon.