views:

527

answers:

5
+3  Q: 

Memory allocators

I want to make a virtual allocator using c++ on windows,, which allocate data on a file on the hard disk, to reduce physical memory usage when allocate large objects !.. I don't want to use system virtual memory with virtualAlloc.. . I want to create a file on the disk and use it to allocate the whole objects and then move the part or the object that I need to the RAM .

I tried to use Memory mapped file , but I faced some problems: I used the mapped file to allocate vector elements, but when I bake to delete any of them, the address of the element changed, also I can't find a method to map the object only when needed "in my test I mapped the whole file"!

Any resources or open source projects can help ???

A: 

From http://www.informit.com/guides/content.aspx?g=cplusplus&seqNum=213

The POSIX header includes memory mapping syscalls and data structures. Because this interface is more intuitive and simpler than that of Windows, I base my memory mapping example on the POSIX library.

The mmap() system call:

caddr_t mmap(caddress_t map_addr, 
       size_t length, 
       int protection, 
       int flags, 
       int fd, 
       off_t offset);

Let's examine what each parameter means.

In the following example, the program maps the first 4 KB of a file passed in command line into its memory and then reads int value from it:

#include <errno.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/types.h>

 int main(int argc, char *argv[])
 {
 int fd;
 void * pregion;
 if (fd= open(argv[1], O_RDONLY) <0)
 {
 perror("failed on open");
 return –1;
 }
 /*map first 4 kilobytes of fd*/
 pregion=mmap(NULL, 4096, PROT_READ,MAP_SHARED,fd,0);
 if (pregion==(caddr_t)-1)
 {
 perror("mmap failed")
 return –1;
 }
 close(fd); //close the physical file because we don't need it
 //access mapped memory; read the first int in the mapped file
 int val= *((int*) pregion);
}

To unmap a mapped region, use the munmap() function:

int munmap(caddr_t addr, int length);

addr is the address of the region being unmapped. length specifies how much of the memory should be unmapped (you may unmap a portion of a previously-mapped region). The following example unmaps the first kilobyte of the previously-mapped file. The remaining three KB still remain mapped to the process's RAM after this call:

munmap(pregion, 1024);
joe
I used CreateFileMapping ,,, and MapViewOfFile to map a part of the file ,, but I can't access unmapped object, and I return the address when allocate an object so when map the unmap the address change !so I want a method to map the object when needed!or even another way to allocate objects in the file !thank you;
Ahmad
+2  A: 

Google can help here. I implemented a custom STL allocator a number of years ago that used a shared memory store. The same techniques can be used to implement a disk-backed allocator. I would start by looking at this SourceForge project for inspiration.

D.Shawley
A: 

Sorry, but you fail to understand how (virtual) memory works. One the one hand you state that "I want to make "custom memory allocator" but without take a large space from the memory" but on the other hand you're surprised that "the address of the element changed".

This is pretty much to be expected. To make sure that the address of a (logical) object doesn't change, you have to keep the memory represented by that address committed to the object. If you free the memory, it becomes available for reuse, and so does the address. And if the address is reused, you can't page back the object to that address.

Ultimately, the problem here it that addresses and memory are very, very deeply connected. Recycling memory means recycling addresses.

MSalters
I am not surprised about the changing of the address, but I want a solution for this,, since when I allocate an object I returned the address value, so the program should use it for the rest of the code!so I want a method to do this in a proper way to re-access the object I allocated !
Ahmad
You're still not getting it. When you return an address, that is the address of the memory for the object. You cannot use the memory for something else, as you told the rest of the program that the memory at this address is where your new object is stored.
MSalters
what I meant is when I map the file in the first time and allocate the object it give me an address,, then unmap this part and map another ,, then in the second time the first part mapped to another address,, so I mapped the whole file once; but I don't need to do this !
Ahmad
+2  A: 

You may find inspiration from Boost.Interprocess, which provides support for memory mapped files, as well as allocators and containers over that memory.

More information about the allocator design can also be found at http://www.boost.org/doc/libs/1_37_0/doc/html/interprocess/architecture.html

Éric Malenfant
thank you,, I read about boost but not tried it yet,, I just tried to work with Mapping file ,,
Ahmad
A: 

Probably the best way to solve this is not to return regular pointers to large objects. Simply return small proxies. These proxy objects implement the full interface of the larger object. However, these proxy objects can deal with the raw data being either in RAM or on disk. The proxies implement a LRU mechanism amongst themselves to optimize RAM use. The caller never sees the address of these proxies change, nor does it get any pointers to raw data.

MSalters
Just to expand on this, the proxy could use an offset into the mmapped chunk of memory, and not pointers. You need the address of the start of the memory (which can change), and from that and the offset can access any of the data. I wouldn't try make an allocator for this, but a new container that supports STL iterators. Then you have access to the existing STL algorithm library, container adapters, etc.
KeithB