Will malloc implementations return free-ed memory back to the system? I have a long-living application with active memory allocation-deallocation. What behavior have ptmalloc 1-2-3, dlmalloc (glibc default), tcmalloc (google threaded malloc), solaris 10-11 default malloc and mtmalloc, FreeBSD 8 default malloc (jemalloc), Hoard malloc? ...
The following C code illustrates a problem I'm seeing on Linux 2.6.30.5-43.fc11.x86_64: #include <sys/types.h> #include <sys/stat.h> #include <sys/mman.h> #include <fcntl.h> #include <unistd.h> #include <stdlib.h> #include <stdio.h> #include <string.h> int main() { char buf[1024]; void *base; int fd; size_t pagesz = sys...
I havea question regarding mmap functionality. when mmap is used in asynchronous mode where the kernel takes care of persisting the data to the mapped file on the disk , is it possible to have the former updates overwrite the later updates ? Lets say at time T, we modify a location in memory that is memory mapped to a file on disk and a...
If i have memory mapped a file of size 10GB in a 1GB machine and if i trigger a file i/o, after making sure that the data requested is not in physical memory, will the fetched data get mapped to the corresponding virtual address in mmap? When i access the same location using mmap, will it again do an i/o (or will it make use of the data...
When using shared memory, each process may mmap the shared region into a different area of its respective address space. This means that when storing pointers within the shared region, you need to store them as offsets of the start of the shared region. Unfortunately, this complicates use of atomic instructions (e.g. if you're trying to ...
I'm writing a high-loaded daemon that should be run on the FreeBSD 8.0 and on Linux as well. The main purpose of daemon is to pass files that are requested by their identifier. Identifier is converted into local filename/file size via request to db. And then I use sequential mmap() calls to pass file blocks with send(). However sometime...
Does anyone know if Python's shelve module uses memory-mapped IO? Maybe that question is a bit misleading. I realize that shelve uses an underlying dbm-style module to do its dirty work. What are the chances that the underlying module uses mmap? I'm prototyping a datastore, and while I realize premature optimization is generally fro...
I have a mmap typecast to a char pointer char *ptr; ptr = (char *)mmap(0, FILESIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); This was my earlier code. But now I want to use a map instead of char * as the requirements changed. Now, my map is declared as map < int, string > i_s_map; How do I change my mmap call to point to the map?...
I am using mmap call to read from a very big file using simple pointer arithmetic in C++. The problem is that when I read small chunks of data (in the order of KBs) multiple times, each read take the same amount of time as the previous one. How can I know if the disk is being accessed to fulfill my request or whether the request is being...
I tried to run the following code from http://docs.python.org/library/mmap.html import mmap # write a simple example file with open("hello.txt", "wb") as f: f.write("Hello Python!\n") with open("hello.txt", "r+b") as f: # memory-map the file, size 0 means whole file map = mmap.mmap(f.fileno(), 0) # read content via st...
I use a file to communicate between Python and Ruby script. But, we have mmap. So here are my questions. Can I do the same thing (communicate between processes) with mmap? What advantage can mmap give us over physical file? Speedup? What would be the easiest way to communicate between two processes? What would be the fastest way to co...
I am confused by the specification of mmap. Let pa be the return address of mmap (the same as the specification) pa = mmap(addr, len, prot, flags, fildes, off); In my opinion after the function call succeed the following range is valid [ pa, pa+len ) My question is whether the range of the following is still valid? [ rou...
Hello, I am developing a active messaging protocol for parallel computation that replaces TCP/IP. My goal is to decrease the latency of a packet. Since the environment is a LAN, i can replace TCP/IP with simpler protocol to reduce the packet latency. I am not writing any device driver and i am just trying to replace the TCP/IP stack wit...
When using mmap() for shared memory (from Linux, or other UNIX-like systems) is it possible (and portable) to use fcntl() (or flock() or lockf() functions) to co-ordinate access to the mapping? Responses to this SO question seems to suggest that it should work. The idea I have in mind would be to have the shared memory structured with...
I want to be able to map memory to a file descriptor so I can use some existing functions that need a file descriptor. Here's essentially what I'm looking for: void do_operation1(int fd); char data[DATA_MAX] = { /* embedded binary data */ }; int fd = addr_to_fd(data, DATA_MAX); do_operation1(fd); /* ... operate on fd ... */ What sy...
I'm working some code including communication between processes, using semaphores. I made structure like this: typedef struct container { sem_t resource, mutex; int counter; } container; and use in that way (in main app and the same in subordinate processes) container *memory; shm_unlink("MYSHM"); //just in case fd = shm_ope...
I've been working with large sparse files on openSUSE 11.2 x86_64. When I try to mmap() a 1TB sparse file, it fails with ENOMEM. I would have thought that the 64 bit address space would be adequate to map in a terabyte, but it seems not. Experimenting further, a 1GB file works fine, but a 2GB file (and anything bigger) fails. I'm guessin...
I have written a parser class for a particular binary format (nfdump if anyone is interested) which uses java.nio's MappedByteBuffer to read through files of a few GB each. The binary format is just a series of headers and mostly fixed-size binary records, which are fed out to the called by calling nextRecord(), which pushes on the stat...
Hello, Some days ago, I decided that it would be fun to write a streambuf subclass that would use mmap and read-ahead. I looked at how my STL (SGI) implemented filebuf and realized that basic_filebuf contains a FILE*. So inheriting from basic_filebuf is out of the question. So I inherited from basic_streambuf. Then i wanted to bind my ...
Hello, My PHP script contains a loop, which does nothing much more than echoing and dereferencing pointers (like in $tab[$othertab[$i]]-> stuff). It was working great until yesterday, when this script starting being VERY slow (like 50 times slower than before). After using strace, i figured out that 90% of the time, the script does m...