I'm working on a program that will be processing files that could potentially be 100GB or more in size. The files contain sets of variable length records. I've got a first implementation up and running and am now looking towards improving performance, particularly at doing I/O more efficiently since the input file gets scanned many times...
POSIX environments provide at least two ways of accessing files. There's the standard system calls open(), read(), write(), and friends, but there's also the option of using mmap() to map the file into virtual memory. When is it preferable to use one over the other? What're their individual advantages that merit including two interfac...
Hi, I'm working on a java web application that uses thousands of small files to build artifacts in response to requests. I think our system could see performance improvements if we could map these files into memory rather than run all over the disk to find them all the time. I have heard of mmap in linux, and my basic understanding of...
I'm currently running several copies of PHP/FastCGI, with APC enabled (under Apache+mod_fastcgi, if that matters). Can I share cache between the processes? How can I check if it's shared already? (I think the apc.mmap_file_mask ini setting might be involved, but I don't know how to use it.) (One of the reasons I think its not shared ...
Does protection flag affect the sharing between processes? If I have PROT_READ|PROT_WRITE -protected mmapped memory region, is it still fully shared as long as I haven't written into it? int prot = PROT_READ|PROT_EXEC; image = mmap(NULL, filesize, prot, MAP_PRIVATE, fildes, 0); vs: int prot = PROT_READ|PROT_WRITE|PROT_EXEC; image = m...
I would like to pass a pointer (I am putting a file with data in memory with mmap) to processes spawned using fork + exec, but I am stuck on how to pass a pointer to the exec() spawned process? UPDATE1: Thanks for your inputs, I do use shared memory creating it with mmap with MAP_INHERIT flag: Each mapped file and shared memory region...
I'm using the driver I posted at http://stackoverflow.com/questions/647783/direct-memory-access-in-linux/ to mmap some physical ram into a userspace address. However, I can't use GDB to look at any of the address; i.e., x 0x12345678 (where 0x12345678 is the return value of mmap) fails with an error "Cannot access memory at address 0x1234...
What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits? ...
I'm trying to allow two different processes to communicate by using memory mapping the same file. However, I'm having some problems with this. I have a feeling this has to do with the way I'm using the open() call and passing my file descriptor to mmap. Here is my code, can you see anything wrong with it? Object 1's code: 16 FIL...
I have several questions regarding the mmap implementation in Linux systems which don't seem to be very much documented: When mapping a file to memory using mmap, how would you handle prefetching the data in such file? I.e. what happens when you read data from the mmaped region? Is that data moved to the L1/L2 caches? Is it read direct...
It's widely known that the most significant mmap() feature is that file mapping is shared between many processes. But it's not less widely known that every process has its own address space. The question is where are memmapped files (more specifically, its data) truly kept, and how processes can get access to this memory? I mean not *(p...
I used mmap(just try to understand how mmap works) to allocate 96k anonymous memory, but looks like it split the 96k into 64k and 32k. But when allocate 960k, it allocate only one chunk whose size is 960k. When solaris will split the allocate mem into several part? Code: #define PROT PROT_READ | PROT_WRITE #define MAP MAP_ANON | MAP_P...
Does FileChannel#map allocate all the memory needed for the resulting ByteBuffer immediately, or is it only allocated on-demand during reads from the buffer? I just tried mapping all of a 500+ MB file in a trivial test program, and looked at the memory usage of the process. (Using both Runtime#totalMemory and eyeballing it in the OS X A...
when use mmap to allocate some anonymous mem, we often set the start address as 0/null so mmap will figure out the starting address by itself. And to get the start address, it will work thought the whole virtual memory space to find a hole which could put the chuck of mem to be allocated. I guess this is calculated as user cpu instead of...
Hi. Does someone know how well the following 3 compare in terms of speed: shared memory tmpfs (/dev/shm) mmap (/dev/shm) Thanks! ...
If I use mmap to write uint32_t's, will I run into issues with big endian/little endian conventions? In particular, if I write some data mmap'ed on a big-endian machine, will I run into issues when I try to read that data on a little-endian machine? ...
I'm trying to optimize handling of large datasets using mmap. A dataset is in the gigabyte range. The idea was to mmap the whole file into memory, allowing multiple processes to work on the dataset concurrently (read-only). It isn't working as expected though. As a simple test I simply mmap the file (using perl's Sys::Mmap module, using...
The clone() system call on Linux takes a parameter pointing to the stack for the new created thread to use. The obvious way to do this is to simply malloc some space and pass that, but then you have to be sure you've malloc'd as much stack space as that thread will ever use (hard to predict). I remembered that when using pthreads I didn...
I have a memory mapped file, from which I wish to parse the contents of the buffer. The mmap() returns success, and I can print out the buffer contents to a file using fprintf successfully. However, when I try to access the buffer as an array in my program directly, I get a segmentation fault. Why is this happening? Here is the code: ...
Hello! I'm implementing persistent large constant arrays via mmap. Is there any tips and tricks or gotchas one should be aware when using mmap? ...