I'm trying to map a VME address space through a PCI bus into user space so I can perform regular read/writes on the memory. I have done this with another PCI device like this :- unsigned long *mapArea(unsigned int barAddr, unsigned int mapSize, int *fd) { unsigned long *mem; *fd = open("/dev/mem", O_RDWR); if ( *fd<0 ) { ...
We need to read and count different types of messages/run some statistics on a 10 GB text file, e.g a FIX engine log. We use Linux, 32-bit, 4 CPUs, Intel, coding in Perl but the language doesn't really matter. I have found some interesting tips in Tim Bray's WideFinder project. However, we've found that using memory mapping is inherent...
Does anyone know what the difference between mmap(2) and mmap(3) is? Man section 3 is described as "This chapter describes all library functions excluding the library functions described in chapter 2, which implement system calls." Doesn't mmap(3) perform a system call? Reading the two man pages, I see that mmap(2) seems to accept a m...
I am trying to display search result data quickly. I have all absolute file paths for files on my network drive(s) in a single, ~50MB text file. The python script makes a single pass over every line in this file [kept on the local drive] in a second or less, and that is acceptable. That is the time it takes to gather results. However, ...
Im struggling to read(and write) to HW registers from Linux user space. The goal is to configure some GPIO pins from and be able to set and read this pins. According to the spec for the processor(imx27 from Freescale) the physical address for the register bank controlling GPIO this is 0x10015000 My assumption was that I could use somet...
It was recently asked how to do a file slurp in python: link text And it was recommended to use something like with open('x.txt') as x: f = x.read() How would I go about doing this to read the file in and convert the endian representation of the data? For example, I have a 1GB binary file that's just a bunch of single precision flo...
I am writing a module that amongst other things allows bitwise read access to files. The files can potentially be large (hundreds of GB) so I wrote a simple class that lets me treat the file like a string and hides all the seeking and reading. At the time I wrote my wrapper class I didn't know about the mmap module. On reading the docum...
Hi, I'm trying to project a file into memory to operate with it. The file contais structs so I'm trying to use a pointer to the start of one struct and then read it and modify some variable. The problem is that the time of execution is high and I suppose that using mmap the time will be less. This is the code, any suggestion? #include ...
Hi, I built two programs, one using malloc and other one using mmap. The execution time using mmap is much less than using malloc. I know for example that when you're using mmap you avoid read/writes calls to the system. And the memory access are less. But are there any other reasons for the advantages when using mmap over malloc? T...
When we map a file to memory, a system call is required. Do subsequent accesses to the file require system calls or is the virtual memory page of the process mapped to the actual page cache in memory? update: what i also want to know is that if multiple processes are accessing the same file through mmap. they will be accessing the same ...
Consider this because this region maps the files like dynamically loaded libraries, i feel it should ideally grow up. this can be implemented by starting the mmap region between RLIMIT_STACK and heap beginning. what problems would occur in this case. if it grows down, then how is a new memory mapped region created. suppose we wish to ma...
Problem Description I need to stream large files from disk. Assume the files are larger than will fit in memory. Furthermore, suppose that I'm doing some calculation on the data and the result is small enough to fit in memory. As a hypothetical example, suppose I need to calculate an md5sum of a 200GB file and I need to do so with gu...
Platofrm - Linux, Arch - ARM Programming lang - C/C++ Objective - map a regular (let say text) file to a pre-known location (physical address) in ram and pass that physical address to some other application. Size of the block which i map at a time is 128K. The way I am trying to go about is- User space process issues the ioctl call t...
I have several fastcgi processes that are supposed to share data. The data is bound to a session (a unique session id string) and should be able to survive a server reboot. Depending on the number of sessions, the shared data might be too big to fit into main memory. Ideally, in the case when the shared data exceeds a certain treshold, t...
I got some huge files I need to parse, and people have been recommending mmap because this should avoid having to allocate the entire file in-memory. But looking at 'top' it does look like I'm opening the entire file into the memory, so I think I must be doing something wrong. 'top shows >2.1 gig' This is a code snippet that shows what...
Helllo, I would like to share small amounts of data (< 1K) between python and processes. The data is physical pc/104 IO data which changes rapidly and often (24x7x365). There will be a single "server" writing the data and multiple clients reading portions of it. The system this will run on uses flash memory (CF card) rather than a hard d...
Exposition: I am on Linux / Mac. Part of my code is in Java, part of my code is in C++. They both have the same file mmapped for fast communication. I want to synchronize the Java & C++ code. I know the following: 1) given two threads in Java, I can use Locks / monitors. 2) given one piece of code in Java, one in C++, I can have t...
I have a program. I want it to be able to mmap a particular region of memory over different runs. I have the source code of the program. C/C++ I control how the program is compiled. gcc I control how the program is linked. gcc I control how the program is run (Linux). I just want to have this particular region of memory, say 0xabcdab...
I have two processes P1 and P2. I have this large read-only resource, called "R" that I want both P1 and P2 to have access to. R is not just a "flat" group of bytes; it's a bunch of C++ objects that point to each other. I would prefer that P1 and P2 only share one copy of R -- somehow have P1 load R into a region in memory (that's mma...
I've been considering using mmap for filereading, and was wondering how portable that is. I'm developing on a linux platform, but would like my program to work on osx and windows. Can I assume mmap is working on these platforms? thanks ...