tags:

views:

250

answers:

4

I've been working with large sparse files on openSUSE 11.2 x86_64. When I try to mmap() a 1TB sparse file, it fails with ENOMEM. I would have thought that the 64 bit address space would be adequate to map in a terabyte, but it seems not. Experimenting further, a 1GB file works fine, but a 2GB file (and anything bigger) fails. I'm guessing there might be a setting somewhere to tweak, but an extensive search turns up nothing.

Here's some sample code that shows the problem - any clues?

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    char * filename = argv[1];
    int fd;
    off_t size = 1UL << 40; // 30 == 1GB, 40 == 1TB

    fd = open(filename, O_RDWR | O_CREAT | O_TRUNC, 0666);
    ftruncate(fd, size);
    printf("Created %ld byte sparse file\n", size);

    char * buffer = (char *)mmap(NULL, (size_t)size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if ( buffer == MAP_FAILED ) {
        perror("mmap");
        exit(1);
    }
    printf("Done mmap - returned 0x0%lx\n", (unsigned long)buffer);

    strcpy( buffer, "cafebabe" );
    printf("Wrote to start\n");

    strcpy( buffer + (size - 9), "deadbeef" );
    printf("Wrote to end\n");

    if ( munmap(buffer, (size_t)size) < 0 ) {
        perror("munmap");
        exit(1);
    }
    close(fd);

    return 0;
}
A: 

Stopping at 2 GB (231 bytes) sounds like you have not got big file support enabled. Are you sure you're compiling 64-bit?

Jonathan Leffler
Yes - I'm compiling with 64-bit - as far as I can see: pat@opensuse:~> file workspace/testmmap/Debug/testmmap workspace/testmmap/Debug/testmmap: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.4, not stripped
metadaddy
Also, the 1TB sparse file gets created fine: pat@opensuse:~> ls -ls csparse 0 -rw-r--r-- 1 pat users 1099511627776 2010-05-25 20:27 csparse
metadaddy
+1  A: 

Is there some sort of per-user quota, limiting the amount of memory available to a user process?

Martin Beckett
Yep - I tried bmargulies' suggestion of trying ulimit -a and that pointed to the 'virtual memory' process limit as the culprit - see my answer below...
metadaddy
A: 

My guess is the the kernel is having difficulty allocating the memory that it needs to keep up with this memory mapping. I don't know how swapped out pages are kept up with in the Linux kernel (and I assume that most of the file would be in the swapped out state most of the time), but it may end up needing an entry for each page of memory that the file takes up in a table. Since this file might be mmapped by more than one process the kernel has to keep up with the mapping from the process's point of view, which would map to another point of view, which would map to secondary storage (and include fields for device and location).

This would fit into your addressable space, but might not fit (at least contiguously) within physical memory.

If anyone knows more about how Linux does this I'd be interested to hear about it.

nategoose
Linux won't create the PTEs (page table entries) until those pages are actually touched. All it does when you create the mapping is create a single VMA (virtual memory area) structure, that basically contains the information from the `mmap()`.
caf
@caf: Thanks for the info
nategoose
+1  A: 

The problem was that the per-process virtual memory limit was set to only 1.7GB. ulimit -v 1610612736 set it to 1.5TB and my mmap() call succeeded. Thanks, bmargulies, for the hint to try ulimit -a!

metadaddy
And, apparently, I can set my desired value (which can be 'unlimited') in /etc/profile to make it persistent.
metadaddy