How to increase the limit of "maximum open files" in C on Mac OS X

Question

The default limit for the max open files on Mac OS X is 256 (ulimit -n) and my application needs about 400 file handlers.

I tried to change the limit with setrlimit() but even if the function executes correctly, i'm still limited to 256.

Here is the test program I use:

#include <stdio.h>
#include <sys/resource.h>

main()
{
  struct rlimit rlp;

  FILE *fp[10000];
  int i;

  getrlimit(RLIMIT_NOFILE, &rlp);
  printf("before %d %d\n", rlp.rlim_cur, rlp.rlim_max);

  rlp.rlim_cur = 10000;
  setrlimit(RLIMIT_NOFILE, &rlp);

  getrlimit(RLIMIT_NOFILE, &rlp);
  printf("after %d %d\n", rlp.rlim_cur, rlp.rlim_max);

  for(i=0;i<10000;i++) {
    fp[i] = fopen("a.out", "r");
    if(fp[i]==0) { printf("failed after %d\n", i); break; }
  }

}

and the output is:

before 256 -1
after 10000 -1
failed after 253

I cannot ask the people who use my application to poke inside a /etc file or something. I need the application to do it by itself.

Answer 1

I know that's sound a silly question, but you really need 400 files opened at the same time? By the way, are you running this code as root are you?

Answer 2

rlp.rlim_cur = 10000;

Two things.

1st. LOL. Apparently you have found a bug in the Mac OS X' stdio. If I fix your program up/add error handling/etc and also replace fopen() with open() syscall, I can easily reach the limit of 10000 (which is 240 fds below my 10.6.3' OPEN_MAX limit 10240)

2nd. RTFM: man setrlimit. Case of max open files has to be treated specifically regarding OPEN_MAX.

Answer 3

This may be a hard limitation of your libc. Some versions of solaris have a similar limitation because they store the fd as an unsigned char in the FILE struct. If this is the case for your libc as well, you may not be able to do what you want.

As far as I know, things like setrlimit only effect how many file you can open with open (fopen is almost certainly implemented in terms on open). So if this limitation is on the libc level, you will need an alternate solution.

Of course you could always not use fopen and instead use the open system call available on just about every variant of unix.

The downside is that you have to use write and read instead of fwrite and fread, which don't do things like buffering (that's all done in your libc, not by the OS itself). So it could end up be a performance bottleneck.

Can you describe the scenario that requires 400 files open ** simultaneously**? I am not saying that there is no case where that is needed. But, if you describe your use case more clearly, then perhaps we can recommend a better solution.

Answer 4

etresoft found the answer on the apple discussion board:

The whole problem here is your printf() function. When you call printf(), you are initializing internal data structures to a certain size. Then, you call setrlimit() to try to adjust those sizes. That function fails because you have already been using those internal structures with your printf(). If you use two rlimit structures (one for before and one for after), and don't print them until after calling setrlimit, you will find that you can change the limits of the current process even in a command line program. The maximum value is 10240.