Benches macOS

15

How to increase the limit of “maximum ope…

apple Verified

Imported from the Apple Support Forums via the Wayback Machine 🔗

The default limit for the max open files on Mac OS X is 256 (ulimit -n) and my application needs about 400 file handlers.

I tried to change the limit with setrlimit() but even if the function executes correctly, i'm still limited to 256.

Here is the test program I use (compiled with gcc):


#include <stdio.h>
#include <sys/resource.h>
main()
{
struct rlimit rlp;
FILE *fp[10000];
int i;
getrlimit(RLIMIT_NOFILE, &rlp);
printf("before %d %d ", rlp.rlim_cur, rlp.rlim_max);
rlp.rlim_cur = 10000;
setrlimit(RLIMIT_NOFILE, &rlp);
getrlimit(RLIMIT_NOFILE, &rlp);
printf("after %d %d ", rlp.rlim_cur, rlp.rlim_max);
for(i=0;i<10000;i++) {
fp = fopen("a.out", "r");
if(fp
==0) { printf("failed after %d ", i); break; }
}
}

and the output is:


before 256 -1
after 10000 -1
failed after 253

I cannot ask the people who use my application to poke inside a /etc file or something. I need the application to do it by itself.

15 replies

Sign in or join Beachball Help to reply.

  1. 0

    etresoft

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    acemtp wrote:

    The default limit for the max open files on Mac OS X is 256 (ulimit -n) and my application needs about 400 file handlers.

    Why?

  2. 0

    acemtp

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    It seems there's a bug in the stdio since open() works after the setrlimit, but not the fopen() function.

  3. 0

    Jongware

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    I think etre rather meant, why does an application need 400 files opened +at the same time+. The default of 256 probably was defined as "let's take a large number that no-one ought to need".

  4. 0

    acemtp

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    The problem is that we have a function 'setrlimit' that doesn't work on Mac OS X. The max limit was unlimited, so it should be possible to increase the current limit from 256 to at least 400 but it seems it's not possible... why?

  5. 0

    laurent demaret

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    acemtp wrote:

    The problem is that we have a function 'setrlimit' that doesn't work on Mac OS X. The max limit was unlimited, so it should be possible to increase the current limit from 256 to at least 400 but it seems it's not possible... why?

    I don't know but what about "not to put useless burden" on the OS ?

    Ok, you are the boss ..

    Did you read the man page ?

    If I use Bwana (thanks to être for the link
    http://www.bruji.com/bwana/) and type man:setrlimit in Safari's navigation field I can read

    " Only the super-user may raise the maximum limits. Other users may only

    alter rlim_cur within the range from 0 to rlim_max or (irreversibly)

    lower rlim_max."

    Same if you type man setrlimit in the Xcode help menu for Man Pages or in the terminal..

    So, maybe, you were not superuser when you tried to put machine at hard work for your purposes ...

    Or maybe any other reason given in the man page.

  6. 0

    acemtp

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    Yes I read the man page...

    There's 2 limits:


    struct rlimit {
    rlim_t rlim_cur; /* Soft limit */
    rlim_t rlim_max; /* Hard limit (ceiling for rlim_cur) */
    };

    The "max" limit on mac os X is "unlimited" and to change it, you need to be root. I don't want to change the "max" limit since it's already "unlimited".

    What I want to change is the "cur" limit, that is 256 on mac. And for changing "cur", there's no need to be root. And if you run the test program i gave in the first post, you'll see that there's no error and getrlimit shows that the "cur" limit changed correctly.

    The problem is that fopen() standard C function didn't take into account this limit change.

  7. 0

    hansz

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    Try this:


    #include <stdio.h>
    #include <stdlib.h>
    #include <sys/resource.h>

    int
    main(void)
    {
    struct rlimit rlp;
    FILE *fp[10000];
    int i, r;
    r = getrlimit(RLIMIT_NOFILE, &rlp);
    if (r == -1) {
    perror("getrlimit()");
    exit(1);
    }
    printf("before %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
    rlp.rlim_cur = 10000;
    r = setrlimit(RLIMIT_NOFILE, &rlp);
    if (r == -1) {
    perror("setrlimit()");
    exit(1);
    }
    r = getrlimit(RLIMIT_NOFILE, &rlp);
    if (r == -1) {
    perror("setrlimit()");
    exit(1);
    }
    printf("after %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
    for (i = 0; i < 10000; i++) {
    fp = fopen("a.out", "r");
    if (fp
    == NULL) {
    perror("fopen()");
    fprintf(stderr, "i == %d ", i);
    exit(1);
    }
    }
    exit(0);
    }

    You will notice that I

    o Added <stdlib.h> to use exit().

    o Added a check for an error return on all system calls

    o Use perror() if a system call fails

    When I run your program either as "me" or as root I see (correctly)


    $ cc -Wall -o rlimit rlimit.c
    $ touch a.out && ./rlimit
    before 256 -1
    after 10000 -1
    fopen(): Too many open files
    i == 9997

    So
    a.out is opened 9997 times, which is what you expect as standard in (0), out (1) and error (2) are open by default.

    As to why you want to have that many files open, having lots of network connections open in a server is not uncommon.

    Regards,

    Hans

  8. 0

    acemtp

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    hansz,

    I bet that if you execute "ulimit -n" in a terminal, you'll have more than 256 (perhaps unlimited). That explains why it works for you. But I tested your program, and of course, it doesn't exactly the same thing as mine, failed after "i==253".

    It's clear that on older mac os x, the limit was higher but if you install 10.6.4 without upgrading, you have only 256.

  9. 0

    hansz

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    {quote}

    I bet that if you execute "ulimit -n" in a terminal, you'll have more than 256 (perhaps unlimited).

    {quote}

    Lucky we didn't enter in a wager 😉


    $ ulimit -n
    256

    Maybe there is a difference between 10.5 and 10.6. I run 10.5.8. What do these two say on your system?


    $ sysctl kern.maxfiles
    kern.maxfiles: 12288
    $ sysctl kern.maxfilesperproc
    kern.maxfilesperproc: 10240

    Regards,

    Hans

  10. 0

    acemtp

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    Wow, incredible. I don't understand at all. So it's really a new bug introduced with 10.6?

    I'm with latest mac os x 10.6.4.

    I have the same values for kernel:


    sysctl kern.maxfiles
    kern.maxfiles: 12288
    sysctl kern.maxfilesperproc
    kern.maxfilesperproc: 10240

  11. 0

    hansz

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    {quote}

    So it's really a new bug introduced with 10.6?

    {quote}

    Highly unlikely, I guess half the (Mac) world would keel over if such an obvious bug was introduced. Are you
    sure
    setrlimit(2) does not return -1 on your system?

    How much memory do you have in your system, I have 4GiB. That could influence the number of files that can be open.

    My 10.4 PowerBook gives the same numbers as 10.5, but does not want to open more than 253 files, both as a normal user and as root. It has a (mere) 512MiB.

    An other way to find the cause is to look through the kernel source, see [http://www.opensource.apple.com/release/mac-os-x-1064/xnu-1504.7.4]. I'll give it a try this afternoon as I have a long and boring train ride ahead...

    Regards,

    Hans

  12. 0

    etresoft

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    hansz wrote:

    As to why you want to have that many files open, having lots of network connections open in a server is not uncommon.

    But a good server would fork enough processes to handle the load instead of trying to do it all in one app.

    There is a good reason for this limit. The select() function, by default, is limited to this value. So, not only is our hapless original poster not using select(), the program is going to require admin authorization each and every time it runs.

    There is a reason why increasing this limit is so hard - you aren't supposed to do it!

  13. 0

    acemtp

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    I have the latest MacBook Pro 15inches. It contains 4GiB of RAM.

    I'm sure the setrlimit doesn't return -1, I just compiled and executed your version of the test program. And since getrlimit displays correctly the new limit (10000) without error too, we can imagine that the new limit is set.

  14. 0

    etresoft

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    There is no reason to have more than 256 files open. Whatever you are doing, you are doing it wrong.

    The setrlimit() function only applies to new processes. It does not affect the current process. You can either run "ulimit -n 10000" in the shell and then start your program or split your program into two segments:


    #include <stdio.h>
    #include <stdlib.h>
    #include <sys/resource.h>
    #include <unistd.h>
    int main(void)
    {
    struct rlimit rlp;
    int r;
    r = getrlimit(RLIMIT_NOFILE, &rlp);
    if (r == -1)
    {
    perror("getrlimit()");
    exit(1);
    }

    printf("before %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
    rlp.rlim_cur = 10000;

    /* I'm curious how anyone ever expected it to work or got it to work
    without setting the max, because it sure seems to be required. */
    rlp.rlim_max = 10000;

    r = setrlimit(RLIMIT_NOFILE, &rlp);
    if (r == -1)
    {
    perror("setrlimit()");
    exit(1);
    }
    r = getrlimit(RLIMIT_NOFILE, &rlp);
    if (r == -1)
    {
    perror("setrlimit()");
    exit(1);
    }

    printf("after %d %d ", (int) rlp.rlim_cur, (int) rlp.rlim_max);
    return execl("/tmp/f", "/tmp/f", NULL);
    }

    And the source to the "f" program is:


    #include <stdio.h>

    int main(void)
    {
    FILE *fp[10000];
    int i;
    for(i = 0; i < 10000; i++)
    {
    fp = fopen("a.out", "r");
    if (fp
    == NULL)
    {
    perror("fopen()");
    fprintf(stderr, "i == %d ", i);

    return 1;
    }
    }
    return 0;
    }

    [jdaniel@Pele:584] /tmp $ a.out
    before 256 256
    setrlimit(): Operation not permitted
    [jdaniel@Pele:585] /tmp $ sudo a.out
    before 256 256
    after 10000 10000
    fopen(): Too many open files
    i == 9997

    As you can see, it is convoluted and a real hassle. It isn't a bug in MacOS X 10.6 or any other version.

    So, I repeat my question yet again. Why do you want to open more than 256 files? It is, of course, a rhetorical question. Don't do it. If you feel you need more than 256 open files, review your architecture and correct it.

  15. 0

    xnav

    Imported from the Apple Support Forums via the Wayback Machine 🔗

    FYI, if I run on my 10.6.4


    Path:~/xcodeprojects/wall/build/debug$touch a.out
    Path:~/xcodeprojects/wall/build/debug$./wall
    before 256 -1
    after 10000 -1
    fopen(): Too many open files
    i == 253

    If I run under Xcode


    run
    [Switching to process 815]
    Running…
    before 9472 -1
    after 10000 -1
    fopen(): Too many open files
    i == 9469
    Debugger stopped.
    Program exited with status value:1.(gdb)

    Seems to validate etresoft.