How to test "Too Many Files Open" problem.

Question

I have a batch process that converts WAV to MP3 sequentially. The problem is that after a few thousand there are too many files left open, and it runs up against the file limit.

The reason it does this is because of the code in SystemCommandTasklet:

FutureTask<Integer> systemCommandTask = new FutureTask<Integer>(new Callable<Integer>() {
    public Integer call() throws Exception {
        Process process = Runtime.getRuntime().exec(command, environmentParams, workingDirectory);
        return process.waitFor();
    }
});

This has the nasty side effect of making me rely on the JVM to clean up the processes, leaving files open and such.

I've rewritten it to be so:

FutureTask<Integer> systemCommandTask = new FutureTask<Integer>(new Callable<Integer>() {
    public Integer call() throws Exception {
        Process process = Runtime.getRuntime().exec(command, environmentParams, workingDirectory);
        int status = process.waitFor();

        process.getErrorStream().close();

        process.getInputStream().close();

        process.getOutputStream().flush();
        process.getOutputStream().close();

        process.destroy();

        return status;
    }

});

I'm 95% certain that this works on my mac (thanks to lsof), but how do I make a proper test that will work on any system to PROVE that what I am trying to do is actually working?

Answer 1

You could try and code to avoid it.

Why don't you proactively limit number of tasks at a time say 100 ?

In that case you could use some pooling mechanism to execute you work like Thread Pools.

Answer 2

A proof will be difficult. But ...

Create a (Dummy)command that doesn't do much but keeps a lock on the files, just as the real thing. This makes sure your test doesn't depend on the actual command used.

Create a test that starts SystemCommandTask, using the old version, but the DummyCommand. Make it start the task often, until you get the expected exception. Lets call the number of Tasks needed N

Change the test to start 100xN Tasks.

Change the Task to the new version. If the test goes green you should be reasonably sure that your code works.

Answer 3

1. You should not use Runtime#exec() for that, because the process is not attached to the JVM's. Please take a look on the j.l.ProcessBuilder which returns a Process which is controlled by the JVM's process. So dumping the process may force the system to free / close resources.
2. You should schedule and limit the processes as well using j.u.c.Executors.
3. You also may read the limit using "ulimit -Sn" ("ulimit -Hn" should not be prefered due to system health ;).
4. Check your tool that converts your media whether it keeps resources reserved after completion (leakage, waiting for caller signals etc).