OutOfMemory when list files in a directory

Question

When I list files of a directory that has 300,000 files with Java, out of memory occurs.

String[] fileNames = file.list();

What I want is a way that can list all files of a directory incrementally no matter how many files in that specific directory and won't have "out of memory" problem with the default 64M heap limit.

I have Google a while, and cannot find such a way in pure Java.
Please help me!!

Note, JNI is a possible solution, but I hate JNI.

Answer 1

I know you said "with the default 64M heap limit", but let's look at the facts - you want to hold a (potentially) large number of items in memory, using the mechanisms made available to you by Java. So, unless there is some dire reason that you can't, I would say increasing the heap is the way to go.

Here is a link to the same discussion at JavaRanch: http://www.coderanch.com/t/381939/Java-General/java/iterate-over-files-directory

Edit, in response to comment: the reason I said he wants to hold a large number of items in memory is because this is the only mechanism Java provides for listing a directory without using the native interface or platform-specific mechanisms (and the OP said he wanted "pure Java").

Answer 2

Having 300 000 files in a dir is not a good idea - AFAIK filesystems are not good at having that many sub-nodes in a single node. Interesting question, though.

EDIT: THE FOLLOWING DOES NOT HELP, see comments.

I think you could use a FileFilter, reject all files, and process them in the filter.

        new File("c:/").listFiles( new FileFilter() {
            @Override   public boolean accept(File pathname) {
                processFile();
                return false;
            }
        });

Answer 3

You are a bit out of luck here. In the least there will need to be created 300k strings. With an average length of 8-10 char and 2 bytes per char thats 6Mb in the minimum. Add object pointer overhead per string (8 bytes) and you run into your memory limit.

If you absolutely must have that many files in a single dir, which i would not recommend as your file system will have problems, your best bet is to run a native process (not JNI) via Runtime.exec. Keep in mind that you will tie yourself down to the OS (ls vs dir). You will be able to get a list of files as one large string and will be responsible for post processing it into what you want.

Hope this helps.

Answer 4

In JDK7 you may find that the [file walker][1] will work (I haven't tested it). For now, I think you are doing something horrible platform specific, such as running /bin/ls and streaming in the result.

[1]: http://download.java.net/jdk7/docs/api/java/nio/file/Files.html#walkFileTree(java.nio.file.Path, java.nio.file.FileVisitor)