How to list a 2 million files directory in java without having a out of memory exception

Question

I have to deal with a directory of about 2 million xml's to be processed.

I've already solved the processing distributing the work between machines and threads using queues and everything goes right.

But now the big problem is the bottleneck of reading the directory with the 2 million files in order to fill the queues incrementally.

I've tried using the File.listFiles() method , but it gives me a java out of memory: heap space Exception. Any ideas?

Answer 1

At fist you could try to increase the memory of your JVM with passing -Xmx1024m e.g.

Answer 2

Why do you store 2 million files in the same directory anyway? I can imagine it slows down access terribly on the OS level already.

I would definitely want to have them divided into subdirectories (e.g. by date/time of creation) already before processing. But if it is not possible for some reason, could it be done during processing? E.g. move 1000 files queued for Process1 into Directory1, another 1000 files for Process2 into Directory2 etc. Then each process/thread sees only the (limited number of) files portioned for it.

Answer 3

Please post the full stack trace of the OOM exception to identify where the bottleneck is, as well as a short, complete Java program showing the behaviour you see.

It is most likely because you collect all of the two million entries in memory, and they don't fit. Can you increase heap space?

Answer 4

Use File.list() instead of File.listFiles() - the String objects it returns consume less memory than the File objects, and (more importantly, depending on the location of the directory) they don't contain the full path name.

Then, construct File objects as needed when processing the result.

However, this will not work for arbitrarily large directories either. It's an overall better idea to organize your files in a hierarchy of directories so that no single directory has more than a few thousand entries.

Answer 5

If file names follow certain rules, you can use File.list(filter) instead of File.listFiles to get manageable portions of file listing.

Answer 6

Try this, it works to me, but I hadn't so many documents...

File dir = new File("directory");
String[] children = dir.list();
if (children == null) {
   //Either dir does not exist or is not a  directory
  System.out.print("Directory doesn't  exist\n");
}
else {
  for (int i=0; i<children.length; i++) {   
    // Get filename of file or directory   
    String filename = children[i];  
}

Answer 7

This is untested and an absolute hack, but you might want to try somthing like this anyway:

Process process = System.getRuntime().exec(new String[]{"ls", "/path"});
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
String line;
while (null != (line = reader.readLine()) {
}