Count the number of lines in a file with Ruby, without reading entire file into memory

Question

I'm processing huge data files (millions of lines each). Before I start processing I'd like to get a count of the number of lines in the file, so I can then indicate how far along the processing is. I am using Ruby, and because of the size of the files, it would not be practical to read the entire file into memory just to count how many lines there are. Does anyone have a good suggestion on how to do this?

Answer 1

It doesn't matter what language you're using, you're going to have to read the whole file if the lines are of variable length. That's because the newlines could be anywhere and theres no way to know without reading the file (assuming it isn't cached, which generally speaking it isn't).

If you want to indicate progress, you have two realistic options. You can extrapolate progress based on assumed line length:

assumed lines in file = size of file / assumed line size
progress = lines processed / assumed lines in file * 100%

since you know the size of the file. Alternatively you can measure progress as:

progress = bytes processed / size of file * 100%

This should be sufficient.

Answer 2

If you are on unix environment, you can just let wc -l do the work. It will not load the whole file into memory; since it is optimized for streaming file and count word/line the performance is good enough rather then streaming the file yourself in ruby.

Answer 3

Reading the file a line at a time:

count = File.foreach(filename).inject(0) {|c, line| c+1}

Will be slower than

count = %x{wc -l #{filename}}.split.first.to_i