How can I handle large files in Ruby?

Question

I'm pretty new to programming, so be gentle. I'm trying to extract IBSN numbers from a library database .dat file. I have written code that works, but it is only searching through about half of the 180MB file. How can I adjust it to search the whole file? Or how can I write a program the will split the dat file into manageable chunks?

edit: Here's my code:

export = File.new("resultsfinal.txt","w+")

File.open("bibrec2.dat").each do |line|
  line.scan(/[a]{1}[1234567890xX]{10}\W/) do |x|
    export.puts x
  end
  line.scan(/[a]{1}[1234567890xX]{13}/) do |x|
    export.puts x
  end
end

Answer 1

If you are programming on a modern operating system and the computer has enough memory (say 512megs), Ruby should have no problem reading the entire file into memory.

Things typically get iffy when you get to about a 2 gigabyte working set on a typical 32bit OS.

Answer 2

(Content entered into original question, please delete this)

Answer 3

As to the performance issue, I can't see anything particularly worrying about the file size: 180MB shouldn't pose any problems. What happens to memory use when you're running your script?

I'm not sure, however, that your Regular Expressions are doing what you want. This, for example:

/[a]{1}[1234567890xX]{10}\W/

does (I think) this:

• one "a". Do you really want to match for an "a"? "a" would suffice, rather than "[a]{1}", in that case.
• exactly 10 of (digit or "x" or "X")
• a single "non-word" character i.e. not a-z, A-Z, 0-9 or underscore

There are a couple of sample ISBN matchers here and here, although they seem to be matching something more like the format that we see on the back cover of a book and I'm guessing your input file has stripped out some of that formatting.

Answer 4

You should try to catch exception to check if the problem is really on the read block or not.

Just so you know I already made a script with kinda the same syntax to search real big file of ~8GB without problem.

export = File.new("resultsfinal.txt","w+")

File.open("bibrec2.dat").each do |line|
  begin
    line.scan(/[a]{1}[1234567890xX]{10}\W/) do |x|
      export.puts x
    end
    line.scan(/[a]{1}[1234567890xX]{13}/) do |x|
      export.puts x
    end
  rescue
    puts "Problem while adding the result"
  end
end