Sampling Large Data Files

Question

Hello,

I currently work in the position of Data Warehouse programmer and as such have to put numerous flat files through ETL process. Of course prior to loading the file I have to be aware of its content, the problem is that majority of the files are > 1 GB large and I can not open them using my dear old friend "notepad". Kidding. I usually use VIM or Notepad++ but it still takes a while to open the file. Could I perform a "partial" read of the file using VIM or some other editor?

P.S. I know that I could write a 10 liner script to "data sample" the file, but it would be simpler to convince team members to use a feature of an editor than a script that I wrote.

Thank you for any insight you might have.

Answer 1

If you want to stick with using vim, you could have a look at the LargeFile script.

Alternatively, I've always found that UltraEdit opens large files extremely quickly.

Answer 2

You said you had VIM, that makes me wonder if you have a unix environment as well?

If you like, you can pipe the input through unix utility top and display the raw imput on your screen. Like this:

EDIT: (thanks Honk)

terminal$> head -N 15 file.csv

(Where that 15 indicates you want to see 15 lines only).

Answer 3

Pretty sure there are loads of similar questions, but hey, Textpad is a good choice for this.

Answer 4

UltraEdit claims to handle files over 4GB...

Answer 5

use the head command.

Answer 6

Use the 'less' on solaris ... use the same through cygwin on windows. On mainframes this problem doesn't appear, ISPF editor handles it pretty well.

Answer 7

Try PilotEdit, PilotEdit works well for files larger than 1GB. http://www.pilotedit.com

Answer 8

EmEditor handles huge files like nobody's business. Up to 248GB, according to their website...and more than that, by opening sections of it. So ~1GB shouldn't be an issue.