How to edit multi-gigabyte text files? Vim doesn't work =(

Question

Are there any editors that can edit multi-gigabyte text files, perhaps by only loading small portions into memory at once? It doesn't seem like Vim can handle it =(

Answer 1

Wow, never managed to get vim to choke, even with a GB or two. I've heard that UltraEdit (on Windows) and BBEdit (on Macs) are even more suitable for even-larger files, but I have no personal experience.

Answer 2

In the past I opened up to a 3 gig file with this tool http://csved.sjfrancke.nl/

Answer 3

I have used TextPad for large log files it doesn't have an upper limit.

Answer 4

You might want to check out this VIM plugin which disables certain vim features in the interest of speed when loading large files.

Answer 5

I've tried to do that, mostly with files around 1 GB when I needed to make some small change to an SQL dump. I'm on Windows, which makes it a major pain. It's seriously difficult.

The obvious question is "why do you need to?" I can tell you from experience having to try this more than once, you probably really want to try to find another way.

So how do you do it? There are a few ways I've done it. Sometimes I can get vim or nano to open the file, and I can use them. That's a really tough pain, but it works.

When that doesn't work (as in your case) you only have a few options. You can write a little program to make the changes you need (for example, search & replaces). You could use a command line program that may be able to do it (maybe it could be accomplished with sed/awk/grep/etc?)

If those don't work, you can always split the file into chunks (something like split being the obvious choice, but you could use head/tail to get the part you want) and then edit the part(s) that need it, and recombine later.

Trust me though, try to find another way.

Answer 6

In the past I've successfully used a split/edit/join approach when files get very large. For this to work you have to know about where the to-be-edited text is, in the original file.

Answer 7

WinVi can handle files up to 2GB in size.

Answer 8

Personally, I like UltraEdit. Here is their little spiel on large files.

Answer 9

I've used FAR Commander's built-in editor/viewer for super-large log files.

Answer 10

The only thing I've been able to use for something like that is my favorite Mac hex editor, 0XED. However, that was with files that I considered large at tens of megabytes. I'm not sure how far it will go. I'm pretty sure it only loads parts of the file into memory at once, though.

Answer 11

It may be plugins that are causing it to choke. (syntax highlighting, folds etc.)

you can run vim without plugins.

vim -u "NONE" hugefile.log

Its minimalist but it will at least give you the vi motions your used to.

syntax off

is another obvious one. Prune your install down and source what you need. You'll find out what it's capable of and if you need to accomplish a task via other means.

Answer 12

If you are on *nix (and assuming u have to modify only parts of file & not often !) ,you may split the files (using split command), edit them individually (awk/sed etc.) & concatenate them after you are done . eg. cat file2 file3 >> file1.

Answer 13

I think it is reasonably common for hex editors to handle huge files. On Windows, I use HxD, which claims to handle files up to 8 EB (8 billion gigabytes).

Answer 14

Try PilotEdit. It may take you 5 minutes to open/save a 1GB file. But after you open it, you can view/edit it very quickly.

Answer 15

EmEditor is basically built for just this purpose. No compromise of speed for size.