any ideas on how to open this file?
i have no choice whatsoever on the size of this file.
its regular text data
the reason i cannot break up the file is because it is stored remotely and the ONLY thing i can do is run sql statements on it
please help!
for windows there is a really old application called "thegun" that opens any file with any size. under linux/mac you have less/head/tail and you can even use "dd".
What protocols do you have available for opening the file?
A 55G file doesn't sound too bad, can you just download it and handle it locally?
Clearly it would be better if the data providers made more smaller files; a very large file is difficult to handle.
I might suggest using rsync (if you can) as that will enable resuming if the transfer fails (it's going to take a couple of hours I guess, if it's from a remote site).
How often do these files get generated? Can you get the party generating them to create deltas instead? That would probably be a better way to handle it (assuming the data aren't entirely new each time)
Once you download the file, you might want to consider splitting it into more manageable chunks.
Granted, you may end up with HUNDREDS of files, and of course, there's the risk of splitting a file mid-record, but hey, it's an option!
Quick search yielded the following link here with step-by-step instructions. The link mentions that it supports splitting very large files (>4GB) which caught my eye.