Deleting x amount of lines from the top of a text file.

Question

I'm writing an application which will use a log file. What I want to do is when the application starts it will check to see if the log file is above a certain size and if it is it will delete 'x' amounts of lines from the top of the log file to shorten it up. What would be a good way of going about doing this? Would it be easier to write the most recent entries to the top of the file and then delete from the bottom when I do delete?

Answer 1

So you have a log file, let's call it "log.log"

First, move log.log to log.log.bak. Open it for reading. Read line by line until you have read x number of lines. Open a new file, log.log, for writing. Continue reading lines from log.log.bak and for each line write it to log.log. When there are no more lines, close log.log and log.log.bak and then delete log.log.bak.

Some pseudo code:

x = number of lines to delete

move log.log to log.log.bak

open log.log.bak for reading
while i have read < x lines
    read a line and throw it away

open log.log for writing
while there are more lines in log.log.bak
    read a line from log.log.bak
    write the line to log.log

close log.log
close log.log.bak

delete log.log.bak

Answer 2

There are plenty of questions left open-

1. What environment are you in? In a Un*x you could simply 'tail -100 input.txt >trimmed.txt'
2. Can you load the file into memory, or will it be too large?
3. Can you use an intermediate file?
4. What languages do you have available/are you familiar with?
5. How often will you perform the trimming? If you're writing more often than trimming, write to the bottom (which is fast) and perform the expensive trimming operation rarely.

If you have C available, you can find the filesize with fseek(f,0,SEEK_END);long size=ftell(f); (immediately after opening f).

If you need to trim, you can fseek(f,size-desired_size,SEEK_SET); and then while (fgetc(f)!='\n') {}, which will take you to the end of the line you intersect.

Then copy what remains to a new file.

Answer 3

I wouldn't remove a certain number of lines. Basically you have to process the whole file if you do that and that could be a lot of processing. The normal practice is just to roll the log file (rename it something else, often just appending the date) and start again.

Also bear in mind that the file being a certain size is no guarantee that there are the requisite lines there in which case you're just renaming the file the expensive way.

I often find it useful when an app startup begins at the top of a file too.

Answer 4

Keep several log files File1 - FileN for each kind of log. Fill file File1 then File2 and so on when each file passes some fixed size. When you fill FileN delete File1 and start over (deleting) and rewriting File1, File2 ...

This gives you a cyclical fixed size log.

Note: this requires you to keep track of which is the current log file to write to. This can be stored in a separate log file.