tags:

views:

256

answers:

4

I have a pair of bash scripts, 1 that dumps mysql dbs, and the second to purge old backups.

I have always relied on date, so any files older than 7 days are purged, and new backups created daily. The result was a set of backups 7 days back.

Well now that I have 20+ dbs I have separated that daily job into a daily, and weekly job. If I continue to use the find and date method I would lose the past week.

QUESTION

How can I sort the output of a find by date, and then purge all but the most recent 7. (so even if the most recent is 2 months old, it remains)

This is as far as I got, but it is not as expected. Perhaps using ls might be the better choice 9find is a remnant from using date as the sole criteria for purging)

OLDFILES=`find -regex .*sql.*`

set -- $OLDFILES

if [ -z $1 ]
then
    printf "\tNo files matching purge criteria\n" | tee -a $LOGFILE
else
    printf "\tSQL Files being Delete from $HERE\n" | tee -a $LOGFILE
    printf "\t\t%s\n" $OLDFILES  | tee -a $LOGFILE
fi
# $RETAIN is the # of recent to keep
# so I want to grab a subset from 0 to (total - retain), and purge them
TOBURN=$(( $# - $RETAIN ))
printf " grab first %s files. %s  - %s" $TOBURN $# $RETAIN
if [ $TOBURN -gt 0 ]
then
    TOPURGE=("$@:0:$TOBURN")
    printf "\n\nREMOVING..\n\t\t%s\n" $TOPURGE
fi

Existing files

            ./webbmaster_bellarose_joomla_01-09-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-25-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-10-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-04-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-25-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-29-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-30-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-28-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-06-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-13-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-24-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-21-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-24-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-27-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-26-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-17-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-03-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-21-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-20-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-16-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-31-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-11-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-05-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-14-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-22-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-12-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-17-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-18-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-20-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-08-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-22-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-27-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-23-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-07-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-26-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-19-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-29-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-15-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-28-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-01-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-18-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-02-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-23-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-19-2009.sql.gz

Targets for purging (which I want to set in variable for simple rm -f $var)

            ./webbmaster_bellarose_joomla_01-09-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-25-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-10-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-04-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-30-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-28-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-06-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-13-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-21-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-24-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-27-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-17-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-03-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-21-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-20-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-16-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-31-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-11-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-05-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-14-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-22-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-12-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-17-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-18-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-20-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-08-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-22-2009.sql.gz
            ./webbmaster_bellarose_joomla_12-23-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-07-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-26-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-19-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-29-2009.sql.gz
            ./webbmaster_bellarose_joomla_01-15-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-01-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-18-2010.sql.gz
            ./webbmaster_bellarose_joomla_01-02-2010.sql.gz
            ./webbmaster_bellarose_joomla_12-19-2009.sql.gz

(dates 1/23 - 1/29 are left intact)

A: 
OLDIFS="$IFS"
IFS=$'\n'
topurge=($(find -mindepth 1 -printf "%T@ %P\n" | sort -n -r | cut -d' ' -f 2- | tail -n +7))
IFS="$OLDIFS"
Ignacio Vazquez-Abrams
This prints at most 1 file in my test, and it doesn't seem to be the oldest or newest, just one random file from the bunch.
Eddie
That would be because it creates an array. Try `for f in "${topurge[@]}" ; do echo "$f" ; done` instead.
Ignacio Vazquez-Abrams
A: 

Here is one thought:

oldest_to_keep=`find . -name \*.sql | xargs \ls -1rc | tail -7 | head -1`
find . -name \*.sql -not -samefile $oldest_to_keep -not -newer $oldest_to_keep

Where the second find ought to have -exec rm \{\} ; if you've convinced yourself this is sane. :)

ccmonkey
this is listing the 7 most recent in my test. .. but a good start I can work with perhaps/.
Eddie
+1  A: 

If the files are all in one directory (i.e. you do not need to use find to recurse though subdirectories), you can use the -t argument to ls(1) to order the file list by time and then remove the first 7 entries with tail(1).

rm -f $(ls -t *sql* | tail -n +8)

[Edit: I added -f in case the substitution results in an empty list]

or in a loop...

ls -t *sql* | tail -n +8 | while read file
    echo "Purging: $file"
    rm "$file"
done

The argument to tail is +8 because you want it to start tailing from the 8th line onwards (i.e. drop the first 7).

After removing the files, you can just burn what's left.

camh
Yeah again, i want to preserver the most recent 7, not remove. for any give n directory that could mean removing 0, or 63 files.
Eddie
This is preserving the most recent 7. It deletes anything after the most recent 7. The comment about removing the first seven entries is to remove them from the list of files to purge (i.e. you keep them).
camh
A: 

Please do yourself (and everyone else) a big favour and start writing dates in ISO-8601 format (i.e. YYYY-MM-DD).

E.g. instead of

./webbmaster_bellarose_joomla_12-29-2009.sql.gz

you would have

./webbmaster_bellarose_joomla_2009-12-29.sql.gz

Doing so have many benefits, most importantly here that chronological and alphabetical order becomes identical. No need for find, just list the files, reverse order, delete the first 7 lines and delete any remaining files:

ls *sql* | tac | sed 1,7d | tr '\012' '\000' | xargs -0 --no-run-if-empty rm

Update: What do you mean by "moving forward"? As fas as I can tell my solution solves your problem 100%:

/tmp/so>cat existing_files
./webbmaster_bellarose_joomla-2009-12-17.sql.gz
./webbmaster_bellarose_joomla-2009-12-18.sql.gz
./webbmaster_bellarose_joomla-2009-12-19.sql.gz
./webbmaster_bellarose_joomla-2009-12-20.sql.gz
./webbmaster_bellarose_joomla-2009-12-21.sql.gz
./webbmaster_bellarose_joomla-2009-12-22.sql.gz
./webbmaster_bellarose_joomla-2009-12-23.sql.gz
./webbmaster_bellarose_joomla-2009-12-24.sql.gz
./webbmaster_bellarose_joomla-2009-12-25.sql.gz
./webbmaster_bellarose_joomla-2009-12-26.sql.gz
./webbmaster_bellarose_joomla-2009-12-27.sql.gz
./webbmaster_bellarose_joomla-2009-12-28.sql.gz
./webbmaster_bellarose_joomla-2009-12-29.sql.gz
./webbmaster_bellarose_joomla-2009-12-30.sql.gz
./webbmaster_bellarose_joomla-2009-12-31.sql.gz
./webbmaster_bellarose_joomla-2010-01-01.sql.gz
./webbmaster_bellarose_joomla-2010-01-02.sql.gz
./webbmaster_bellarose_joomla-2010-01-03.sql.gz
./webbmaster_bellarose_joomla-2010-01-04.sql.gz
./webbmaster_bellarose_joomla-2010-01-05.sql.gz
./webbmaster_bellarose_joomla-2010-01-06.sql.gz
./webbmaster_bellarose_joomla-2010-01-07.sql.gz
./webbmaster_bellarose_joomla-2010-01-08.sql.gz
./webbmaster_bellarose_joomla-2010-01-09.sql.gz
./webbmaster_bellarose_joomla-2010-01-10.sql.gz
./webbmaster_bellarose_joomla-2010-01-11.sql.gz
./webbmaster_bellarose_joomla-2010-01-12.sql.gz
./webbmaster_bellarose_joomla-2010-01-13.sql.gz
./webbmaster_bellarose_joomla-2010-01-14.sql.gz
./webbmaster_bellarose_joomla-2010-01-15.sql.gz
./webbmaster_bellarose_joomla-2010-01-16.sql.gz
./webbmaster_bellarose_joomla-2010-01-17.sql.gz
./webbmaster_bellarose_joomla-2010-01-18.sql.gz
./webbmaster_bellarose_joomla-2010-01-19.sql.gz
./webbmaster_bellarose_joomla-2010-01-20.sql.gz
./webbmaster_bellarose_joomla-2010-01-21.sql.gz
./webbmaster_bellarose_joomla-2010-01-22.sql.gz
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so>cat targets_for_purging
./webbmaster_bellarose_joomla-2009-12-17.sql.gz
./webbmaster_bellarose_joomla-2009-12-18.sql.gz
./webbmaster_bellarose_joomla-2009-12-19.sql.gz
./webbmaster_bellarose_joomla-2009-12-20.sql.gz
./webbmaster_bellarose_joomla-2009-12-21.sql.gz
./webbmaster_bellarose_joomla-2009-12-22.sql.gz
./webbmaster_bellarose_joomla-2009-12-23.sql.gz
./webbmaster_bellarose_joomla-2009-12-24.sql.gz
./webbmaster_bellarose_joomla-2009-12-25.sql.gz
./webbmaster_bellarose_joomla-2009-12-26.sql.gz
./webbmaster_bellarose_joomla-2009-12-27.sql.gz
./webbmaster_bellarose_joomla-2009-12-28.sql.gz
./webbmaster_bellarose_joomla-2009-12-29.sql.gz
./webbmaster_bellarose_joomla-2009-12-30.sql.gz
./webbmaster_bellarose_joomla-2009-12-31.sql.gz
./webbmaster_bellarose_joomla-2010-01-01.sql.gz
./webbmaster_bellarose_joomla-2010-01-02.sql.gz
./webbmaster_bellarose_joomla-2010-01-03.sql.gz
./webbmaster_bellarose_joomla-2010-01-04.sql.gz
./webbmaster_bellarose_joomla-2010-01-05.sql.gz
./webbmaster_bellarose_joomla-2010-01-06.sql.gz
./webbmaster_bellarose_joomla-2010-01-07.sql.gz
./webbmaster_bellarose_joomla-2010-01-08.sql.gz
./webbmaster_bellarose_joomla-2010-01-09.sql.gz
./webbmaster_bellarose_joomla-2010-01-10.sql.gz
./webbmaster_bellarose_joomla-2010-01-11.sql.gz
./webbmaster_bellarose_joomla-2010-01-12.sql.gz
./webbmaster_bellarose_joomla-2010-01-13.sql.gz
./webbmaster_bellarose_joomla-2010-01-14.sql.gz
./webbmaster_bellarose_joomla-2010-01-15.sql.gz
./webbmaster_bellarose_joomla-2010-01-16.sql.gz
./webbmaster_bellarose_joomla-2010-01-17.sql.gz
./webbmaster_bellarose_joomla-2010-01-18.sql.gz
./webbmaster_bellarose_joomla-2010-01-19.sql.gz
./webbmaster_bellarose_joomla-2010-01-20.sql.gz
./webbmaster_bellarose_joomla-2010-01-21.sql.gz
./webbmaster_bellarose_joomla-2010-01-22.sql.gz
/tmp/so>sed 's/\(.\{29\}\)_\(..-..\)-\(....\)/\1-\3-\2/' existing_files | sort > existing_files.8601
/tmp/so>sed 's/\(.\{29\}\)_\(..-..\)-\(....\)/\1-\3-\2/' targets_for_purging | sort > targets_for_purging.8601
/tmp/so>mkdir files
/tmp/so>cd files
/tmp/so/files>xargs touch < ../existing_files.8601
/tmp/so/files>ls -1
webbmaster_bellarose_joomla-2009-12-17.sql.gz
webbmaster_bellarose_joomla-2009-12-18.sql.gz
webbmaster_bellarose_joomla-2009-12-19.sql.gz
webbmaster_bellarose_joomla-2009-12-20.sql.gz
webbmaster_bellarose_joomla-2009-12-21.sql.gz
webbmaster_bellarose_joomla-2009-12-22.sql.gz
webbmaster_bellarose_joomla-2009-12-23.sql.gz
webbmaster_bellarose_joomla-2009-12-24.sql.gz
webbmaster_bellarose_joomla-2009-12-25.sql.gz
webbmaster_bellarose_joomla-2009-12-26.sql.gz
webbmaster_bellarose_joomla-2009-12-27.sql.gz
webbmaster_bellarose_joomla-2009-12-28.sql.gz
webbmaster_bellarose_joomla-2009-12-29.sql.gz
webbmaster_bellarose_joomla-2009-12-30.sql.gz
webbmaster_bellarose_joomla-2009-12-31.sql.gz
webbmaster_bellarose_joomla-2010-01-01.sql.gz
webbmaster_bellarose_joomla-2010-01-02.sql.gz
webbmaster_bellarose_joomla-2010-01-03.sql.gz
webbmaster_bellarose_joomla-2010-01-04.sql.gz
webbmaster_bellarose_joomla-2010-01-05.sql.gz
webbmaster_bellarose_joomla-2010-01-06.sql.gz
webbmaster_bellarose_joomla-2010-01-07.sql.gz
webbmaster_bellarose_joomla-2010-01-08.sql.gz
webbmaster_bellarose_joomla-2010-01-09.sql.gz
webbmaster_bellarose_joomla-2010-01-10.sql.gz
webbmaster_bellarose_joomla-2010-01-11.sql.gz
webbmaster_bellarose_joomla-2010-01-12.sql.gz
webbmaster_bellarose_joomla-2010-01-13.sql.gz
webbmaster_bellarose_joomla-2010-01-14.sql.gz
webbmaster_bellarose_joomla-2010-01-15.sql.gz
webbmaster_bellarose_joomla-2010-01-16.sql.gz
webbmaster_bellarose_joomla-2010-01-17.sql.gz
webbmaster_bellarose_joomla-2010-01-18.sql.gz
webbmaster_bellarose_joomla-2010-01-19.sql.gz
webbmaster_bellarose_joomla-2010-01-20.sql.gz
webbmaster_bellarose_joomla-2010-01-21.sql.gz
webbmaster_bellarose_joomla-2010-01-22.sql.gz
webbmaster_bellarose_joomla-2010-01-23.sql.gz
webbmaster_bellarose_joomla-2010-01-24.sql.gz
webbmaster_bellarose_joomla-2010-01-25.sql.gz
webbmaster_bellarose_joomla-2010-01-26.sql.gz
webbmaster_bellarose_joomla-2010-01-27.sql.gz
webbmaster_bellarose_joomla-2010-01-28.sql.gz
webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>ls *sql* | tac | sed 1,7d | tr '\012' '\000' | xargs -0 --no-run-if-empty rm
/tmp/so/files>ls -1 ./*
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>comm -3 ../existing_files.8601 ../targets_for_purging.8601
./webbmaster_bellarose_joomla-2010-01-23.sql.gz
./webbmaster_bellarose_joomla-2010-01-24.sql.gz
./webbmaster_bellarose_joomla-2010-01-25.sql.gz
./webbmaster_bellarose_joomla-2010-01-26.sql.gz
./webbmaster_bellarose_joomla-2010-01-27.sql.gz
./webbmaster_bellarose_joomla-2010-01-28.sql.gz
./webbmaster_bellarose_joomla-2010-01-29.sql.gz
/tmp/so/files>
hlovdal
Great tip moving forward.
Eddie