views:

1618

answers:

4

I'm trying to do what I think is a simple thing under Linux. I have a bash script which runs various test programs, and I want to determine which files in the current directory were created by the test programs. So I am doing something like this:

touch timestamp-file
run the test
find -newer timestamp-file -type f > list-of-files
rm -f timestamp-file

Turns out the granularity of find -newer is poor, so what typically happens is that some files which were generated by the test program show up as OLDER than the timestamp file. So then I tried this:

ls -tr1 | sed '1,/timestamp-file/d'

to generate the same list. This usually works, but not always. I still end up with the situation where files which were generated by the test show up as older than the timestamp file.

Thanks!

P.S. I can accomplish this another way by taking two snapshots of the directory, one before the test program runs, and then one after, and comparing them. Any files in the second list which are not in the first must have been created by the test program (I'm not concerned with background jobs or other users writing to the dir). But this method is not what I want because if an output file wasn't removed prior to running the test (they're supposed to be, but in some cases they may not be), this method will say it was not created by the test program because it was in the dir before the test program was run.

+3  A: 

You can actually use touch to force the timestamps of all current files in the directory to far in the past, such as:

touch -t 200801010000.00 *

If you do this before running your test, there should be more than enough difference in the time for "find -newer" to pick it up. If the granularity was two minutes, you could set all current files to ten minutes ago, the timestamp file to 5 minutes ago, then run your test.

So your script becomes:

touch -t (current time - 10 minutes) *
touch -t (current time -  5 minutes) timestamp-file
run the test
find -newer timestamp-file -type f > list-of-files
rm -f timestamp-file

Assuming you have a decent install of Perl, you can do the following to get 5 minutes ago (or use -600 for 10 minutes) in the correct format for "date -t":

use Date::Manip;
print UnixDate(DateCalc(ParseDateString("now"),"-300"),"%Y%m%d%H%M.%S") . "\n";

If, for some reason, you're not permitted to change the timestamps, use:

sleep 300
touch timestamp-file
sleep 300
run the test
find -newer timestamp-file -type f > list-of-files
rm -f timestamp-file

which has the same effect, but gives you ten minutes to go have a coffee (or your poison of choice if you're not a coffee drinker).

paxdiablo
Actually, just putting usleep 500000 after the touch works, but it seems clumsy. The whole suite takes a long time, so adding in these kinds of delays is problematic. Most tests take 10-60 sec. to run, so there is no race condition. But a test which takes a fraction of a second causes problems.
Dave Wade-Stein
Your first solution is compelling. There is no real problem changing the timestamps, so I could set them as you've described. Too bad touch -t doesn't take "seconds since 1970" as an arg. (=
Dave Wade-Stein
@Dave, assuming you have Perl installed, the Date::Manip stuff from CPAN should do the trick of converting UNIX timestamps to and from strings quite easily.
paxdiablo
A: 

Why not create a temporary directory in which to run the tests? Use a timestamp-based name on the directory to help keep track of which results happened when, and simply delete the whole directory when finished.

Shannon Nelson
Thanks...this is a reasonable suggestion. The problem is that there are several expectations of what will files will be found in the dir, and they can be different per test. So I don't know a priori which input files I would need to copy to the temp dir. I'll think about this some more...
Dave Wade-Stein
timestamp granularity is notoriously none standard. Just:mkdir tmp(cd tmp#reference files in .. rather than .)rm -Rf tmp
pixelbeat
+1  A: 
Johannes Schaub - litb
This is awesome. I had thought the solution was to combine my two methods, but I would have used the clumsy 'ls -l' and I completely forgot about comm (I've never been that familiar with it, but I am now). THANKS!
Dave Wade-Stein
Turns out find doesn't work this way on Darwin (Mac OS), so I am back to using ls -l because I need it work on many platforms, but your solution is still the best for me. Thanks!
Dave Wade-Stein
Dave Wade-Stein, you can use stat instead of find too. find -print0 | xargs -0 stat -c "%n %Y" . maybe that's available?
Johannes Schaub - litb
+1  A: 

If you consider how find(1) is implemented it will be clear why this sometimes might not work as you expect. Here's a hint:

  $ touch timestamp ; touch newer ; find . -newer timestamp 
  $ rm timestamp newer
  $ touch timestamp ; sleep 1 ; touch newer ; find . -newer timestamp
  .
  ./newer
  $

find(1) obtains the file mtime/ctime/atime values using the system call stat(2). Here are the elements of struct stat from <sys/stat.h> (Linux):

  time_t    st_atime;   /* time of last access */
  time_t    st_mtime;   /* time of last modification */
  time_t    st_ctime;   /* time of last status change */

On Linux (and unices generally) time_t is an integer representing "seconds from the start of 1970". Therefore the finest granuality that -newer can comprehend is just one second.

Martin Carpenter