Strategy for handling user input as files

Question

I'm creating a script to process files provided to us by our users. Everything happens within the same UNIX system (running on Solaris 10)

Right now our design is this

1. User places file into upload directory
2. Script placed on cron to run every 10 minutes.
3. Script looks for files in upload directory, processes them, deletes immediately afterward

For historical/legacy reasons, #1 can't change. Also, deleting the file after processing is a requirement.

My primary concern is concurrency. It is very likely that the situation will arise where the analysis script runs while an input file is still being written to. In this case, data will be lost and this (obviously) unacceptable.

Since we have no control over the user's chosen means of placing the input file, we cannot require them to obtain a file lock. As I understand, file locks are advisory only on UNIX. Therefore a user must choose to adhere to them.

I am looking for advice on best practices for handling this problem. Thanks

Answer 1

Obviously all the best solutions involve the client providing some kind of trigger indicating that it has finished uploading. That could be a second file, an atomic move of the file to a processing directory after writing it to a stage directory, or a REST web service. I will assume you have no control over your clients and are unable or unwilling to change anything about them.

In that case, you still have a few options:

• You can use a pretty simple heuristic: check the file size, wait 5 seconds, check the file size. If it didn't change, it's probably good to go.

• If you have super-user privileges, you can use lsof to determine if anyone has this file open for writing.

• If you have access to the thing that handles upload (HTTP, FTP, a setuid script that copies files?) you can put triggers in there of course.