views:

210

answers:

4

I have a Python test suite that creates and deletes many temporary files. Under Windows 7, the shutil.rmtree operations sometimes fail (<1% of the time). The failure is apparently random, not always on the same files, not always in the same way, but it's always on rmtree operations. It seems to be some kind of timing issue. It is also reminiscent of Windows 7's increased vigilance about permissions and administrator rights, but there are no permission issues here (since the code had just created the files), and there are no administrator rights in the mix.

It also looks like a timing issue between two threads or processes, but there is no concurrency here either.

Two examples of (partial) stack traces:

  File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
    shutil.rmtree(cleandir)
  File "c:\python23\lib\shutil.py", line 142, in rmtree
    raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 5] Access is denied removing xml_1

  File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
    shutil.rmtree(cleandir)
  File "c:\python23\lib\shutil.py", line 142, in rmtree
    raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 3] The system cannot find the path specified removing out

On Windows XP, it never failed. On Windows 7, it fails like this, across a few different Python versions (2.3-2.6, not sure about 3.1).

Anyone seen anything like this and have a solution? The code itself is on bitbucket for the truly industrious.

+1  A: 

My guess is that you should check up on the code that creates the file, and make SURE they are closed explicitly before moving on to delete it. If nothing is obvious there in the code, download a copy of Process Explorer and watch what's happening on the file system there. This tool will give you the exact error code coming from Windows, and should shed some light on the situation.

Dave Markle
That's a good idea, I'll look into it, but how would it explain the "cannot find the path specified" errors? Those seem to be rmtree trying to remove the tree *after* it had already been deleted?
Ned Batchelder
+2  A: 

We had similar problems with shutil.rmtree on Windows, particularly looking like your first stack trace. We solved it by using an exception handler with rmtree. See this answer for details.

ThomasH
Oddly, if I simply provide an onerror function that retries the operations, the problem seems to stop.
Ned Batchelder
Which might support your idea of a timing issue.
ThomasH
+1  A: 

It's a long shot, but are you running anything that scans directories in the background? I'm thinking antivirus/backup (maybe Windows 7 has something like that built in? I don't know). I have experienced occasional glitches when deleting/moving files from the TSVNCache.exe process that TortoiseSVN starts -- seems it watches directories for changes, then presumably opens them for scanning the files.

j_random_hacker
I like this as a theory of why it's happening.
Ned Batchelder
In a comment on his post, Ryan Ginstrom mentions search indexing -- that sounds like another likely candidate to me.
j_random_hacker
+1  A: 

Just a thought, but if the test behavior (creating and deleting lots of temp files) isn't typical of what the app actually does, maybe you could move those test file operations to (c)StringIO, and keep a suite of function tests that exercises your app's actual file creation/deletion behavior.

That way, you can make sure that your app behaves correctly, without introducing extra complexity not related to the app.

Ryan Ginstrom
This is exactly the right thinking for improving the testability of code. In my case, the files and directories are important to the test, though.
Ned Batchelder
Right. In that case, make sure you turn off search indexing, "shadow copy," etc. I've also found Windows 7 to be very picky about deleting files from under other processes.
Ryan Ginstrom