Google App Engine: "Cannot create a file when that file already exists"

Question

I'm running the Google App Engine devserver 1.3.3 on Windows 7.

Usually, this method works fine, but this time it gave an error:

def _deleteType(type):
    results = type.all().fetch(1000)
    while results:
        db.delete(results)
        results = type.all().fetch(1000)

The error:

  File "src\modelutils.py", line 38, in _deleteType
    db.delete(results)
  File "C:\Program Files\Google\google_appengine\google\appengine\ext\db\__init__.py", line 1302, in delete
    datastore.Delete(keys, rpc=rpc)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.py", line 386, in Delete
    'datastore_v3', 'Delete', req, datastore_pb.DeleteResponse(), rpc)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.py", line 186, in _MakeSyncCall
    rpc.check_success()
  File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_stub_map.py", line 474, in check_success
    self.__rpc.CheckSuccess()
  File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_rpc.py", line 149, in _WaitImpl
    self.request, self.response)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 667, in MakeSyncCall
    response)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\apiproxy_stub.py", line 80, in MakeSyncCall
    method(request, response)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 775, in _Dynamic_Delete
    self.__WriteDatastore()
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 610, in __WriteDatastore
    self.__WritePickled(encoded, self.__datastore_file)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_file_stub.py", line 656, in __WritePickled
    os.rename(tmpfile.name, filename)
WindowsError: [Error 183] Cannot create a file when that file already exists

What am I doing wrong? How could this have failed this time, but usually it doesn't?

UPDATE I restarted the devserver, and when it came back online, the datastore was empty.

Answer 1

Unfortunately, 1.3.3 is too far back for me to look at its sources and try to diagnose your problem precisely - the SDK has no 1.3.3 release tag and I can't guess which revision of the datastore_filestub.py was in 1.3.3. Can you upgrade to the current version, 1.3.5, and try again? Running old versions (especially 2+ versions back) is not recommended since they'll be possibly a little out of sync with what's actually available on Google's actual servers, anyway (and/or have bugs that are fixed in later versions). Anyway...

On Windows, os.rename doesn't work if the destination exists -- but the revisions I see are careful to catch the OSError that results (WindowsError derives from it), remove the existing file, and try renaming again. So I don't know what could explain your bug -- if the sources of the SDK you're running have that careful arrangement, and I think they do.

Plus, I'd recommend to --use_sqlite (see Nick Johnson's blog announcing it here) in lieu of the file-stub for your SDK datastore - it just seems to make more sense!-)

Answer 2

(disclaimer: i'm not answering your question but helping you optimize the code you're running)

your code seems to be massively deleting objects. in the SDK/dev server, you can accomplish wiping out the datastore using this command as a quicker and more convenient alternative:

$ dev_appserver.py -c helloworld

now, that is, if you want to wipe your entire SDK datastore. if not, then of course, don't use it. :-)

more importantly, you can make your code run faster and use less CPU on production if you change your query to be something like:

results = type.all(keys_only=True).fetch(SIZE)

this works the same as your's except it only fetches the keys as you don't need full entities retrieved from the datastore in order to delete them. also, your code is currently setting SIZE=1000, but you can make it larger than that, esp. if you have an idea of how many entities you have in your system... the 1000 result limit was lifted in 1.3.1 http://bit.ly/ahoLQp

one minor nit... try not to use type as a variable name... that's one of the most import objects and built-in/factory functions in Python. your code may act odd if do this -- in your case, it's only fractionally better since you're inside a function/method, but that's not going to be true as a global variable.

hope this helps!