Transaction collision for sequential insert on Google App Engine. Why?

Question

I am inserting a set of records on Google App Engine. I insert them in batch to avoid deadline exceptions.

When there is a large number of records (for example 1k) I always receive an unexpected:

Transaction collision for entity group with key

datastore_types.Key.from_path(u'GroupModel', u'root', _app=u'streamtomail'). Retrying...

This situation happen always.

In local environment instead it works without any problem.

How is it possible to have transaction collisions if I am using a sequential process and no one is using the system in the meanwhile?

Here is the code that I use for batching:

def deferred_worker():
 if next_chunk():
   process_chunk()
   deferred.defer(deferred_worker)

where in *process_chunk()* I do 50 inserts in the database

Answer 1

The collision is on an instance of your 'GroupModel' entity with the key name 'root'. Based on that, I'm guessing you're putting everything inside a single entity group with that as the parent. As documented here, every entity with the same parent is in the same entity group, to which transactions are serialized. Thus, any concurrent updates to any entity in that group will potentially conflict with any other.