How to add more than 500 entries to the datastore with put() in google app engine?

Question

I tried adding batches of data in a list with a couple of calls to db.put(). But it still timeouts occasionally.

Anyone have some tips?

Answer 1

Use multiple puts, and use smaller batch sizes (halving them, for example) if you encounter timeouts. If you're adding 500+ entities in one user request, though, you're probably doing something wrong, though.

Answer 2

The bulkload client that Google provides with App Engine (bulkload_client source) defaults batch size to 10, so that tells me that you have to very careful about the size of your batches.

I was recently deleting several thousand objects and found that the most I could pass to db.delete() was around 400. But, sometimes that would timeout and I'd go down to 350. I'm not sure if deletes or puts are more expensive though.