GAE uploading to database - or - how to create big dictionary in GAE?

Question

Hi,

I have a GAE application. A new request is to have a huge database to store many items for some functionality.

The database needs to populated every week or so - and populating it is done by querying many external links which overall takes around 1-2 hours. So, doing it via GAE Cron is now out of the question.

Another option I was thinking was to create it in external server and then uploading it to GAE DB. Is it possible to upload the data created externally to GAE DB straightaway? The question is because there will be close to 10 M items in it, so adding items individually to a DB is not going to work due to 30 second time limit.

Thanks

K

Answer 1

It might be possible to use the bulk loader for your task. Or simply create your own method that you talk to from your external cron script for populating the new data

Answer 2

You can use task queues to poll your external sites and load the data.