App engine - Uploading a large file and parsing data from it onto the datastore.

Question

I have a file that contains ~16,000 lines of information on entities. The user is supposed to upload the file using an HTML upload form, then the system handles this by reading line by line and creating then put()'ing entities onto the datastore.

I'm limited by the 30 second request time limit. I have tried a lot of different work-arounds using Task Queue, forced HTML redirecting, etc. and nothing has worked for me.

I am using forced HTML redirecting to delete all data and this works, albeit VERY slowly. (4th answer here: http://stackoverflow.com/questions/108822/delete-all-data-for-a-kind-in-google-app-engine)

I can't seem to apply this to my uploading problem, since my method has to be a POST method. Is there a solution somehow? Sample code would be much appreciated since I'm very new to web development in general.

Answer 1

To solve a similar problem, I stored the dataset in a model with a single TextProperty, then spawn a taskqueue task that:

1. Fetches a dataset from the datastore if there are any left.

2. Checks if the length of the dataset is <= N, where N is some small number of entities you can put() without a timeout. I used 5. If so, write the individual entities, delete the dataset record, and spawn a new copy of the task.

3. If the dataset size was bigger than N, split it into N parts in the same format and write those to the datastore, delete the original entity, and spawn a new copy of the task.

Answer 2

If you're doing this to bulk load data, why not use the bulk loader?

If you need the interface to be accessible to non-admin users, then, as suggested, you need to break the file up into decent sized chunks (by taking blocks of n lines each) put them into the datastore, and start a task to deal with each of them.