using task queues to schedule the fetching/parsing of a number of feeds in appengine python

Question

Say I had over 10,000 feeds that I wanted to periodically fetch/parse. If the period were say 1h that would be 24x10000 = 240,000 fetches.

The current 10k limit of the labs taskqueue api would preclude one from setting up one task per fetch. How then would one do this?

Update: re: fetching nurls per task: given the 30second timeout per request at some point this would hit a ceiling. Is there anyway to paralellize it so each tasqueue initiates a bunch of async paralell fetches each of which would take less than 30sec to finish but the lot togethere may take more than that

Answer 1

2 fetches per task? 3?

Answer 2

Group up the fetches, so instead of queuing 1 fetch you queue up, say, a work unit that does 10 fetches.

Answer 3

Here's the asynchronous urlfetch API:

http://code.google.com/appengine/docs/python/urlfetch/asynchronousrequests.html

Set of a bunch of requests with a reasonable deadline (give yourself some headroom under your timeout, so that if one request times out you still have time to process the others). Then wait on each one in turn and process as they complete.

I haven't used this technique myself in GAE, so you're on your own finding any non-obvious gotchas. Sadly there doesn't seem to be a select() style call in the API to wait for the first of several requests to complete.