App Engine query.fetch() timing out on large database?

Question

def get(self):
    links = Link.all().order("author")
    response = ""
    links.fetch(10)
    for link in links:
        response += "[link][/link]"

For some reason, that code is taking up an obnoxious amount of CPU, and is eventually just timing out, when run on a database of over 8,000 entries. Isn't App Engine supposed to be able to handle large datasets? Am I doing something stupid?

Answer 1

This section:

links.fetch(10)
for link in links:
    response += "[link][/link]"

is incorrect. .fetch() returns the fetched records, but you're not assigning the returned records to anything. Then, you're iterating over the query object, which causes App Engine to return all 8000 results in batches of 20, which is a lot less efficient than a single bulk fetch.

If you only need 10 results, you should do this:

for link in links.fetch(10):
    response += "[link][/link]"

If you need all 8000 results, well, you're probably better off reconsidering if your users can actually handle 8000 results in one page.