Query.fetch(limit=2000) only moves cursor forward by 1000 entities?

Question

Let's say I have 2500 MyModel entities in my datastore, and I run this code:

query = MyModel.all()
first_batch = query.fetch(2000)
len(first_batch) # 2000

next_query = MyModel.all().with_cursor(query.cursor())
next_batch = next_query.fetch(2000)

What do you think len(next_batch) is? 500, right? Nope - it's 1500. Apparently the query cursor never moves forward by more than 1000, even when the query itself returns more than 1000 entities.

Should I do something different or is it just an App Engine bug?

Answer 1

It's an App Engine explicit limit, no more than 1000 in a fetch offset. Which kinda implies that Qql can't retrieve more than 1000 which I thought I saw documented but cannot find.

Oddly, Query.count() has a maximum value of 1000, also. They don't come out and say it, but queries returning more than 1000 entities seems antithetical to the GQL model.

Answer 2

Here's another Stack Overflow question with more info.