Solr paging performance

Question

I have read (http://old.nabble.com/using-q%3D--,-adding-fq%3D-to26753938.html#a26805204):

FWIW: limiting the number of rows per request to 50, but not limiting the start doesn't make much sense -- the same amount of work is needed to handle start=0&rows=5050 and start=5000&rows=50.

Than he completes:

There are very few use cases for allowing people to iterate through all the rows that also require sorting.

Is that right? Is that true just for sorted results?

How many pages of 10 rows each do you recommend to allow the user to iterate?

Does Solr 1.4 suffer the same limitation?

Answer 1

Yes that's true, also for Solr 1.4. That does not mean that start=0&rows=5050 has the same performance as start=5000&rows=50, since the former has to return 5050 documents while the latter only 50. Less data to transfer -> faster.

Solr doesn't have any way to get ALL results in a single page since it doesn't make much sense. As a comparison, you can't fetch the whole Google index in a single query. Nobody really needs to do that.

The page size of your application should be user-definable (i.e. the user might choose to see 10, 25, 50, or 100 results at once).

The default page size depends on what kind of data you're paging and how relevant the results really are. For example, when searching on Google you usually don't look beyond the first few results, so 10 elements are enough. eBay, on the other hand, is more about browsing the results, so it shows 50 results per page by default, and it doesn't even offer 10 results per page.

You also have to take scrolling into account. Users would probably get lost when trying to browse through a 200-result page, not to mention that it takes considerably longer to load.

Answer 2

start=0&rows=5050 and start=5000&rows=50

Depends how you jump to start=5000. If you scroll through all results from 0 to 4999 ignoring them all and then continue scrolling from 5000 to 5050 then yes, same amount of work is done here. Best thing to do is to limit the rows fetched from database itself by using something like ROWNUM in Oracle.

.

iterate through all the rows that also require sorting

Few but yes there are use cases that have this requirement. Examples would be CSV/Excel/PDF exports.