cx_Oracle - what is the best way to iterate over a result set?

Question

There are several ways, what is the best one?

Answer 1

There are three ways to iterate over a result set. The best way in terms of both readability and performance is usually to use the built-in cursor iterator.

curs.execute('select * from people')
for row in curs:
    print row

You can fetch all the rows into a list, but this can have some bad side effects if the result set is large.

• You have to wait for the entire result set to be returned to your client process.

• You may eat up a lot of memory in your client to hold the built-up list.

• It may take a while for Python to construct and deconstruct the list which you are going to immediately discard anyways.

for row in curs.fetchall():
    print row

Finally, you can loop over the result set fetching one row at a time. In general, there's no particular advantage in doing this over using the iterator. If there is something in your programming logic that seems to indicate there is an advantage in doing this, perhaps you should reconsider your programming logic.

row = curs.fetchone()
while row:
    print row
    row = curs.fetchone()

Answer 2

There's also the way psyco-pg seems to do it... From what I gather, it seems to create dictionary-like row-proxies to map key lookup into the memory block returned by the query. In that case, fetching the whole answer and working with a similar proxy-factory over the rows seems like useful idea. Come to think of it though, it feels more like Lua than Python.

Also, this should be applicable to all PEP-249 DBAPI2.0 interfaces, not just Oracle, or did you mean just fastest using Oracle?

Answer 3

My preferred way is the cursor iterator, but setting first the arraysize property of the cursor.

curs.execute('select * from people')
curs.arraysize = 256
for row in curs:
    print row

In this example, cx_Oracle will fetch rows from Oracle 256 rows at a time, reducing the number of network round trips that need to be performed