Py3k memory conservation by returning iterators rather than lists

Question

Many methods that used to return lists in Python 2.x now seem to return iterators in Py3k

Are iterators also generator expressions? Lazy evaluation?

Thus, with this the memory footprint of python is going to reduce drastically. Isn't it?

What about for the programs converted from 2to3 using the builtin script?

Does the builtin tool explicitly convert all the returned iterators into lists, for compatibility? If so then the lower memory footprint benefit of Py3k is not really apparent in the converted programs. Is it?

Answer 1

Many of them are not exactly iterators, but special view objects. For instance range() now returns something similar to the old xrange object - it can still be indexed, but lazily constructs the integers as needed.

Similarly dict.keys() gives a dict_keys object implementing a view on the dict, rather than creating a new list with a copy of the keys.

How this affects memory footprints probably depends on the program. Certainly there's more of an emphasis towards using iterators unless you really need lists, whereas using lists was generally the default case in python2. That will cause the average program to probably be more memory efficient. Cases where there are really big savings are probably going to already be implemented as iterators in python2 programs however, as really large memory usage will stand out, and is more likely to be already addressed. (eg. the file iterator is already much more memory efficient than the older file.readlines() method)

Converting is done by the 2to3 tool, and will generally convert things like range() to iterators where it can safely determine a real list isn't needed, so code like:

for x in range(10): print x

will switch to the new range() object, no longer creating a list, and so will obtain the reduced memory benefit, but code like:

x = range(20)

will be converted as:

x = list(range(20))

as the converter can't know if the code expects a real list object in x.

Answer 2

Are iterators also generator expressions? Lazy evaluation?

An iterator is just an object with a next method. What the documentation means most of the time when saying that a function returns an iterator is that its result is lazily loaded.

Thus, with this the memory footprint of python is going to reduce drastically. Isn't it?

It depends. I'd guess that the average program wouldn't notice a huge difference though. The performance advantages of iterators over lists is really only significant if you have a large dataset. You may want to see this question.