How can I learn more about Python’s internals?

Question

I have been programming using Python for slightly more than half an year now and I am more interested in Python internals rather than using Python to develop applications. Currently I am working on porting a few libraries from Python2 to Python3. However, I have a rather abstract view on how to make port stuff over from Python2 to Python3 as most of the changes deal with design issues in Python2.x

I'd like to learn more about Python internals; should I go for a top-down or a bottom-up approach? Are there any references you could recommend?

Answer 1

Have you tried this?

• Automated Python 2 to 3 code translation

Answer 2

I would first read the What's New document for Python 3. It gives a good high-level overview and touches on the detailed changes.

You might also do a search for 'porting to python 3' or similar. There are lots of good resources and tools.

One tool that's new and hard to find is six, by Benjamin Peterson. It enables writing of code that is compatible across the Python 2*3 gap.

The part I found most difficult about maintaining Python 2 and Python 3 -compatible code was deployment. I could write code that would run just fine, but when I went do package and deploy, it was unclear when the conversion should happen. I ultimately found a distutils command build_py_2_to_3 that would do the trick. By using that command in my setup.py, I could release a source distribution that would deploy on either Python 2 or Python 3. An example can be found in jaraco.util.

You also asked about the internals. If you really want to get at the internals, you can view the source for Python 2.x and Python 3.x, though honestly, I would stick with reading the tutorials and maybe some of the .py files in the Python libs.

Answer 3

It sounds like you want to know more about the rationale behind the design of the language, rather than internals. "internals" to me means things like how objects are laid out in memory, how reference counting works, and so on.

If you're looking for a deeper understanding of the design decisions, try reading the PEPs: they are the proposals for changes in the language, and often include detailed discussions of the reasons for the changes, rejected alternatives, and so on. Even the rejected PEPs are useful, because they show the thinking that has shaped the language.

For example:

• 3105: Making print a function
• 3110: Catching exceptions in Python 3.x
• 3131: Supporting non-ASCII identifiers

and so on..

If you really want to learn about Python internals, then start by reading about the Python C API, which is used to build Python itself: my talk A Whirlwind Excursion through Python C Extensions is one place to start. Then you can dive into the Python source code itself for anything you need to learn about.

Answer 4

should I go for a top-down or a bottom-up approach?

Both! Seriously.