tags:

views:

67

answers:

1

How are greenlets implemented? Python uses the C stack for the interpreter and it heap-allocates Python stack frames, but beyond that, how does it allocate/swap stacks, how does it hook into the interpreter and function call mechanisms, and how does this interact with C extensions? (Any quirks)?

There are some comments at the top of greenlet.c in the source, but they're a bit opaque.

(One data point: they don't use ucontext.h.)

+1  A: 

If get and study the greenlet's sources, you'll see at the top of greenlet.c a long comment that starts at line 16 with the following summary...:

A PyGreenlet is a range of C stack addresses that must be saved and restored in such a way that the full range of the stack contains valid data when we switch to it.

and continues to line 82, summarizing exactly what you're asking about. Have you studies these lines (and the following 1000+ implementing them;-)...? I don't see a way to further squeeze these 66 lines down while still making sense, nor any added value in copying and pasting them here.

Basically, you'll see there is no real "hooking" to speak of (the C level stack is switched back and forth "under the interpreter's nose", so to speak) except for the delicate interactions with thread state in multi-threaded code, and the saving and restoring of a greenlet's state from/to the stack is based on memcpy calls plus some calls to the Python memory manager to allocate/reallocate and free space coming from, or going back to, the stack. The three functions in line 227-295 handle the grunt work, and they're wrapped in a couple C macros at 298-310 "in order to simplify maintenance", as the comment there says.

The interface through which other C extensions can interact with the greenlet extension is implemented at lines 956-1045, and exposed through the "CObject API" (via greenlet.h, of course) documented here.

Alex Martelli
That comment block is confusing to me, and doesn't really answer my questions. I was just hoping for a concise, high-level summary/answer. Thanks for the pointers anyway - hope they're useful to others (or myself when I find more time to source-dive).
Yang
@Yang, those 86 lines **are** a concise, high-level summary -- hitting most of the highlights of the 1410 lines of code that together make up the `.c` and `.h` files! "Stack slices are saved by memcpy to memory that's allocated and reallocated by the Python memory manager, and restored by memcpy back into the stack (then the Python memory is freed)" is even more concise and higher level (but I did already say all this in my answer!) but obviously missing some important details (since it's two lines of text, not 86;-). What magic text do you expect to fall in-between and make you happy?!
Alex Martelli
For starters: what's "greenlet stack data"? Is that just bookkeeping for the greenlet? Or does it actually include certain C stack frames? What's a greenlet's "correct place in the stack"? Why are there always two greenlet blocks/why's the older one on heap? What's "data unrelated to this greenlet" below "greenlet stack data"? Diff btwn "unrelated data" and "newer data"? Etc. It's a small amount of C, but I'm also busy and this is not at all related to my current work - just asking out of curiosity. The question just popped into my head. Again, happy to source-dive later once I find time.
Yang
@Yang, the stack data of a greenlet is everything that's on the stack due to code executing in the greenlet, thus it most certainly includes stack frames (not sure what you think distinguishes a "C" stack frame from one from another language? C, Assembly, Fortran, whatever, they're stack frames). The correct place is exactly where the stack data was originally (since pointers into it are always involved it could not be usefully reloaded elsewhere). The small greenlet block that's always on the stack keeps that place. And this obvious info exhausts the space available in a comment, so, 'bye.
Alex Martelli