[Stackless] What goes on behind the scenes?
richard.m.tew at gmail.com
Fri Jun 19 23:58:46 CEST 2009
On Fri, Jun 19, 2009 at 9:27 PM, Horace Blegg<tkjthingone at gmail.com> wrote:
> Anyway! I'm currently slogging through the source code for stackless as time
> allows (work + school = no free time), and finding it rather interesting. I
Sounds like a good start :-)
> was wondering if there was another resource other the comments in the code
> and my own knowledge (not excellent) of C that would help me follow
> stackless's source? (other than lots of time + sheer brainpower, which is my
> current approach)
I am not sure there is any current and directly relevant documentation.
> I mainly interested in what happens behind the scenes when someone makes a
> tasklet (or tasklets), what happens while it's running, and how the
> interpreter/(scheduler?) keeps track of/handles tasklets (if at all?), etc,
> for an academic/pet project. Expanding my horizons, for lack of a better
The core aspect common to all of these three things is Stackless'
stack saving. No longer based on continuations, the basic method used
to block a running tasklet is to take the area of stack used by it and
save it somewhere. Correspondingly, to resume a tasklet, the saved
area of stack it has used is copied back onto the stack.
In assembly level execution, when a function in calls another,
typically the program counter and other registers/variables are saved
onto the stack. In order for the CPU to know where to return when the
second function exits, it pops these register values off the stack and
through normal operation returns to the first calling function.
So let's say the second function is one that blocks a calling tasklet.
It can then save out all the area of stack from a known starting
point early on in interpreter startup up to where it is now, and this
will include the register values which tell it how to return and
continue executing that calling tasklet function when that area of
stack is later restored. Just like the standard way a function call
from one function into a second returns.
This is called hard switching. If you've heard of greenlets, you can
see that the hard switching has been extracted into a standalone
extension module there.
There is also soft switching, where lower on the call stack a
dispatcher function waits for an "unwind token" to be returned that
holds a reference to a Python execution frame. I'm not going to go
into a lot of detail about how it works, as I don't have much more
time to work on this email. Soft switching is supposed to be a lot
faster than hard switching. The code behind the Stackless fork
includes minimal changes to the Python interpreter itself, but almost
all of those minimal changes are to support soft switching.
The stackless module has a chain of tasklets _per Python thread_.
This is the circular list of tasklets, where one within it is the
tasklet currently being executed (the "current" tasklet). The thread
of execution running when the interpreter started up is a special
tasklet called the "main" tasklet. You can run a scheduler on each
thread. Channels have special code to allow communication between
tasklets running on different schedulers in different threads.
Between channels, tasklets, blocking (using soft and hard switching)
and the scheduler, this is the core part of the Stackless source code.
There is of course also the pickling of running code, but that can be
ignored for the purposes of understanding Stackless for the most part.
More information about the Stackless