[Stackless] micro-threads & co-routines

Christian Tismer tismer at appliedbiometrics.com
Fri Oct 13 18:45:05 CEST 2000

Gordon McMillan wrote:
> Just wrote:
> > At 11:36 AM +0200 12-10-2000, Dirk-Ulrich Heise wrote:
> [snip]
> > >A completely different question: All these concepts by
> > >Christian seem to be very complex. Did somebody
> > >consider modeling them using Python, that is, create a sort of
> > >prototype engine that would allow testing them before the C
> > >coding starts? Just asking for curiosity.

According to Gordon's comment below: This would be simple,
after rewriting the whole Python interpreter in Python (and
the latter is not so simple:)

> > There needs to be _some_ C-level support for restartable frames
> > (a.k.a. continuations. Well, sortof.) to be able to experiment
> > with this. But maybe that support can be extremely minimal at
> > first (maybe even so bare that it's possible to crash Python if
> > not used properly...). On the other hand, Christian is a speed
> > freak, and if his continuations can't be as fast as can be, he
> > won't have as much fun with them ;-)

The speed thang is true. But there is something hard to
understand, between implementation details, interface
design and continuations at all:
You may throw away most of the "magic" that my continuations
can do at the moment. Just thinks that would reduce the
complexity of the implementation. I would be happy if this
were true. Basically, it is not.
I can drop co.update, co.call, assignment to the caller,
but that does not make the implementation any simpler.

Just try it: Remove almost all of the coObject's properties
and methods. Then you can see: What remains is a system
that desperately tries to keep an invisible ownership property
alive. It does so by checking the number of referers to a frame,
and it tries to make a copy of a frame at the very last possible
moment: When ceval wants to run a frame, it checks whether it
owns that frame and takes the necessary actions.

Today this is done via callback functions.
Things can be made clearer by coding this directly
into the core, without callbacks. But the complexity is
there, and if you want continuations, then you have it.

The real problem with continuations is this: Basically,
with continuations, the *callee* must be the owner of
the continuation. But actually, with standard and also
Stackless Python, the caller owns it. Stackless repairs that
by creating the necessary copy at the last possible moment,
in order to avoid most of the copies.

In a true, continuation driven world, every function call would
mean to pass a newly created continuation object with all
state info. This is possible, but this imposes an extra
effort which would affect every Python program's performance.
And as Just said, I'm a speed maniac, and I'm unable to write
such a program. Unless I'm convinced that this can be made
as efficient as Python is today, my fingers are bending away.

Therefore, I believe the complexity of creating continuations
only if they are needed is necessary. Well, moving over to the
frame split idea *might* be this difference, and if its cost is
small enough, it may be the right way. But I have to think
it though completely.

> > For a while now I've been trying to convince Christian to make
> > continuations much more primitive -- less options, simpler
> > semantics -- but I'm not sure how much he is willing to give up.
> > Everything that the current continuations can do, should be
> > modelable (is that a word?) with even simpler building blocks.

I agree that the UI might need an improvement, also that
the number of methods may be too big, and I do dislike
the existence of flags, of course. That has evolved through
the time where I try to understand continuations, and today
I think this can be done in a better way.
But as said: This is not the real problem. Remove the coObject
implementation completely, and look what remains: A set of callback
functions that perform a difficult task.
My current goal is to find an equally efficient layout which
is easier to implement, yields the same power, and can be
incorporated into the core as simply as possible.

The continuation interface can then be implemented on top of
that, in the same or a different flavor. What I just say is
that we must get the core machinery right.

This will never be a simple task, whatever we do. We can make
it look nicer, but not substantially simpler. Howgh.

> Actually, I don't really think that's true. This isn't like thread.py
> vs. threading.py. In that case the low-level module exposes
> things that exist in the OS, and the higher makes it sane. But
> stackless / continuation isn't exposing "something else" to
> Python, it's manipulating Python's guts so the language itself
> works differently. Doing that with Python would be like tying
> your shoes with boxing gloves on.

I totally agree.

thank you all for taking care of this difficult issue - chris

Christian Tismer             :^)   <mailto:tismer at appliedbiometrics.com>
Applied Biometrics GmbH      :     Have a break! Take a ride on Python's
Kaunstr. 26                  :    *Starship* http://starship.python.net
14163 Berlin                 :     PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint       E182 71C7 1A9D 66E9 9D15  D3CC D4D7 93E2 1FAE F6DF
     where do you want to jump today?   http://www.stackless.com
Stackless mailing list
Stackless at starship.python.net

More information about the Stackless mailing list