[Stackless] rough PyCon'07 draft; feedback wanted

Richard Tew richard.m.tew at gmail.com
Tue Feb 13 12:31:05 CET 2007


On 2/13/07, Andrew Dalke <dalke at dalkescientific.com> wrote:
>    - what should I stress / deemphasize?  Are the examples
> interesting to people (eg, XML processing, format identification,
> handling blocking function calls and I/O)?

The most interesting one to me is the IO one as that one really shows
the benefits of using Stackless.  I don't see it for the other examples
but that would be because of my own lack of interest in those
activities.  Perhaps because of that it feels so much stronger than
the other examples to me.

>    - if people build a stackless binary do they usually call
>       that binary "python", "stackless", "spython" or something else?

I use it on Windows so can't help with this.  But I would expect
that most people isolate it out of fear because it is some arcane
fork and may clash with their distributions installation.

> The code still busy waits, and I've added a new problem.  The
> sleep_scheduler never finishes so "run" never finishes.  My code no
> longer exits.
>
> In talking things over with people on the Stackless mailing list, I
> think the problem is that "run" isn't quite smart enough.  As I showed
> earlier it doesn't know about threads, and here it doesn't know that
> some tasklets shouldn't be counted as running.  In looking through the
> archives the "run" function was once known as "run_watchdog".  It
> isn't really meant to be the main loop of your program.
>
> The recommendation is that you should implement your own main loop for
> non-trivial tasks in Stackless.  I'll develop variations of that
> through the rest of this talk.
>
> I'll call is "run_all" to make it distinct from the Stackless version.
> The simplest implementation of "run_all" is something like
>
> def run_all():
>    while 1:
>      stackless.schedule()
>      if <<not stackless jobs are running>>:
>          break
>
> I used << >> here because there are many ways to do the test.  One is
> to look at the number of running tasklets.  The main tasklet is always
> runnable and the sleep_scheduler is also always running.  If the
> number of runnable tasklets is 2 or less then run_all can exit.
>
>      if stackless.getruncount() <= 2:
>          break
>
> (BTW, I test after the schedule() to make sure all tasklets have a
> chance to run.)
>
> If you're writing an application you probably want a user command to
> shut things down.  That might look like
>
> _finished = False
> def exit():
>      global _finished
>      _finished = True
>
> def run_all():
>      global _finished
>      _finished = True
>      while 1:
>          stackless.schedule()
>          if _finished:
>              break

I replied to your post about this, but didn't have the energy or time
to go into further detail then.

I said something along the lines that it was better to discourage
users from expecting stackless.run to work as a magic "exit when
things are done" method by encouraging them to use another approach
from the start.  Well, I think I said it and if I didn't it was
what I was thinking anyway :-)

The way I see running the scheduler working is a standard form.

yieldChannel = stackless.channel()
sleepQueue = []

while CheckExitCondition():
    RunNiceTasklets()			# Sends to all on yieldChannel
    CheckSleepingTasklets()		# Processes sleepQueue.

    runawayTasklet = stackless.run(2000000)
    if runawayTasklet:
        print_a_stacktrace_for(runawayTasklet)
        runawayTasklet.kill()

    DoPolling()				# Check events on async resources

If it were up to me and it is not, I would encourage this pattern for
all use of Stackless.  And I have provided some code in the form of
the run method in the uthread module to do something like this.  Your
sleep method is better of course.

http://svn.python.org/view/stackless/sandbox/libraries/uthread-ccp/uthread.py

However as you demonstrate in this presentation there is no way to
generically know to "exit when things are done".  And this is why I
have come to think that whatever sits in the place of CheckExitCondition
should explicitly check whether the application has a reason to exit
rather than whether any arbitrary Stackless resources are still
around.

But on the other hand I also think people should be able to just use
Stackless without having to worry about such complicated matters
and gain its benefits as easily as possible.

I wouldn't let this affect your presentation unless you see some value
in it.

> and have a tasklet call "exit" when it's time to finish.  What I
> decided to do for this talk was define a new tasklet creation function
> called "main_tasklet".  It works like "stackless.tasklet" expect that
> my run_all function will not exit until all of the "main" tasklets are
> finished.
>
> main_tasklet_count = 0
>
> def main_tasklet(f):
>      def main_wrapper(*args, **kwargs):
>          global main_tasklet_count
>          main_tasklet_count += 1
>          try:
>              f(*args, **kwargs)
>          finally:
>              main_tasklet_count -= 1
>      return stackless.tasklet(main_wrapper)
>
> The code is a bit tricky because it defines a function in one tasklet
> which will be run in another tasklet.  The main reason for this was to
> keep the same API as "stackless.tasklet".

How about another name instead of main tasklet?  On one hand it is what
Stackless calls the tasklet that you are creating these in and on the
other when you launch several, as in the example at the end, it looks
a little confusing.

> Either works fine for this case and I originally put the sleep in
> run_all.  Aftwards I found a conflict when I added support for I/O.  I
> wanted to use asyncore's timeout option to schedule the wakeup and
> because of the implementation that meant the asyncore loop cannot run
> in the main tasklet.  Why?  Stackless does not allow the main tasklet
> to block on a channel, which I used to let asyncore communicate with
> the tasklets waiting for data.

And you use the term here in its Stackless terminology sense, which
may confuse people.

> The stackless code implements very little of the functionality in
> urllib2.  It's really more a implementation of the socket class than
> an HTTP interface, with StacklessFileAdapter more like the result of
> socket.makefile()
>
> Richard Tew wrote a "stacklesssocket.py" module which implements the
> same API as the standard library socket module using stackless and
> asyncore.  Getting the details right is a bit tricky and I won't cover
> them here.  It uses the same principles I've already covered so you
> should be able to figure out most of it from the source code.
>
> That module implements its own scheduler using a busy-wait loop which
> polls for 0.05 seconds then yields control.  I don't like the
> busy-wait so I'll replace the function which starts that scheduler and
> have it use my own code.
>
> import stacklesssocket
> from stacklesssocket import stdsocket as socket
>
> def slib_socket(family=socket.AF_INET, type=socket.SOCK_STREAM,
> proto=0):
>      new_socket = socket.socket(family, type, proto)
>      return stacklesssocket.stacklesssocket(new_socket)
>
> stacklesssocket.socket = slib_socket

I do this if I want to bypass the 'automatic poller':

import stacklesssocket
stacklesssocket.managerRunning = True

But maybe you want this approach to illustrate monkeypatching.

> === Conclusion ===
>
> Most people who currently use Stackless got into it for game
> programming, or more generically for programming "actors" in a
> simulation.  Stackless is more general purpose than that.  It's a type
> of threading package and can replace a lot of what you might otherwise
> use system threads for.  Because it's more light-weight than threads
> it easier to use for different sorts of programming constructs you
> wouldn't otherwise do using normal Python with threads.
>
> Getting started in Stackless is an effort.  It's a different way of
> thinking for most people and it doesn't have an easy learning curve.
> Unlike system threads, doing "normal" things like network I/O requires
> a decent understanding of how the pieces go together.
>
> I think I've explained enough of it now to help, but it could be made
> easier.  With a bit more work, perhaps a few weeks, the "slib" module
> could be finished up and made into a package so people can more easily
> start writing Stackless code with very few changes to existing code.

It looks like a good overview with excellent coverage of everything
someone needs to know to start making use of Stackless.

I've been thinking that the uthread-ccp module should really be
cleaned up and possibly included as part of Stackless.  Perhaps
if you think it suits you can integrate the things you think help
into it, perhaps extending the uthread.run method?  Improving
the sleep method or whatever.  That's something to think about
anyway.

Cheers,
Richard.

_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://www.stackless.com/mailman/listinfo/stackless



More information about the Stackless mailing list