[Stackless] The case for Twisted

Joachim König-Baltes joachim.koenig-baltes at emesgarten.de
Fri Aug 10 10:59:04 CEST 2007


Christopher Armstrong wrote:
> A lot of people seem to want to write their own event loop for doing
> Stackless I/O. I think this is a waste of time. It's a *lot* of work
> to get a robust and portable event loop.
>   
IMO the problem is not primarily the event loop but having concurrent 
non blocking tasks (in the sense
that one blocking task does not block the others from doing something 
useful), see e.g.

http://www.kegel.com/c10k.html

The event loop is only one part of the problem and there are a lot of 
solutions for it. But avoiding
blocking in a task is not only about doing I/O on OS level resources.

But instead of proposing one solution for avoiding blocking (e.g. 
twisted, pyevent, ...) which are
procedural approaches to the problem (APIs and splitting up sequential 
flows into chunks
that often look unrelated to each other, thus loosing the readability of 
sequential code) I'd
prefer a declarative approach where a task can "inform" an upper level 
(typically a scheduler)
of the (possibly blocking) events it is waiting on:

- readability/writeability of a socket/file (OS level resource) or 
channel (stackless resource)
- process/thread (OS level) or task (stackless) event such as died, 
stopped, ...
- signal
- timers

The upper level (scheduler) can then collect the events of all the tasks 
and decide what to do,
e.g. directly continue a task that is waiting on resources completely 
managed by by stackless
(e.g. channels) or, if no task can be continued without "asking" the OS 
doing a select/poll/
whatever and when one of the resources signals "availability" continuing 
that task.

So I think one thing that we could agree on is an API for:

- defining events for all the resource types that can be handled by the 
scheduler
- an API for infoming an upper level abou the list of events a task is 
interested
  in (which would  do an implicit yield to that upper level) and the API 
call
 would resume at exactly that location once one of the events occured.

How we then implement the scheduler(s) could be left open, but multiple 
schedulers
could exist, e.g. based on twisted/pyevent/pykqueue ...

One OS level API which I like very much for it is kqueue, as it already 
supports
defining events for files/sockets, processes, timers and signals. Its a 
simple
C structure, but the pykqueue module gives more freedom in extending it for
stackless purposes.

It is however important that the event structure allows to support 
additional
resource types that can be evaluated by a scheduler in order to also handle
the stackless resources (e.g. channels) and possibly other custom resources
where events are possible.

This would be a very simple API, a structure (class) for declaring 
events and
a method that hands over a list of events (plus an optional timeout)  to 
the scheduler
that could be used in tasks (tasklets).

Joachim








_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://stackless.com/cgi-bin/mailman/listinfo/stackless



More information about the Stackless mailing list