[Stackless] The case for Twisted
Joachim König-Baltes
joachim.koenig-baltes at emesgarten.de
Fri Aug 10 10:59:04 CEST 2007
Christopher Armstrong wrote:
> A lot of people seem to want to write their own event loop for doing
> Stackless I/O. I think this is a waste of time. It's a *lot* of work
> to get a robust and portable event loop.
>
IMO the problem is not primarily the event loop but having concurrent
non blocking tasks (in the sense
that one blocking task does not block the others from doing something
useful), see e.g.
http://www.kegel.com/c10k.html
The event loop is only one part of the problem and there are a lot of
solutions for it. But avoiding
blocking in a task is not only about doing I/O on OS level resources.
But instead of proposing one solution for avoiding blocking (e.g.
twisted, pyevent, ...) which are
procedural approaches to the problem (APIs and splitting up sequential
flows into chunks
that often look unrelated to each other, thus loosing the readability of
sequential code) I'd
prefer a declarative approach where a task can "inform" an upper level
(typically a scheduler)
of the (possibly blocking) events it is waiting on:
- readability/writeability of a socket/file (OS level resource) or
channel (stackless resource)
- process/thread (OS level) or task (stackless) event such as died,
stopped, ...
- signal
- timers
The upper level (scheduler) can then collect the events of all the tasks
and decide what to do,
e.g. directly continue a task that is waiting on resources completely
managed by by stackless
(e.g. channels) or, if no task can be continued without "asking" the OS
doing a select/poll/
whatever and when one of the resources signals "availability" continuing
that task.
So I think one thing that we could agree on is an API for:
- defining events for all the resource types that can be handled by the
scheduler
- an API for infoming an upper level abou the list of events a task is
interested
in (which would do an implicit yield to that upper level) and the API
call
would resume at exactly that location once one of the events occured.
How we then implement the scheduler(s) could be left open, but multiple
schedulers
could exist, e.g. based on twisted/pyevent/pykqueue ...
One OS level API which I like very much for it is kqueue, as it already
supports
defining events for files/sockets, processes, timers and signals. Its a
simple
C structure, but the pykqueue module gives more freedom in extending it for
stackless purposes.
It is however important that the event structure allows to support
additional
resource types that can be evaluated by a scheduler in order to also handle
the stackless resources (e.g. channels) and possibly other custom resources
where events are possible.
This would be a very simple API, a structure (class) for declaring
events and
a method that hands over a list of events (plus an optional timeout) to
the scheduler
that could be used in tasks (tasklets).
Joachim
_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://stackless.com/cgi-bin/mailman/listinfo/stackless
More information about the Stackless
mailing list