[Stackless] Comments on a Twisted/Stackless Integration Example

Andrew Francis andrewfr_ice at yahoo.com
Fri Jun 29 21:07:36 CEST 2007


Hello Colleagues:

The following is a sample programming illustrating
Twisted/Stackless integration. The sample code solves
a number of problems that I have experienced over the
past year. I feel a Stackless/Twisted combo can
provide most of the advantages of an event based and
thread based system. 

That said, the solutions I am presenting are specific
to solving problems involving my project: prototyping
WS-BPEL processors. I don't believe 80% of projects
require these techniques to benefit from Twisted.
Still it is nice to figure out the issues involving
how Stackless and Twisted interact.


For background follow:

http://www.stackless.com/pipermail/stackless/2006-June/001743.html
http://www.stackless.com/pipermail/stackless/2007-April/002453.html
http://www.stackless.com/pipermail/stackless/2007-April/002496.html
http://www.stackless.com/pipermail/stackless/2007-May/002521.html

(have I really been at this for about a year?)

In the example, I am invoking a sequence of calls that
correspond to a receiving a HTTP request, making a
call to a remove server, then making a reply. A
practical use would be a CGI tasklet that makes a
network connection to backend database or invokes a
web service as a part of a response to a client. 

In the course of learning how to use Twisted with
Stackless, I would encounter two problems:

1) Allowing tasklets that could run, to run, while the
reactor "tasklet" was blocked (i.e., waiting for a
HTTP request). I leant that the problem was not that
reactor blocked indefinitely. Rather the reactor
tasklet did not yield to the Stackless scheduler.

2) If a (HTTP)RequestHandler blocked on a channel,
this had the effect of halting the reactor tasklet
(more specifically the reactor mainLoop does not
iterate). As a result, subsequent calls to the Twisted
reactor will fail; twisted will appear  to be "losing"
deferred calls.

In the simple HTTP request, client.getPage, HTTP
response sequence example, this had the effect of
causing the "CGI" tasklet to hang since the getPage
call never returned. 

The key to solving the problems was understanding that
the reactor was a tasklet and which operations are
being done as a part of the reactor tasklet. 

To solve the first problem, I use a Twisted
task.LoopingCall that calls stackless.schedule() (I
also implement sleep() functions using
task.LoopingCall).  The aforementioned approach works
because scheduled tasks are being invoked as a part of
the reactor tasklet. As a result, the reactor tasklet
yielded to the Stackless scheduler.

To solve the second problem,  I have the
RequestHandler create a separate tasklet that does the
bulk of the request processing. As a result, the
Reactor tasklet can continue along its merry way,
processing network requests, while the RequestHandler
tasklet can block for an indefinite time.

I have not thoroughly tested this technique (I still
need to run battery of tests). Besides working, the
advantages of this approach is that it does not
require changes to the Twisted reactor, for example, a
custom reactor or a thread (over the months, what
often confused me were applications that used a
separate threads). The main drawbacks of this approach
are

1) An extra tasklet is created for each connection.
2) The approach seems to be usually slow (for now, I
care more about ease-of-use than performance). 

Meanwhile, I would appreciate comments. If folks feel
this is a proper approach, I will write it up in the
Wiki.

Cheers,
Andrew



 
____________________________________________________________________________________
Now that's room service!  Choose from over 150,000 hotels
in 45,000 destinations on Yahoo! Travel to find your fit.
http://farechase.yahoo.com/promo-generic-14795097
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Solution.py
Type: text/x-python
Size: 9765 bytes
Desc: 2700386180-Solution.py
URL: <http://www.stackless.com/pipermail/stackless/attachments/20070629/407b8001/attachment.py>
-------------- next part --------------
_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://www.stackless.com/mailman/listinfo/stackless


More information about the Stackless mailing list