[Stackless] Tying the Pieces Together Re: Stackless and Twisted Threads Redux

Andrew Francis andrewfr_ice at yahoo.com
Sat Jul 14 15:51:15 CEST 2007

Hi David, Carlos et al:

David, I have just downloaded your code. When I have
time I will look at it. In a strange way, I think we
are doing similar stuff in regards to controlling NPC
like entities. That said, it is great that you are
pursuing this stuff. I think the more we learn and
share, the better off we are.

>As I followed the Stackless - Twisted integration and
>also talked about it with Andrew, Chris Armstrong, we
>thought that Stackless and Twisted works nicely
>together without threads.

Carlos, this is true. However I think an important
point is that people know how to use Stackless with
Twisted and threads. After all there are situations
that may require using threads. Especially if one has
to develop something quick that works and performance
is not an issue. I think this later point gets lost in
many of the discussions.

We should figure out the use cases and write them in
the Wiki or on this mailing list.

>Also I found benefits using this. Having two threads,
>I had about 40 - 50% CPU usage. Using the unique
>thread approach, the CPU dropped to
>about 20 - 25%.

My latest solution does not use a thread. However I
think it would be interesting to know exactly why 
threads take such performance hit.

Again, in the process of using threads, I gained some
insights. For instance, it helps if one determines
whether one is using a channel for communications or
synchronization. If the former, then one gets a
performance gain if one replaces the channel with an
uThread queue. Again, I am interested in tying all the
pieces together.

>def twistedReactor():
>    l = task.LoopingCall(uthread.BeNice)
>    l.start(.0001)
>    reactor.run()

My solutions make extensive use of task.LoopingCall.
Task.LoopingCall can be used to make the Twisted
Reactor yield to the Stackless scheduler without
modifying Twisted.

The use of TaskingLoop is an idiom that should be
written up on the Stackless Wiki.

Back to threads for a moment. What I was essentially
doing (through transitivity) was allowing a Twisted
request handler to make a Twisted call and block on
the result before issuing a reply. However I was
"losing" deferreds. I originally explored threads
because I thought I could get around problems
involving tasklets losing deferred. The problem was
more subtle, and the solution involved placing the
request handler in its own tasklet.

That said, I am exploring two things. First, although
my current solution works, it takes a performance hit.
I want to know why. I also want to use to figure out
how to use slprofile to see what is happening. 

Secondly I want to figure out if the new Twisted
inline callbacks that use generators, will provide a
cleaner solution (or if it will work under that
situation). I think the inline callbacks/with
generators show how an asynchronous network framework
and a synchronous threading model can work nicely


Take the Internet to Go: Yahoo!Go puts the Internet in your pocket: mail, news, photos & more. 

Stackless mailing list
Stackless at stackless.com

More information about the Stackless mailing list