[Stackless] Stackless Twisted Integration Example

Andrew Francis andrewfr_ice at yahoo.com
Sun Feb 25 18:45:26 CET 2007


Hello Colleagues:

As per Richard Tew's request, I have provided an
example of how to use Twisted with Stackless Python. I
feel that using the Twisted framework allows Stackless
Python network programmers to be more productive. Also
if one can replace heavy weight OS threads with
lightweight  tasklets, this is another major plus. 

In the first example, I provide a simple HTTP server
that runs Twisted and Stackless in a single thread.
The server is based on the web server example on pages
41 of Abe Fettig "Twisted Network Programming
Essentials."

For simplicity, I have the "CGI" tasklet and the
Twisted server share a single channel.

To see what is happening, execute the application and
access the server using a web browser and URL
http://localhost:8000

The browser should see:

cgiTasklet-1 received path: /count :0

The console will echo :

tick : count

what one will notice is that the Tick tasklet executes
only after an incoming request.

As I explained in previous posts, the problem is that
when Twisted blocks, the entire Python application
blocks. This severely cuts down concurrency.

To avoid this programme, I execute Stackless Python
and Twisted in separate threads. Other than the
Stackless component being executed in a thread via the
Twisted callInThread() method, the examples are not
much different. This includes having the threads
execute via a channel (I think it is safer to use a
deque). 

The all important difference is that while the CGI
tasklet waits for requests, Tick executes. 

In the near future, I will provide more examples,
illustrating how to use deferreds with the
reactor.callFromThread() method. This is useful if one
wishes to use say, the client.getPage() method.

Feedback, especially problems with this approach would
be appreciated. I would also be interested if someone
could stress test this approach to see where it breaks
down.

Cheers,
Andrew

P.S : Note I am using Stackless 2.4.3 with Twisted 2.4

~~~

#!/usr/bin/env python
"""
Webserver.py
Andrew Francis
February 25th, 2007

Example of Twisted and Stackless integration that
blocks.

The server listens on http://localhost:8000

The programme and is fine for many purposes. However
there is a
flaw. Whenever the server tasklet blocks, it blocks
the entire
programme. Ideally other tasklets, such as the Tick
tasklet should
run while the server tasklet waits for connections.

<song>The Bleeding Heart Show - The New
Pornographers</song>
"""


import stackless
from twisted.web           import http

class Server(object):
    
    def execute(self, port, requestChannel):
        MyRequestHandler.requestChannel =
requestChannel
        reactor.listenTCP(port, MyHttpFactory())
        reactor.run()


class Cgi(object):
    def __init__(self, name, channel):
        self.name = name
        self.channel = channel
        self.count = 0
        return

    def execute(self):
        while (1):

            path = self.channel.receive()
            print path
            self.channel.send("<html><body>" + \
                               self.name + " received
path: " + path + "count :" + str(self.count) + \
            "</body></html>")
            
            self.count = self.count + 1
            stackless.schedule()
            

def tick():
    count = 0
    while (1):
        print "tick: ", count
        count += 1
        stackless.schedule()
            
            

class MyRequestHandler(http.Request):
    
    def process(self):
       
MyRequestHandler.requestChannel.send(self.path)
        result =
MyRequestHandler.requestChannel.receive()
        self.write(result)
        self.finish()
        
        
class MyHttp(http.HTTPChannel):
    requestFactory = MyRequestHandler
    
    
class MyHttpFactory(http.HTTPFactory):
    protocol = MyHttp
    
    
if __name__ == "__main__":
    from twisted.internet import reactor
    
    channel = stackless.channel()
    
    cgiTasklet = Cgi("cgiTasklet-1", channel)
    server = Server()
    
    stackless.tasklet(cgiTasklet.execute)()
    stackless.tasklet(server.execute)(8000, channel)
    stackless.tasklet(tick)()
    
    while (stackless.getruncount() > 1):
       stackless.schedule()

~~

#!/usr/bin/env python
"""
ThreadedWebserver.py
Andrew Francis
February 25th, 2007

Example of Twisted and Stackless integration that
allows non-blocking
tasklets to execute

The server listens on http://localhost:8000

Unlike the previous example, ThreadedWebserver runs
Stackless in a
separate thread. While Twisted is blocked waiting for
connections,
the tick tasklet (or any other tasklet) can execute.

<song>Tom Sawyer - Rush</song>
"""


import stackless
from twisted.web           import http


class Server(object):
    
    def execute(self, port, requestChannel):
        MyRequestHandler.channel = requestChannel
        reactor.listenTCP(port, MyHttpFactory())
        reactor.run()


class Cgi(object):
    def __init__(self, name, channel):
        self.name = name
        self.channel = channel
        self.count = 0
        return

    def execute(self):
        while (1):

            path = self.channel.receive()
            print path
            self.channel.send("<html><body>" + \
                               self.name + " received
path: " + path + "count :" + str(self.count) + \
            "</body></html>")
            
            self.count = self.count + 1
            stackless.schedule()
            

def tick():
    count = 0
    while (1):
        # we don't want too much output
        if count % 1000000 == 0:
            print "tick ", count
        count += 1
        stackless.schedule()
            
            

class MyRequestHandler(http.Request):
    
    def process(self):
        channel.send(self.path)
        result = channel.receive()
        self.write(result)
        self.finish()
        
        
class MyHttp(http.HTTPChannel):
    requestFactory = MyRequestHandler
    
    
class MyHttpFactory(http.HTTPFactory):
    protocol = MyHttp

    
def stacklessThread(requestChannel):
    
    cgiTasklet = Cgi("cgiTasklet-1", requestChannel)
    stackless.tasklet(cgiTasklet.execute)()

    """
    in this example, if the tick tasklet did not
exist,
    Stackless would complain about deadlock since the
    last runnable tasklet (cgiTasklet) would be
blocked
    """
    stackless.tasklet(tick)()
        
    while (stackless.getruncount() > 1):
          stackless.schedule()
    

if __name__ == "__main__":
    from twisted.internet import reactor
    
    channel = stackless.channel()
    reactor.callInThread(stacklessThread,channel)
    Server().execute(8000, channel)
    reactor.run()   
    
    
    
    


 
____________________________________________________________________________________
Want to start your own business?
Learn how on Yahoo! Small Business.
http://smallbusiness.yahoo.com/r-index

_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://www.stackless.com/mailman/listinfo/stackless



More information about the Stackless mailing list