[Stackless] need help on debugging a deadlock

Paul Sijben sijben at eemvalley.com
Mon Jan 12 14:35:29 CET 2009


Hi Andrew and Richard,

My problem was a kind of circular wait like Andrew described.

I sorted it out. Then I ran into the issue that a channel is not a
queue. Sorted that too by adding queue-like behaviour to my channel
wrapper and now my app is running faster thanks to stackless.

Andrew Francis wrote:
> Hi Paul:
>
>   
>> Message: 1
>> Date: Fri, 09 Jan 2009 16:50:44 +0100
>> From: Paul Sijben <sijben at eemvalley.com>
>> Subject: [Stackless] need help on debugging a deadlock
>> To: stackless at stackless.com
>> Message-ID: <49677254.1040804 at eemvalley.com>
>> Content-Type: text/plain; charset=UTF-8
>>
>> I am creating a deadlock somewhere.
>>
>> Is there a way to ensure that a tasklet does not receive
>> more than a
>> certain number of milliseconds and is then stopped with
>> some message on
>> the console so I can see which one it was?
>>
>> Or are there even better ways to find out what is going on?
>>     
>
> Gee whiz, my first deadlock programme was in December 2005!
> http://www.stackless.com/pipermail/stackless/2005-December/000290.html
>
> I'll assume that Stackless complains about deadlock? Or the programme is simply hanging?
>
> I feel the best way to find out what is going on concerning deadlock, is to understand how deadlock occurs. Deadlock occurs when four conditions happen simultaneously: mutual exclusion, non-preemption, hold-and-wait, circular wait.
>
> Stackless in non-preemption mode essentially give you the first two conditions. I'll argue hold-and-wait looks like:
>
> ch1.receive()   # I am waiting on a resource
> ch2.send()      # I am holding a resource 
>
> Circular wait:
>
> tasklet1():
>     ch1.send()
>     ch2.receive()
>
> tasklet2():
>     ch2.send()
>     ch1.receive()
>
> However, due to transitivity, this is harder to detect. A wait-for graph is needed. I believe this is the way it works. 
>
> Tasklet channel operations are nodes. Channels are resources. If a tasklet does an operation on channel owned by a particular node, draw an directed edge to that node. When the transaction is finished, remove the edge. If there is a cycle, there is deadlock.
>
> In the following case, a line would be drawn from tasklet1 to tasklet2. 
> In turn, there is a line drawn from tasklet2 to tasklet1. Deadlock.
>
> As for debugging. Often I find the following helpful
>
> 1. create a function, say makeChannel(). As a part of the function, store the channel in some list.
>
> 2. In my main loop, I use the construct
>
> while (stackless.getruncount() > 1):
>     stackless.schedule()
>
> (I guess stackless.run() would work too)
>
> depending on your programme, if all the tasklets are blocked (i.e., due to deadlock), control will fall out of the loop.
>
> 3. Iterate over the list and do a __reduce__ on each channel. You will see the channel balances and the associated tasklets. This will give you insights about what is happening.
>
> Looking back at the December 2005 posts, perhaps with decorators, we could get enough information to build a graph walker/cycle detector? In 2009, we should be much better at solving this :-|
>
> Anyhow, if you can, post code.
>
> Cheers,
> Andrew
>
> P.S - Shameless plug: I explain deadlock in my Pycon 2008 talk "Adventures in Stackless Python Twisted Integration".....
>
>
>
>
>       
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>   

-- 
Paul Sijben                    	tel: +31334566488
Eemvalley Technology BV       	fax: +31334557523
the Netherlands                	http://eemvalley.com    
-----------------------------------------------------
EemValley Technology werft python & wxpython 
programmeurs: http://www.eemvalley.nl/jobs

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20090112/6893aeea/attachment.htm>


More information about the Stackless mailing list