[Stackless] Tasklet cleanup?

Sylvain Prat sylvain.prat at gmail.com
Mon May 21 16:24:02 CEST 2012


Thanks for the clarification Kristján! So, we should not expect a
TaskletExit to be raised ;)

I fully understand the problem with __del__ and circular references,
which is also documented there:
http://docs.python.org/reference/datamodel.html#object.__del__

The problem is, when I try to cleanup resources by myself, I also get
myself into the __del__ & circular references trap, and I can't find a
workaround. If I try to replace a hard reference by a weakref in the
circular references loop, I also get some errors (I've not
investigated them yet, maybe they are avoidable). I suspect the only
way to get cleanup work is to do something in stackless itself.

Also, what do you mean by "it were safer to put such tasklets into
gc.garbage"? How can we do the cleanup then? Do you mean we have to
poll gc.garbage (maybe in a cleaning tasklet), identify tasklets that
have to be destroyed, and kill them manually, so that we can control
the moment when the tasklet will be destroyed?

About the problem of "switching" to the dying tasklet and returning
back to the caller: why not flag the "garbage collectable" tasklet as
"to be killed", put it in the runnables (it is not garbage collectable
anymore, which is not recommended but possible - see
http://docs.python.org/reference/datamodel.html#object.__del__), and
let it wait for its turn (scheduling) to actually kill it? In this
case, it would have a chance to cleanup, just not immediately.

Any suggestions/workarounds for resources cleanup?

Cheers,
Sylvain


On Mon, May 21, 2012 at 1:21 PM, Kristján Valur Jónsson
<kristjan at ccpgames.com> wrote:
> Well.
> It is due to this code here, in taskletojbect:
>
> static int
> tasklet_traverse(PyTaskletObject *t, visitproc visit, void *arg)
> {
>    PyFrameObject *f;
>    PyThreadState *ts = PyThreadState_GET();
>
>    /* tasklets that need to be switched to for the kill, can't be collected.
>     * Only trivial decrefs are allowed during GC collect
>     */
>    if (tasklet_has_c_stack(t))
>        PyObject_GC_Collectable((PyObject *)t, visit, arg, 0);
>
> What can be done during garbage collection has always been a bit restricted.  Tranditionally, gcmodule disallows all objects with __del__ methods and puts them in gc.garbage.  I have recently discovered, by asking on python-dev, that this is not due to a fundamental fragility of the garbage collection process, but rather the fact that it is likely that __del__ methods invoked during garbage collection will cause exceptions, since the members of the cycle would refer to each other.  See this discussion:
> http://grokbase.com/p/python/python-dev/124hc36dtk/issue-9141-finalizers-and-gc-module
>
> Anyway, for tasklets, the problem is that when killing tasklets, we have to "switch" to them.  If other tasklets are runnable, this may kick off a sequence of running tasklets, there is no _guarantee_ that we will switch immediately back to the tasklet currently doing the garbage collection.
> In the past, we have had a number of crashes related to this.  I don't have the details anymore.
>
> So, anyway, at one point I decided that it were safer to put such tasklets into gc.garbage.
>
> It is possible that my fears are unfounded.  We could possibly engineer tasklet.kill() in such a way that we are very sure that no other tasklets will run and we will switch immediately back to the originating tasklet.
>
> K
>
>
>
>> -----Original Message-----
>> From: Richard Tew [mailto:richard.m.tew at gmail.com]
>> Sent: 20. maí 2012 00:06
>> To: The Stackless Python Mailing List
>> Cc: Kristján Valur Jónsson
>> Subject: Re: [Stackless] Tasklet cleanup?
>>
>> On Sun, May 20, 2012 at 10:29 AM, Sylvain Prat <sylvain.prat at gmail.com>
>> wrote:
>> > I'm wondering how tasklets can clean themselves up when they are
>> > destroyed due to garbage collection (i.e. when they are not in the
>> > runnables and not referenced by any object anymore). Greenlet solves
>> > this problem by raising a GreenletExit exception in the greenlet's run
>> > function when the greenlet is about to die due to garbage collection.
>> > However, in stackless, it seems that no TaskletExit exception is
>> > raised when the tasklet is about to die, so we can't simply use a
>> > try/finally in the tasklet's callable to clean up resources.
>>
>> Greenlet is derived from Stackless, so similarly TaskletExit should be raised
>> on a tasklet that is being garbage collected.
>>
>> > I tried to wrap my tasklet in a parent object which has the same
>> > lifespan as my tasklet and has a __del__ function for cleaning up, but
>> > I keep having problems with circular references (the wrapper/parent
>> > object also provides the callable of the tasklet, i.e. a bound method)
>> > that make the tasklet/parent object uncollectable (circular references
>> > : wrapper --> tasklet --> stackless machinery? --> callable stack
>> > frame (bound method of wrapper) --> wrapper). Same problem when
>> trying
>> > to inherit from tasklet.
>> >
>> > So, how can I clean up resources in tasklets? (I'm pretty sure I've
>> > missed something obvious)
>>
>> I've attached two example scripts where a tasklet dies, but does not get the
>> TaskletExit raised on it.  This is something Kristjan Valur has been working on,
>> but there have been upsides and downsides to the different approaches and
>> as I understand it the ideal solution to tasklet destruction is yet to be found.
>>
>> Kristjan, why is TaskletExit not being raised?  Any ideas?
>>
>> Cheers,
>> Richard.
>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless



-- 
Sylvain PRAT
+33 06 78 71 51 21



More information about the Stackless mailing list