[Stackless] Tasklet cleanup?

Sylvain Prat sylvain.prat at gmail.com
Wed May 23 16:41:25 CEST 2012


Hello,

1. The solution I proposed in my previous email doesn't work since the
last dying tasklet would never get a chance to be cleaned up.

2. I finally sorted my tasklet cleanup problem out by breaking the
reference cycle. I didn't succeed to break the cycle by using a
weakref on the cycle: some elements were garbage collected too early.
I finally added a level of indirection, namely a "current" variable
which holds the currently executing wrapper, which works similarly to
stackless.current. Then my function use the "current" variable and is
not a bound method anymore. Reference cycle broken, and now I can use
__del__ to kill my tasklets/wrappers and get a similar behavior than
greenlets for cleanup. I hope I am clear ;)

3. I still consider that an TaskletExit should be raised for cleanup,
but at least, I have found a workaround ;)

Cheers,
Sylvain


On Mon, May 21, 2012 at 4:24 PM, Sylvain Prat <sylvain.prat at gmail.com> wrote:
> Thanks for the clarification Kristján! So, we should not expect a
> TaskletExit to be raised ;)
>
> I fully understand the problem with __del__ and circular references,
> which is also documented there:
> http://docs.python.org/reference/datamodel.html#object.__del__
>
> The problem is, when I try to cleanup resources by myself, I also get
> myself into the __del__ & circular references trap, and I can't find a
> workaround. If I try to replace a hard reference by a weakref in the
> circular references loop, I also get some errors (I've not
> investigated them yet, maybe they are avoidable). I suspect the only
> way to get cleanup work is to do something in stackless itself.
>
> Also, what do you mean by "it were safer to put such tasklets into
> gc.garbage"? How can we do the cleanup then? Do you mean we have to
> poll gc.garbage (maybe in a cleaning tasklet), identify tasklets that
> have to be destroyed, and kill them manually, so that we can control
> the moment when the tasklet will be destroyed?
>
> About the problem of "switching" to the dying tasklet and returning
> back to the caller: why not flag the "garbage collectable" tasklet as
> "to be killed", put it in the runnables (it is not garbage collectable
> anymore, which is not recommended but possible - see
> http://docs.python.org/reference/datamodel.html#object.__del__), and
> let it wait for its turn (scheduling) to actually kill it? In this
> case, it would have a chance to cleanup, just not immediately.
>
> Any suggestions/workarounds for resources cleanup?
>
> Cheers,
> Sylvain
>
>
> On Mon, May 21, 2012 at 1:21 PM, Kristján Valur Jónsson
> <kristjan at ccpgames.com> wrote:
>> Well.
>> It is due to this code here, in taskletojbect:
>>
>> static int
>> tasklet_traverse(PyTaskletObject *t, visitproc visit, void *arg)
>> {
>>    PyFrameObject *f;
>>    PyThreadState *ts = PyThreadState_GET();
>>
>>    /* tasklets that need to be switched to for the kill, can't be collected.
>>     * Only trivial decrefs are allowed during GC collect
>>     */
>>    if (tasklet_has_c_stack(t))
>>        PyObject_GC_Collectable((PyObject *)t, visit, arg, 0);
>>
>> What can be done during garbage collection has always been a bit restricted.  Tranditionally, gcmodule disallows all objects with __del__ methods and puts them in gc.garbage.  I have recently discovered, by asking on python-dev, that this is not due to a fundamental fragility of the garbage collection process, but rather the fact that it is likely that __del__ methods invoked during garbage collection will cause exceptions, since the members of the cycle would refer to each other.  See this discussion:
>> http://grokbase.com/p/python/python-dev/124hc36dtk/issue-9141-finalizers-and-gc-module
>>
>> Anyway, for tasklets, the problem is that when killing tasklets, we have to "switch" to them.  If other tasklets are runnable, this may kick off a sequence of running tasklets, there is no _guarantee_ that we will switch immediately back to the tasklet currently doing the garbage collection.
>> In the past, we have had a number of crashes related to this.  I don't have the details anymore.
>>
>> So, anyway, at one point I decided that it were safer to put such tasklets into gc.garbage.
>>
>> It is possible that my fears are unfounded.  We could possibly engineer tasklet.kill() in such a way that we are very sure that no other tasklets will run and we will switch immediately back to the originating tasklet.
>>
>> K
>>
>>
>>
>>> -----Original Message-----
>>> From: Richard Tew [mailto:richard.m.tew at gmail.com]
>>> Sent: 20. maí 2012 00:06
>>> To: The Stackless Python Mailing List
>>> Cc: Kristján Valur Jónsson
>>> Subject: Re: [Stackless] Tasklet cleanup?
>>>
>>> On Sun, May 20, 2012 at 10:29 AM, Sylvain Prat <sylvain.prat at gmail.com>
>>> wrote:
>>> > I'm wondering how tasklets can clean themselves up when they are
>>> > destroyed due to garbage collection (i.e. when they are not in the
>>> > runnables and not referenced by any object anymore). Greenlet solves
>>> > this problem by raising a GreenletExit exception in the greenlet's run
>>> > function when the greenlet is about to die due to garbage collection.
>>> > However, in stackless, it seems that no TaskletExit exception is
>>> > raised when the tasklet is about to die, so we can't simply use a
>>> > try/finally in the tasklet's callable to clean up resources.
>>>
>>> Greenlet is derived from Stackless, so similarly TaskletExit should be raised
>>> on a tasklet that is being garbage collected.
>>>
>>> > I tried to wrap my tasklet in a parent object which has the same
>>> > lifespan as my tasklet and has a __del__ function for cleaning up, but
>>> > I keep having problems with circular references (the wrapper/parent
>>> > object also provides the callable of the tasklet, i.e. a bound method)
>>> > that make the tasklet/parent object uncollectable (circular references
>>> > : wrapper --> tasklet --> stackless machinery? --> callable stack
>>> > frame (bound method of wrapper) --> wrapper). Same problem when
>>> trying
>>> > to inherit from tasklet.
>>> >
>>> > So, how can I clean up resources in tasklets? (I'm pretty sure I've
>>> > missed something obvious)
>>>
>>> I've attached two example scripts where a tasklet dies, but does not get the
>>> TaskletExit raised on it.  This is something Kristjan Valur has been working on,
>>> but there have been upsides and downsides to the different approaches and
>>> as I understand it the ideal solution to tasklet destruction is yet to be found.
>>>
>>> Kristjan, why is TaskletExit not being raised?  Any ideas?
>>>
>>> Cheers,
>>> Richard.
>>
>>
>> _______________________________________________
>> Stackless mailing list
>> Stackless at stackless.com
>> http://www.stackless.com/mailman/listinfo/stackless
>
>
>
> --
> Sylvain PRAT

-- 
Sylvain PRAT



More information about the Stackless mailing list