[Stackless] Tasklet cleanup?
sylvain.prat at gmail.com
Wed May 23 16:41:25 CEST 2012
1. The solution I proposed in my previous email doesn't work since the
last dying tasklet would never get a chance to be cleaned up.
2. I finally sorted my tasklet cleanup problem out by breaking the
reference cycle. I didn't succeed to break the cycle by using a
weakref on the cycle: some elements were garbage collected too early.
I finally added a level of indirection, namely a "current" variable
which holds the currently executing wrapper, which works similarly to
stackless.current. Then my function use the "current" variable and is
not a bound method anymore. Reference cycle broken, and now I can use
__del__ to kill my tasklets/wrappers and get a similar behavior than
greenlets for cleanup. I hope I am clear ;)
3. I still consider that an TaskletExit should be raised for cleanup,
but at least, I have found a workaround ;)
On Mon, May 21, 2012 at 4:24 PM, Sylvain Prat <sylvain.prat at gmail.com> wrote:
> Thanks for the clarification Kristján! So, we should not expect a
> TaskletExit to be raised ;)
> I fully understand the problem with __del__ and circular references,
> which is also documented there:
> The problem is, when I try to cleanup resources by myself, I also get
> myself into the __del__ & circular references trap, and I can't find a
> workaround. If I try to replace a hard reference by a weakref in the
> circular references loop, I also get some errors (I've not
> investigated them yet, maybe they are avoidable). I suspect the only
> way to get cleanup work is to do something in stackless itself.
> Also, what do you mean by "it were safer to put such tasklets into
> gc.garbage"? How can we do the cleanup then? Do you mean we have to
> poll gc.garbage (maybe in a cleaning tasklet), identify tasklets that
> have to be destroyed, and kill them manually, so that we can control
> the moment when the tasklet will be destroyed?
> About the problem of "switching" to the dying tasklet and returning
> back to the caller: why not flag the "garbage collectable" tasklet as
> "to be killed", put it in the runnables (it is not garbage collectable
> anymore, which is not recommended but possible - see
> http://docs.python.org/reference/datamodel.html#object.__del__), and
> let it wait for its turn (scheduling) to actually kill it? In this
> case, it would have a chance to cleanup, just not immediately.
> Any suggestions/workarounds for resources cleanup?
> On Mon, May 21, 2012 at 1:21 PM, Kristján Valur Jónsson
> <kristjan at ccpgames.com> wrote:
>> It is due to this code here, in taskletojbect:
>> static int
>> tasklet_traverse(PyTaskletObject *t, visitproc visit, void *arg)
>> PyFrameObject *f;
>> PyThreadState *ts = PyThreadState_GET();
>> /* tasklets that need to be switched to for the kill, can't be collected.
>> * Only trivial decrefs are allowed during GC collect
>> if (tasklet_has_c_stack(t))
>> PyObject_GC_Collectable((PyObject *)t, visit, arg, 0);
>> What can be done during garbage collection has always been a bit restricted. Tranditionally, gcmodule disallows all objects with __del__ methods and puts them in gc.garbage. I have recently discovered, by asking on python-dev, that this is not due to a fundamental fragility of the garbage collection process, but rather the fact that it is likely that __del__ methods invoked during garbage collection will cause exceptions, since the members of the cycle would refer to each other. See this discussion:
>> Anyway, for tasklets, the problem is that when killing tasklets, we have to "switch" to them. If other tasklets are runnable, this may kick off a sequence of running tasklets, there is no _guarantee_ that we will switch immediately back to the tasklet currently doing the garbage collection.
>> In the past, we have had a number of crashes related to this. I don't have the details anymore.
>> So, anyway, at one point I decided that it were safer to put such tasklets into gc.garbage.
>> It is possible that my fears are unfounded. We could possibly engineer tasklet.kill() in such a way that we are very sure that no other tasklets will run and we will switch immediately back to the originating tasklet.
>>> -----Original Message-----
>>> From: Richard Tew [mailto:richard.m.tew at gmail.com]
>>> Sent: 20. maí 2012 00:06
>>> To: The Stackless Python Mailing List
>>> Cc: Kristján Valur Jónsson
>>> Subject: Re: [Stackless] Tasklet cleanup?
>>> On Sun, May 20, 2012 at 10:29 AM, Sylvain Prat <sylvain.prat at gmail.com>
>>> > I'm wondering how tasklets can clean themselves up when they are
>>> > destroyed due to garbage collection (i.e. when they are not in the
>>> > runnables and not referenced by any object anymore). Greenlet solves
>>> > this problem by raising a GreenletExit exception in the greenlet's run
>>> > function when the greenlet is about to die due to garbage collection.
>>> > However, in stackless, it seems that no TaskletExit exception is
>>> > raised when the tasklet is about to die, so we can't simply use a
>>> > try/finally in the tasklet's callable to clean up resources.
>>> Greenlet is derived from Stackless, so similarly TaskletExit should be raised
>>> on a tasklet that is being garbage collected.
>>> > I tried to wrap my tasklet in a parent object which has the same
>>> > lifespan as my tasklet and has a __del__ function for cleaning up, but
>>> > I keep having problems with circular references (the wrapper/parent
>>> > object also provides the callable of the tasklet, i.e. a bound method)
>>> > that make the tasklet/parent object uncollectable (circular references
>>> > : wrapper --> tasklet --> stackless machinery? --> callable stack
>>> > frame (bound method of wrapper) --> wrapper). Same problem when
>>> > to inherit from tasklet.
>>> > So, how can I clean up resources in tasklets? (I'm pretty sure I've
>>> > missed something obvious)
>>> I've attached two example scripts where a tasklet dies, but does not get the
>>> TaskletExit raised on it. This is something Kristjan Valur has been working on,
>>> but there have been upsides and downsides to the different approaches and
>>> as I understand it the ideal solution to tasklet destruction is yet to be found.
>>> Kristjan, why is TaskletExit not being raised? Any ideas?
>> Stackless mailing list
>> Stackless at stackless.com
> Sylvain PRAT
More information about the Stackless