[Stackless] Tasklet cleanup?

Kristján Valur Jónsson kristjan at ccpgames.com
Wed May 23 21:14:28 CEST 2012


Yes, i´m sure we need to revisit this.

> -----Original Message-----
> From: stackless-bounces at stackless.com [mailto:stackless-
> bounces at stackless.com] On Behalf Of Sylvain Prat
> Sent: 23. maí 2012 14:41
> To: The Stackless Python Mailing List
> Subject: Re: [Stackless] Tasklet cleanup?
> 
> Hello,
> 
> 1. The solution I proposed in my previous email doesn't work since the last
> dying tasklet would never get a chance to be cleaned up.
> 
> 2. I finally sorted my tasklet cleanup problem out by breaking the reference
> cycle. I didn't succeed to break the cycle by using a weakref on the cycle:
> some elements were garbage collected too early.
> I finally added a level of indirection, namely a "current" variable which holds
> the currently executing wrapper, which works similarly to stackless.current.
> Then my function use the "current" variable and is not a bound method
> anymore. Reference cycle broken, and now I can use __del__ to kill my
> tasklets/wrappers and get a similar behavior than greenlets for cleanup. I
> hope I am clear ;)
> 
> 3. I still consider that an TaskletExit should be raised for cleanup, but at least,
> I have found a workaround ;)
> 
> Cheers,
> Sylvain
> 
> 
> On Mon, May 21, 2012 at 4:24 PM, Sylvain Prat <sylvain.prat at gmail.com>
> wrote:
> > Thanks for the clarification Kristján! So, we should not expect a
> > TaskletExit to be raised ;)
> >
> > I fully understand the problem with __del__ and circular references,
> > which is also documented there:
> > http://docs.python.org/reference/datamodel.html#object.__del__
> >
> > The problem is, when I try to cleanup resources by myself, I also get
> > myself into the __del__ & circular references trap, and I can't find a
> > workaround. If I try to replace a hard reference by a weakref in the
> > circular references loop, I also get some errors (I've not
> > investigated them yet, maybe they are avoidable). I suspect the only
> > way to get cleanup work is to do something in stackless itself.
> >
> > Also, what do you mean by "it were safer to put such tasklets into
> > gc.garbage"? How can we do the cleanup then? Do you mean we have to
> > poll gc.garbage (maybe in a cleaning tasklet), identify tasklets that
> > have to be destroyed, and kill them manually, so that we can control
> > the moment when the tasklet will be destroyed?
> >
> > About the problem of "switching" to the dying tasklet and returning
> > back to the caller: why not flag the "garbage collectable" tasklet as
> > "to be killed", put it in the runnables (it is not garbage collectable
> > anymore, which is not recommended but possible - see
> > http://docs.python.org/reference/datamodel.html#object.__del__), and
> > let it wait for its turn (scheduling) to actually kill it? In this
> > case, it would have a chance to cleanup, just not immediately.
> >
> > Any suggestions/workarounds for resources cleanup?
> >
> > Cheers,
> > Sylvain
> >
> >
> > On Mon, May 21, 2012 at 1:21 PM, Kristján Valur Jónsson
> > <kristjan at ccpgames.com> wrote:
> >> Well.
> >> It is due to this code here, in taskletojbect:
> >>
> >> static int
> >> tasklet_traverse(PyTaskletObject *t, visitproc visit, void *arg) {
> >>    PyFrameObject *f;
> >>    PyThreadState *ts = PyThreadState_GET();
> >>
> >>    /* tasklets that need to be switched to for the kill, can't be collected.
> >>     * Only trivial decrefs are allowed during GC collect
> >>     */
> >>    if (tasklet_has_c_stack(t))
> >>        PyObject_GC_Collectable((PyObject *)t, visit, arg, 0);
> >>
> >> What can be done during garbage collection has always been a bit
> restricted.  Tranditionally, gcmodule disallows all objects with __del__
> methods and puts them in gc.garbage.  I have recently discovered, by asking
> on python-dev, that this is not due to a fundamental fragility of the garbage
> collection process, but rather the fact that it is likely that __del__ methods
> invoked during garbage collection will cause exceptions, since the members
> of the cycle would refer to each other.  See this discussion:
> >> http://grokbase.com/p/python/python-dev/124hc36dtk/issue-9141-
> finaliz
> >> ers-and-gc-module
> >>
> >> Anyway, for tasklets, the problem is that when killing tasklets, we have to
> "switch" to them.  If other tasklets are runnable, this may kick off a sequence
> of running tasklets, there is no _guarantee_ that we will switch immediately
> back to the tasklet currently doing the garbage collection.
> >> In the past, we have had a number of crashes related to this.  I don't have
> the details anymore.
> >>
> >> So, anyway, at one point I decided that it were safer to put such tasklets
> into gc.garbage.
> >>
> >> It is possible that my fears are unfounded.  We could possibly engineer
> tasklet.kill() in such a way that we are very sure that no other tasklets will run
> and we will switch immediately back to the originating tasklet.
> >>
> >> K
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Richard Tew [mailto:richard.m.tew at gmail.com]
> >>> Sent: 20. maí 2012 00:06
> >>> To: The Stackless Python Mailing List
> >>> Cc: Kristján Valur Jónsson
> >>> Subject: Re: [Stackless] Tasklet cleanup?
> >>>
> >>> On Sun, May 20, 2012 at 10:29 AM, Sylvain Prat
> >>> <sylvain.prat at gmail.com>
> >>> wrote:
> >>> > I'm wondering how tasklets can clean themselves up when they are
> >>> > destroyed due to garbage collection (i.e. when they are not in the
> >>> > runnables and not referenced by any object anymore). Greenlet
> >>> > solves this problem by raising a GreenletExit exception in the
> >>> > greenlet's run function when the greenlet is about to die due to
> garbage collection.
> >>> > However, in stackless, it seems that no TaskletExit exception is
> >>> > raised when the tasklet is about to die, so we can't simply use a
> >>> > try/finally in the tasklet's callable to clean up resources.
> >>>
> >>> Greenlet is derived from Stackless, so similarly TaskletExit should
> >>> be raised on a tasklet that is being garbage collected.
> >>>
> >>> > I tried to wrap my tasklet in a parent object which has the same
> >>> > lifespan as my tasklet and has a __del__ function for cleaning up,
> >>> > but I keep having problems with circular references (the
> >>> > wrapper/parent object also provides the callable of the tasklet,
> >>> > i.e. a bound method) that make the tasklet/parent object
> >>> > uncollectable (circular references
> >>> > : wrapper --> tasklet --> stackless machinery? --> callable stack
> >>> > frame (bound method of wrapper) --> wrapper). Same problem when
> >>> trying
> >>> > to inherit from tasklet.
> >>> >
> >>> > So, how can I clean up resources in tasklets? (I'm pretty sure
> >>> > I've missed something obvious)
> >>>
> >>> I've attached two example scripts where a tasklet dies, but does not
> >>> get the TaskletExit raised on it.  This is something Kristjan Valur
> >>> has been working on, but there have been upsides and downsides to
> >>> the different approaches and as I understand it the ideal solution to
> tasklet destruction is yet to be found.
> >>>
> >>> Kristjan, why is TaskletExit not being raised?  Any ideas?
> >>>
> >>> Cheers,
> >>> Richard.
> >>
> >>
> >> _______________________________________________
> >> Stackless mailing list
> >> Stackless at stackless.com
> >> http://www.stackless.com/mailman/listinfo/stackless
> >
> >
> >
> > --
> > Sylvain PRAT
> 
> --
> Sylvain PRAT
> 
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless





More information about the Stackless mailing list