[Stackless] exceptions.TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell when unpickling stackless thread

Crispin Wellington retrogradeorbit at gmail.com
Tue Oct 13 06:47:54 CEST 2009


Tested this patched stackless against my full server expressing this
unpickle problem. The pickle/unpickle now works perfectly! Thankyou very
much for this high speed fix!

All the best

Crispin

2009/10/12 Kristján Valur Jónsson <kristjan at ccpgames.com>

>  This has been fixed in the stackless trunk, and merged into 2.6, 3.0, 3.1
> and py3k in revisions 75377-75381
>
> K
>
> *From:* stackless-bounces at stackless.com [mailto:
> stackless-bounces at stackless.com] *On Behalf Of *Kristján Valur Jónsson
> *Sent:* 12. október 2009 11:19
> *To:* Crispin Wellington; Richard Tew
>
> *Cc:* stackless at stackless.com
> *Subject:* Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected
> cell, found stackless._wrap.cell when unpickling stackless thread
>
>
>
> Ok, I understand the issue.
>
> Cell objects potentially can be part in a cycle.  Therefore for pickling,
> the three argument __reduce__ protocol is used, with an initial ‚new‘,
> followed by a ‚setstate‘ later.
>
> Only in the ‚setstate‘ call will the type of the object be changed from the
> special „_wrap“ type.  But precisely because a reference cycle is being
> constructed, a fresh cell is being inserted into a function, before having
> had its state set, and the function constructor balks at this.
>
> I see no way around this using the current wrapped pickle technique except
> to fix the test in the function object.  Once I have created a simple
> regression test, I will submit a fix.
>
>
>
> K
>
>
>
> *From:* stackless-bounces at stackless.com [mailto:
> stackless-bounces at stackless.com] *On Behalf Of *Crispin Wellington
> *Sent:* 12. október 2009 05:27
> *To:* Richard Tew
> *Cc:* stackless at stackless.com
> *Subject:* Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected
> cell, found stackless._wrap.cell when unpickling stackless thread
>
>
>
> OK. I boiled down the code to a small snippet that triggers the bug. It is
> a bug that is exposed through my use of Twisted. If i create a factory, and
> then two callbacks that are closures, each callback setting thread level
> variables. Then I add the callbacks to the factory's deferred, and then I
> sleep the threadlet forever.
>
> Initiate the tasklet, pickle it, then unpickle it, and BAM, you have your
> error:
>
> $ /usr/local/stackless/bin/python break-stackless.py
> Traceback (most recent call last):
>   File "break-stackless.py", line 47, in <module>
>     test()
>   File "break-stackless.py", line 37, in test
>     obj =  pickle.loads(data)
>   File "/usr/local/stackless/lib/python2.6/pickle.py", line 1409, in loads
>     return Unpickler(file).load()
>   File "/usr/local/stackless/lib/python2.6/pickle.py", line 893, in load
>     dispatch[key](self)
>   File "/usr/local/stackless/lib/python2.6/pickle.py", line 1252, in
> load_build
>     setstate(state)
> TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell
>
> The code is....
>
> -----------------------------
>
> import stackless
> import pickle
>
> from twisted.web import client
>
> def tasklet():
>     # create an example state
>     factory = client.HTTPClientFactory("http://www.google.com/", agent =
> "stackless/1.0" )
>
>     get_complete = [False]
>     get_failed = [False]
>
>     def _doFailure(data):
>         get_failed[0] = factory.status
>
>     def _doSuccess(data):
>         get_complete[0] = factory.status
>
>     factory.deferred.addCallback(_doSuccess).addErrback(_doFailure)
>
>     while True:
>         stackless.schedule()
>
> def test():
>     # run the task
>     task = stackless.tasklet(tasklet)
>     task.setup()
>     task.run()
>
>     # pump once
>     stackless.schedule()
>
>     # now serialise
>     data = pickle.dumps(task)
>
>     # now deserialise
>     obj =  pickle.loads(data)
>
>     # resume
>     task.kill()
>     obj.insert()
>
>     # pump again
>     stackless.schedule()
>
> if __name__=="__main__":
>     test()
>
> -----------------------
>
> If code formatting is an issue, I've attached the code to the email as a
> file...
>
> Can anyone shed any light on this bug?
>
> Kind Regards
>
> Crispin
>
> On Sat, Oct 10, 2009 at 2:21 AM, Richard Tew <richard.m.tew at gmail.com>
> wrote:
>
> On Fri, Oct 9, 2009 at 10:38 PM, Crispin Wellington
> <retrogradeorbit at gmail.com> wrote:
> > I have written a server that contains stackless threads. When the server
> > shuts down these threads are serialised to disk. This all happens without
> > incident and I end up with a reasonably sized file per pickled object.
> >
> > When I go to deserialise them I hit a problem. I get this...
> > -------------------
> >       File
> >
> "/home/cwellington/Development/yabi/yabi-be-twisted/trunk/TaskManager/Tasklets.py",
> > line 54, in load
> >         task = pickle.loads(data)
> >       File "/usr/local/stackless/lib/python2.6/pickle.py", line 1415, in
> > loads
> >         return Unpickler(file).load()
> >       File "/usr/local/stackless/lib/python2.6/pickle.py", line 898, in
> load
> >         dispatch[key](self)
> >       File "/usr/local/stackless/lib/python2.6/pickle.py", line 1258, in
> > load_build
> >         setstate(state)
> >     exceptions.TypeError: arg 5 (closure) expected cell, found
> > stackless._wrap.cell
> > --------------------
> >
> > I think this might be a bug. The arguments are being passed back into the
> > constructor of the stackless._wrap.function and the constructor is
> throwing
> > that exception. So in essence, the __reduce__ or __reduce_ex__ functions
> are
> > creating instatiation values that do not work on reinstantiation.
> >
> > I changed pickle.py to print out some debug. setstate is... <built-in
> method
> > __setstate__ of stackless._wrap.function object at 0x21ca0c8> and
> (state)[4]
> > is... (<cell at 0x2605f30: empty>, <cell at 0x2940718: list object at
> > 0x259f518>, <cell at 0x2940670: str object at 0x2940768>, <cell at
> > 0x2940750: str object at 0x29407d8>, <cell at 0x29407c0: int object at
> > 0x25cd338>)
> >
> > These all seem to be "cell" objects. Why aren't they
> (stackless._wrap.cell)
> > recognised as "cells"?
> >
> > I can provide a pickletools.dis() dump of the pickle stream if wanted.
> >
> > Using Stackless Python 2.6.2 on 64 bit linux (x86_64).
> >
> > Can anybody help me decode my objects? or even find a way to encode them
> so
> > the are decodable? Is this a bug, or am I doing it wrong?
>
> It definitely looks like a bug.  If you can provide the smallest
> possible reproduction case, it would help us look into it and we could
> add it to the test suite as a regression test.
>
> If you want your objects back in the meantime, you might try compiling
> 2.6 with the relevant code raising these errors commented out.
>
> Richard.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20091013/a46ad43d/attachment.htm>


More information about the Stackless mailing list