[Stackless] exceptions.TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell when unpickling stackless thread
Kristján Valur Jónsson
kristjan at ccpgames.com
Mon Oct 12 16:28:01 CEST 2009
This has been fixed in the stackless trunk, and merged into 2.6, 3.0, 3.1 and py3k in revisions 75377-75381
K
From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of Kristján Valur Jónsson
Sent: 12. október 2009 11:19
To: Crispin Wellington; Richard Tew
Cc: stackless at stackless.com
Subject: Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell when unpickling stackless thread
Ok, I understand the issue.
Cell objects potentially can be part in a cycle. Therefore for pickling, the three argument __reduce__ protocol is used, with an initial 'new', followed by a 'setstate' later.
Only in the 'setstate' call will the type of the object be changed from the special "_wrap" type. But precisely because a reference cycle is being constructed, a fresh cell is being inserted into a function, before having had its state set, and the function constructor balks at this.
I see no way around this using the current wrapped pickle technique except to fix the test in the function object. Once I have created a simple regression test, I will submit a fix.
K
From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of Crispin Wellington
Sent: 12. október 2009 05:27
To: Richard Tew
Cc: stackless at stackless.com
Subject: Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell when unpickling stackless thread
OK. I boiled down the code to a small snippet that triggers the bug. It is a bug that is exposed through my use of Twisted. If i create a factory, and then two callbacks that are closures, each callback setting thread level variables. Then I add the callbacks to the factory's deferred, and then I sleep the threadlet forever.
Initiate the tasklet, pickle it, then unpickle it, and BAM, you have your error:
$ /usr/local/stackless/bin/python break-stackless.py
Traceback (most recent call last):
File "break-stackless.py", line 47, in <module>
test()
File "break-stackless.py", line 37, in test
obj = pickle.loads(data)
File "/usr/local/stackless/lib/python2.6/pickle.py", line 1409, in loads
return Unpickler(file).load()
File "/usr/local/stackless/lib/python2.6/pickle.py", line 893, in load
dispatch[key](self)
File "/usr/local/stackless/lib/python2.6/pickle.py", line 1252, in load_build
setstate(state)
TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell
The code is....
-----------------------------
import stackless
import pickle
from twisted.web import client
def tasklet():
# create an example state
factory = client.HTTPClientFactory("http://www.google.com/", agent = "stackless/1.0" )
get_complete = [False]
get_failed = [False]
def _doFailure(data):
get_failed[0] = factory.status
def _doSuccess(data):
get_complete[0] = factory.status
factory.deferred.addCallback(_doSuccess).addErrback(_doFailure)
while True:
stackless.schedule()
def test():
# run the task
task = stackless.tasklet(tasklet)
task.setup()
task.run()
# pump once
stackless.schedule()
# now serialise
data = pickle.dumps(task)
# now deserialise
obj = pickle.loads(data)
# resume
task.kill()
obj.insert()
# pump again
stackless.schedule()
if __name__=="__main__":
test()
-----------------------
If code formatting is an issue, I've attached the code to the email as a file...
Can anyone shed any light on this bug?
Kind Regards
Crispin
On Sat, Oct 10, 2009 at 2:21 AM, Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>> wrote:
On Fri, Oct 9, 2009 at 10:38 PM, Crispin Wellington
<retrogradeorbit at gmail.com<mailto:retrogradeorbit at gmail.com>> wrote:
> I have written a server that contains stackless threads. When the server
> shuts down these threads are serialised to disk. This all happens without
> incident and I end up with a reasonably sized file per pickled object.
>
> When I go to deserialise them I hit a problem. I get this...
> -------------------
> File
> "/home/cwellington/Development/yabi/yabi-be-twisted/trunk/TaskManager/Tasklets.py",
> line 54, in load
> task = pickle.loads(data)
> File "/usr/local/stackless/lib/python2.6/pickle.py", line 1415, in
> loads
> return Unpickler(file).load()
> File "/usr/local/stackless/lib/python2.6/pickle.py", line 898, in load
> dispatch[key](self)
> File "/usr/local/stackless/lib/python2.6/pickle.py", line 1258, in
> load_build
> setstate(state)
> exceptions.TypeError: arg 5 (closure) expected cell, found
> stackless._wrap.cell
> --------------------
>
> I think this might be a bug. The arguments are being passed back into the
> constructor of the stackless._wrap.function and the constructor is throwing
> that exception. So in essence, the __reduce__ or __reduce_ex__ functions are
> creating instatiation values that do not work on reinstantiation.
>
> I changed pickle.py to print out some debug. setstate is... <built-in method
> __setstate__ of stackless._wrap.function object at 0x21ca0c8> and (state)[4]
> is... (<cell at 0x2605f30: empty>, <cell at 0x2940718: list object at
> 0x259f518>, <cell at 0x2940670: str object at 0x2940768>, <cell at
> 0x2940750: str object at 0x29407d8>, <cell at 0x29407c0: int object at
> 0x25cd338>)
>
> These all seem to be "cell" objects. Why aren't they (stackless._wrap.cell)
> recognised as "cells"?
>
> I can provide a pickletools.dis() dump of the pickle stream if wanted.
>
> Using Stackless Python 2.6.2 on 64 bit linux (x86_64).
>
> Can anybody help me decode my objects? or even find a way to encode them so
> the are decodable? Is this a bug, or am I doing it wrong?
It definitely looks like a bug. If you can provide the smallest
possible reproduction case, it would help us look into it and we could
add it to the test suite as a regression test.
If you want your objects back in the meantime, you might try compiling
2.6 with the relevant code raising these errors commented out.
Richard.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20091012/84bcf91e/attachment.htm>
More information about the Stackless
mailing list