Tested this patched stackless against my full server expressing this unpickle problem. The pickle/unpickle now works perfectly! Thankyou very much for this high speed fix!<br><br>All the best<br><br>Crispin<br><br><div class="gmail_quote">
2009/10/12 Kristján Valur Jónsson <span dir="ltr"><<a href="mailto:kristjan@ccpgames.com">kristjan@ccpgames.com</a>></span><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div link="blue" vlink="purple" lang="IS">
<div>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">This has been fixed in the stackless trunk, and merged into 2.6,
3.0, 3.1 and py3k in revisions 75377-75381</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">K</span></p>
<div style="border-style: none none none solid; border-color: -moz-use-text-color -moz-use-text-color -moz-use-text-color blue; border-width: medium medium medium 1.5pt; padding: 0cm 0cm 0cm 4pt;">
<div>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span style="font-size: 10pt;" lang="EN-US">From:</span></b><span style="font-size: 10pt;" lang="EN-US"> <a href="mailto:stackless-bounces@stackless.com" target="_blank">stackless-bounces@stackless.com</a>
[mailto:<a href="mailto:stackless-bounces@stackless.com" target="_blank">stackless-bounces@stackless.com</a>] <b>On Behalf Of </b>Kristján Valur
Jónsson<br>
<b>Sent:</b> 12. október 2009 11:19<br>
<b>To:</b> Crispin Wellington; Richard Tew<div><div></div><div class="h5"><br>
<b>Cc:</b> <a href="mailto:stackless@stackless.com" target="_blank">stackless@stackless.com</a><br>
<b>Subject:</b> Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected
cell, found stackless._wrap.cell when unpickling stackless thread</div></div></span></p>
</div>
</div><div><div></div><div class="h5">
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Ok, I understand the issue.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Cell objects potentially can be part in a cycle. Therefore
for pickling, the three argument __reduce__ protocol is used, with an initial
‚new‘, followed by a ‚setstate‘ later.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Only in the ‚setstate‘ call will the type of the
object be changed from the special „_wrap“ type. But
precisely because a reference cycle is being constructed, a fresh cell is being
inserted into a function, before having had its state set, and the function
constructor balks at this.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">I see no way around this using the current wrapped pickle
technique except to fix the test in the function object. Once I have
created a simple regression test, I will submit a fix.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">K</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<div style="border-style: none none none solid; border-color: -moz-use-text-color -moz-use-text-color -moz-use-text-color blue; border-width: medium medium medium 1.5pt; padding: 0cm 0cm 0cm 4pt;">
<div>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span style="font-size: 10pt;" lang="EN-US">From:</span></b><span style="font-size: 10pt;" lang="EN-US"> <a href="mailto:stackless-bounces@stackless.com" target="_blank">stackless-bounces@stackless.com</a>
[mailto:<a href="mailto:stackless-bounces@stackless.com" target="_blank">stackless-bounces@stackless.com</a>] <b>On Behalf Of </b>Crispin Wellington<br>
<b>Sent:</b> 12. október 2009 05:27<br>
<b>To:</b> Richard Tew<br>
<b>Cc:</b> <a href="mailto:stackless@stackless.com" target="_blank">stackless@stackless.com</a><br>
<b>Subject:</b> Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected
cell, found stackless._wrap.cell when unpickling stackless thread</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="margin-bottom: 12pt;">OK. I boiled down the code to a
small snippet that triggers the bug. It is a bug that is exposed through my use
of Twisted. If i create a factory, and then two callbacks that are closures,
each callback setting thread level variables. Then I add the callbacks to the
factory's deferred, and then I sleep the threadlet forever.<br>
<br>
Initiate the tasklet, pickle it, then unpickle it, and BAM, you have your
error:<br>
<br>
$ /usr/local/stackless/bin/python break-stackless.py<br>
Traceback (most recent call last):<br>
File "break-stackless.py", line 47, in <module><br>
test()<br>
File "break-stackless.py", line 37, in test<br>
obj = pickle.loads(data)<br>
File "/usr/local/stackless/lib/python2.6/pickle.py", line
1409, in loads<br>
return Unpickler(file).load()<br>
File "/usr/local/stackless/lib/python2.6/pickle.py", line 893,
in load<br>
dispatch[key](self)<br>
File "/usr/local/stackless/lib/python2.6/pickle.py", line
1252, in load_build<br>
setstate(state)<br>
TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell<br>
<br>
The code is....<br>
<br>
-----------------------------<br>
<br>
import stackless<br>
import pickle<br>
<br>
from twisted.web import client<br>
<br>
def tasklet():<br>
# create an example state<br>
factory = client.HTTPClientFactory("<a href="http://www.google.com/" target="_blank">http://www.google.com/</a>", agent =
"stackless/1.0" )<br>
<br>
get_complete = [False]<br>
get_failed = [False]<br>
<br>
def _doFailure(data):<br>
get_failed[0] = factory.status<br>
<br>
def _doSuccess(data):<br>
get_complete[0] = factory.status<br>
<br>
factory.deferred.addCallback(_doSuccess).addErrback(_doFailure)<br>
<br>
while True:<br>
stackless.schedule()<br>
<br>
def test():<br>
# run the task<br>
task = stackless.tasklet(tasklet)<br>
task.setup()<br>
task.run()<br>
<br>
# pump once<br>
stackless.schedule()<br>
<br>
# now serialise<br>
data = pickle.dumps(task)<br>
<br>
# now deserialise<br>
obj = pickle.loads(data)<br>
<br>
# resume<br>
task.kill()<br>
obj.insert()<br>
<br>
# pump again<br>
stackless.schedule()<br>
<br>
if __name__=="__main__":<br>
test()<br>
<br>
-----------------------<br>
<br>
If code formatting is an issue, I've attached the code to the email as a
file...<br>
<br>
Can anyone shed any light on this bug?<br>
<br>
Kind Regards<br>
<br>
Crispin</p>
<div>
<p class="MsoNormal">On Sat, Oct 10, 2009 at 2:21 AM, Richard Tew <<a href="mailto:richard.m.tew@gmail.com" target="_blank">richard.m.tew@gmail.com</a>> wrote:</p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom: 12pt;">On Fri, Oct 9, 2009 at 10:38
PM, Crispin Wellington<br>
<<a href="mailto:retrogradeorbit@gmail.com" target="_blank">retrogradeorbit@gmail.com</a>>
wrote:<br>
> I have written a server that contains stackless threads. When the server<br>
> shuts down these threads are serialised to disk. This all happens without<br>
> incident and I end up with a reasonably sized file per pickled object.<br>
><br>
> When I go to deserialise them I hit a problem. I get this...<br>
> -------------------<br>
> File<br>
> "/home/cwellington/Development/yabi/yabi-be-twisted/trunk/TaskManager/Tasklets.py",<br>
> line 54, in load<br>
> task = pickle.loads(data)<br>
> File
"/usr/local/stackless/lib/python2.6/pickle.py", line 1415, in<br>
> loads<br>
> return Unpickler(file).load()<br>
> File "/usr/local/stackless/lib/python2.6/pickle.py",
line 898, in load<br>
> dispatch[key](self)<br>
> File
"/usr/local/stackless/lib/python2.6/pickle.py", line 1258, in<br>
> load_build<br>
> setstate(state)<br>
> exceptions.TypeError: arg 5 (closure) expected cell,
found<br>
> stackless._wrap.cell<br>
> --------------------<br>
><br>
> I think this might be a bug. The arguments are being passed back into the<br>
> constructor of the stackless._wrap.function and the constructor is
throwing<br>
> that exception. So in essence, the __reduce__ or __reduce_ex__ functions
are<br>
> creating instatiation values that do not work on reinstantiation.<br>
><br>
> I changed pickle.py to print out some debug. setstate is... <built-in
method<br>
> __setstate__ of stackless._wrap.function object at 0x21ca0c8> and
(state)[4]<br>
> is... (<cell at 0x2605f30: empty>, <cell at 0x2940718: list
object at<br>
> 0x259f518>, <cell at 0x2940670: str object at 0x2940768>,
<cell at<br>
> 0x2940750: str object at 0x29407d8>, <cell at 0x29407c0: int object
at<br>
> 0x25cd338>)<br>
><br>
> These all seem to be "cell" objects. Why aren't they
(stackless._wrap.cell)<br>
> recognised as "cells"?<br>
><br>
> I can provide a pickletools.dis() dump of the pickle stream if wanted.<br>
><br>
> Using Stackless Python 2.6.2 on 64 bit linux (x86_64).<br>
><br>
> Can anybody help me decode my objects? or even find a way to encode them
so<br>
> the are decodable? Is this a bug, or am I doing it wrong?</p>
</div>
</div>
<p class="MsoNormal">It definitely looks like a bug. If you can provide the
smallest<br>
possible reproduction case, it would help us look into it and we could<br>
add it to the test suite as a regression test.<br>
<br>
If you want your objects back in the meantime, you might try compiling<br>
2.6 with the relevant code raising these errors commented out.<br>
<span style="color: rgb(136, 136, 136);"><br>
Richard.</span></p>
</div>
<p class="MsoNormal"> </p>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br>