[Stackless] exceptions.TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell when unpickling stackless thread

Kristján Valur Jónsson kristjan at ccpgames.com
Mon Oct 12 16:28:01 CEST 2009


This has been fixed in the stackless trunk, and merged into 2.6, 3.0, 3.1 and py3k in revisions 75377-75381
K
From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of Kristján Valur Jónsson
Sent: 12. október 2009 11:19
To: Crispin Wellington; Richard Tew
Cc: stackless at stackless.com
Subject: Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell when unpickling stackless thread

Ok, I understand the issue.
Cell objects potentially can be part in a cycle.  Therefore for pickling, the three argument __reduce__ protocol is used, with an initial 'new', followed by a 'setstate' later.
Only in the 'setstate' call will the type of the object be changed from the special "_wrap" type.  But precisely because a reference cycle is being constructed, a fresh cell is being inserted into a function, before having had its state set, and the function constructor balks at this.
I see no way around this using the current wrapped pickle technique except to fix the test in the function object.  Once I have created a simple regression test, I will submit a fix.

K

From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of Crispin Wellington
Sent: 12. október 2009 05:27
To: Richard Tew
Cc: stackless at stackless.com
Subject: Re: [Stackless] exceptions.TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell when unpickling stackless thread

OK. I boiled down the code to a small snippet that triggers the bug. It is a bug that is exposed through my use of Twisted. If i create a factory, and then two callbacks that are closures, each callback setting thread level variables. Then I add the callbacks to the factory's deferred, and then I sleep the threadlet forever.

Initiate the tasklet, pickle it, then unpickle it, and BAM, you have your error:

$ /usr/local/stackless/bin/python break-stackless.py
Traceback (most recent call last):
  File "break-stackless.py", line 47, in <module>
    test()
  File "break-stackless.py", line 37, in test
    obj =  pickle.loads(data)
  File "/usr/local/stackless/lib/python2.6/pickle.py", line 1409, in loads
    return Unpickler(file).load()
  File "/usr/local/stackless/lib/python2.6/pickle.py", line 893, in load
    dispatch[key](self)
  File "/usr/local/stackless/lib/python2.6/pickle.py", line 1252, in load_build
    setstate(state)
TypeError: arg 5 (closure) expected cell, found stackless._wrap.cell

The code is....

-----------------------------

import stackless
import pickle

from twisted.web import client

def tasklet():
    # create an example state
    factory = client.HTTPClientFactory("http://www.google.com/", agent = "stackless/1.0" )

    get_complete = [False]
    get_failed = [False]

    def _doFailure(data):
        get_failed[0] = factory.status

    def _doSuccess(data):
        get_complete[0] = factory.status

    factory.deferred.addCallback(_doSuccess).addErrback(_doFailure)

    while True:
        stackless.schedule()

def test():
    # run the task
    task = stackless.tasklet(tasklet)
    task.setup()
    task.run()

    # pump once
    stackless.schedule()

    # now serialise
    data = pickle.dumps(task)

    # now deserialise
    obj =  pickle.loads(data)

    # resume
    task.kill()
    obj.insert()

    # pump again
    stackless.schedule()

if __name__=="__main__":
    test()

-----------------------

If code formatting is an issue, I've attached the code to the email as a file...

Can anyone shed any light on this bug?

Kind Regards

Crispin
On Sat, Oct 10, 2009 at 2:21 AM, Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>> wrote:
On Fri, Oct 9, 2009 at 10:38 PM, Crispin Wellington
<retrogradeorbit at gmail.com<mailto:retrogradeorbit at gmail.com>> wrote:
> I have written a server that contains stackless threads. When the server
> shuts down these threads are serialised to disk. This all happens without
> incident and I end up with a reasonably sized file per pickled object.
>
> When I go to deserialise them I hit a problem. I get this...
> -------------------
>       File
> "/home/cwellington/Development/yabi/yabi-be-twisted/trunk/TaskManager/Tasklets.py",
> line 54, in load
>         task = pickle.loads(data)
>       File "/usr/local/stackless/lib/python2.6/pickle.py", line 1415, in
> loads
>         return Unpickler(file).load()
>       File "/usr/local/stackless/lib/python2.6/pickle.py", line 898, in load
>         dispatch[key](self)
>       File "/usr/local/stackless/lib/python2.6/pickle.py", line 1258, in
> load_build
>         setstate(state)
>     exceptions.TypeError: arg 5 (closure) expected cell, found
> stackless._wrap.cell
> --------------------
>
> I think this might be a bug. The arguments are being passed back into the
> constructor of the stackless._wrap.function and the constructor is throwing
> that exception. So in essence, the __reduce__ or __reduce_ex__ functions are
> creating instatiation values that do not work on reinstantiation.
>
> I changed pickle.py to print out some debug. setstate is... <built-in method
> __setstate__ of stackless._wrap.function object at 0x21ca0c8> and (state)[4]
> is... (<cell at 0x2605f30: empty>, <cell at 0x2940718: list object at
> 0x259f518>, <cell at 0x2940670: str object at 0x2940768>, <cell at
> 0x2940750: str object at 0x29407d8>, <cell at 0x29407c0: int object at
> 0x25cd338>)
>
> These all seem to be "cell" objects. Why aren't they (stackless._wrap.cell)
> recognised as "cells"?
>
> I can provide a pickletools.dis() dump of the pickle stream if wanted.
>
> Using Stackless Python 2.6.2 on 64 bit linux (x86_64).
>
> Can anybody help me decode my objects? or even find a way to encode them so
> the are decodable? Is this a bug, or am I doing it wrong?
It definitely looks like a bug.  If you can provide the smallest
possible reproduction case, it would help us look into it and we could
add it to the test suite as a regression test.

If you want your objects back in the meantime, you might try compiling
2.6 with the relevant code raising these errors commented out.

Richard.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20091012/84bcf91e/attachment.htm>


More information about the Stackless mailing list