[Stackless] Fwd: deepcopying (pickling) channels

Kristján Valur Jónsson kristjan at ccpgames.com
Sat Oct 12 16:08:04 CEST 2013


Let me try to explain:
This has nothing to do with channels.  Stackless extends python by allowing execution frames to be pickled.
You can pickle a tasklet that is paused.  When you pickle a channel with a tasklet sleeping on it, the tasklet is also pickled.  So, stop thinking about channels, think execution frames.
For execution frames to be picklable, certain restrictions are in effect.  For example, the tasklet running it can only be pure python (no C state, etc.).  This is however, not the case here.
Everything else in a frame must be pickleable.  a 'frame' is the stack frame of your program.  It contains local variables, function arguments, and so on.
If anything on that frame is not-pickleable, then the frame as a whole is not picklable.

Python debuggers work by setting a callback hook to sys.settrace.  The function in there is called for every code line.
I don't know the exact details of how the wingware debugger works, but apparently it has inserted an extra execution frame for the tasklet, and, this frame contains an unpickleable object.

Wingware were probably not aware that execution frames ought to be pickleable.  Also, the debugger is an active "state" for the program.  pickling a running function under a debugger, does that not mean that you pickle the debugger state as well?
Anyway, we can probably approach them and explain the problem to them.  I could ask them for details of the module in question.  Maybe this is fixable.  But let's be clear that it is not a Stackless problem:  An object in an execution frame over which we have no problem, (an frame inserted into your program by your actively running debugger) is not pickleable.

By the way, I don't quite understand your problem.  If it can be solved by just setting channel=None, then it means that you really have no interest in the channel, or don't want to unpickle it again and continue sending to it.  And as such, leaving a blocked tasklet on the channel to me looks like a bug in your program.  If you really had no intention to unpickle this channel and continue to send to its blocked (and pickled-unpickled) tasklet, then you shouldn't be pickling it in the first place :)

K

From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of lars van Gemerden
Sent: 11. október 2013 18:33
To: The Stackless Python Mailing List
Subject: Re: [Stackless] Fwd: deepcopying (pickling) channels

So i can pickle tasklets/channels, but then i cannot debug my program ;-)

I don't know how debuggers work, but it seems still odd that channels that are sending/wating to receive cannot be pickled, but empty channels can (if you let the loops in the little demo end at the same count the object can be pickled in wing).

Is there an easy way to clear a channel that is sending/waiting?

Anyway, for now i just set create all channels before starting stackless.run and set them to None afterwards.

Cheers, Lars

On Fri, Oct 11, 2013 at 5:44 PM, Kristján Valur Jónsson <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:
Ok, this makes sense.  What you are seeing is the failure to pickle inaccessible functions inside the wingware debugger.  It is the debug callback frame that is failing, and this frame contains local variables, including "has_key" that point to anonymous functions.  Anonymous functions cannot (currently) be pickled, since all functions are (currently) pikled by name.
I suppose we _could_ add function recreation (i.e. pickle the code objects) in such cases....
But anyway, the reason you are failing is because you are running inside the debugger :)

K

From: stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com> [mailto:stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com>] On Behalf Of Kristján Valur Jónsson
Sent: 11. október 2013 14:48

To: The Stackless Python Mailing List
Subject: Re: [Stackless] Fwd: deepcopying (pickling) channels

Ok, running this in the wingware debugger, I see this is happening because it is trying to pickle the debugging frame!
I need to investigate more what Is happening, but functions cannot be pickled if they are not accessible by name, and some of the debugging code internal functions might be anonymous.
Stackless can pickle frames and functions, as long as those functions are pickleable by name, IIRC.

K

From: stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com> [mailto:stackless-bounces at stackless.com] On Behalf Of Kristján Valur Jónsson
Sent: 11. október 2013 11:21
To: The Stackless Python Mailing List
Subject: Re: [Stackless] Fwd: deepcopying (pickling) channels

"I run this with the default settings in wing IDE. If i run it in IDLE, there is no exception."
That sounds like the problem.
Boy, do I hate how hard it is to debug pickling problems.  for "pickle" we really should have a flag that prints out the hierarchy as it is pickling.
Maybe someone has already done that?

Anyway, I'll install Wingware and see if I can reproduce it there.  I'm sure wingware is messing with pickling in some unexpected way.
(I'm unfamiliar with wingware, btw.   If you feel up to it, try putting debugging code inside pickle.py to dump out what it is doing, that might be faster.  I would guess that it is doing something with  the copy_reg module and this is messing with our function pickling.

K



From: stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com> [mailto:stackless-bounces at stackless.com] On Behalf Of lars van Gemerden
Sent: 10. október 2013 15:24
To: The Stackless Python Mailing List
Subject: Re: [Stackless] Fwd: deepcopying (pickling) channels

the callstack is:

pickle.PicklingError: Can't pickle <type 'function'>: it's not found as __builtin__.function
File "d:\Documents\Code\python\floware\toolshed\tests\stackless_test.py", line 34, in <module>
  print pickle.dumps(t) #ERROR
File "C:\Python27\lib\pickle.py", line 1411, in dumps
  Pickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 231, in dump
  self.save(obj)
File "C:\Python27\lib\pickle.py", line 338, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 426, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 661, in save_dict
  self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems
  save(v)
File "C:\Python27\lib\pickle.py", line 338, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 426, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 569, in save_tuple
  save(element)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 607, in save_list
  self._batch_appends(iter(obj))
File "C:\Python27\lib\pickle.py", line 622, in _batch_appends
  save(x)
File "C:\Python27\lib\pickle.py", line 338, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 426, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 569, in save_tuple
  save(element)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 607, in save_list
  self._batch_appends(iter(obj))
File "C:\Python27\lib\pickle.py", line 622, in _batch_appends
  save(x)
File "C:\Python27\lib\pickle.py", line 338, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 426, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 569, in save_tuple
  save(element)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 661, in save_dict
  self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems
  save(v)                                                                               <== THIS LINE
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 802, in save_function
  return self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 426, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 569, in save_tuple
  save(element)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 661, in save_dict
  self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems
  save(v)
File "C:\Python27\lib\pickle.py", line 338, in save
  self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 426, in save_reduce
  save(state)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 661, in save_dict
  self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems
  save(v)
File "C:\Python27\lib\pickle.py", line 293, in save
  f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 760, in save_global
  (obj, module, name))

Probably not that usefull without values ... in THIS LINE above,  v is module C:\python27\lib\new.pyc.

I don't think there is a framework in place that would influence this script, the imports (pickle, stackless) are unaltered.

I run this with the default settings in wing IDE. If i run it in IDLE, there is no exception.

On Thu, Oct 10, 2013 at 3:22 PM, Kristján Valur Jónsson <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:
The error is weird.  It seems to be trying to pickle the "function" type.  It shouldn't try doing that.
Now, I know that pickling channels with tasklets on them works.  This is what stackless does.
I tried saving your code into a module and just running it from the command line.
I wonder if there is some copy_reg magic that is interfering with the code.  How are you running it?  Is this when running the code as a module, or when typing it in from the console?
Are you trying this in a stand-alone environment or in some application framework?

Finally, can I see the full traceback for the pickle exception.  If you are using pickle (and not cPickle) a traceback can help us find out why it is failing.

K

From: stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com> [mailto:stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com>] On Behalf Of lars van Gemerden
Sent: 9. október 2013 12:48

To: The Stackless Python Mailing List
Subject: Re: [Stackless] Fwd: deepcopying (pickling) channels

Hi Kristjan,

the version i am using is: python 2.7.5 Stackless 3.1b3 (from binary).

I think the stackless mailsystem was a little off, i tried with pickle.dumps (see mail history below), with the error: pickle.PicklingError: Can't pickle <type 'function'>: it's not found as __builtin__.function.

"stackless.run() returns when there are no runnable tasklets ...", ok, clear, thanks.

Cheers, Lars

On Wed, Oct 9, 2013 at 2:36 PM, lars van Gemerden <lars at rational-it.com<mailto:lars at rational-it.com>> wrote:
I am reasonably familiar with pickle (e.g. i know you cant pickle most functions), but this is pretty stackless specific: why can i pickle an empty channel, but not a sending/receiving channel?

Cheers, Lars

On Mon, Oct 7, 2013 at 7:50 PM, Anselm Kruis <a.kruis at science-computing.de<mailto:a.kruis at science-computing.de>> wrote:
Hi Lars,

you are observing well known deficiencies of the pickle/cPickle implementation of Python. Perhaps sPickle (http://pypi.python.org/pypi/sPickle) can pickle the channels.

Regards
  Anselm


Am 06.10.2013 08<tel:06.10.2013%2008>:58, schrieb lars van Gemerden:
maybe double but i got a mail bounce ..

---------- Forwarded message ----------
From: lars van Gemerden <lars at rational-it.com<mailto:lars at rational-it.com>>
Date: Thu, Oct 3, 2013 at 6:15 PM
Subject: Re: [Stackless] deepcopying (pickling) channels
To: The Stackless Python Mailing List <stackless at stackless.com<mailto:stackless at stackless.com>>


actually both give an error. The only thing different is the error message.
with pickle the code would be:

---------------------------------------------------------------------------
import stackless, pickle

class Test(object):
     def __init__(self):
         self.channel = stackless.channel()

     def run(self):
         stackless.tasklet(self.sender)()
         stackless.tasklet(self.receiver)()
         stackless.run()

     def sender(self):
         counter = 0
         while counter < 10:
             self.channel.send(counter)
             counter += 1

     def receiver(self):
         counter = 0
         while counter < 12:
             counter = self.channel.receive()
             print counter,
         print "done"

if __name__ == "__main__":

     t = Test()
     t.run()
     t.channel = None
     s = pickle.dumps(t) #OK

     t = Test()
     t.run()
     s = pickle.dumps(t) #ERROR
-----------------------------------------------------------------------------
and the error:

pickle.PicklingError: Can't pickle <type 'function'>: it's not found as
__builtin__.function

otherwise the error behavior is the same as decribed above.

Cheers, Lars

(as far as i have read and tested a bit, deepcopy allows more to be copied
then pickle, but cPickle is faster, so i have my own deepcopy:

def deepcopy(obj):
     try:
         return cPickle.loads(cPickle.dumps(obj, -1))
     except PicklingError:
         return copy.deepcopy(obj)
)


On Thu, Oct 3, 2013 at 5:19 AM, Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>> wrote:
I'm a little confused.  Are you pickling the tasklets, or copying
them?  These are different things to Python.

As far as I know using copy or deepcopy has never been supported for
tasklets and therefore channels.

Pickling however, is another matter.

Cheers,
Richard.

_______________________________________________
Stackless mailing list
Stackless at stackless.com<mailto:Stackless at stackless.com>
http://www.stackless.com/mailman/listinfo/stackless





maybe double but i got a mail bounce ..

---------- Forwarded message ----------
From: *lars van Gemerden* <lars at rational-it.com<mailto:lars at rational-it.com>
<mailto:lars at rational-it.com<mailto:lars at rational-it.com>>>
Date: Thu, Oct 3, 2013 at 6:15 PM
Subject: Re: [Stackless] deepcopying (pickling) channels
To: The Stackless Python Mailing List <stackless at stackless.com<mailto:stackless at stackless.com>
<mailto:stackless at stackless.com<mailto:stackless at stackless.com>>>


actually both give an error. The only thing different is the error
message. with pickle the code would be:

---------------------------------------------------------------------------
import stackless, pickle

class Test(object):
     def __init__(self):
         self.channel = stackless.channel()
     def run(self):
         stackless.tasklet(self.sender)()
         stackless.tasklet(self.receiver)()
         stackless.run()
     def sender(self):
         counter = 0
         while counter < 10:
             self.channel.send(counter)
             counter += 1
     def receiver(self):
         counter = 0
         while counter < 12:
             counter = self.channel.receive()
             print counter,
         print "done"
if __name__ == "__main__":
     t = Test()
     t.run()
     t.channel = None
     s = pickle.dumps(t) #OK
     t = Test()
     t.run()
     s = pickle.dumps(t) #ERROR
-----------------------------------------------------------------------------
and the error:

pickle.PicklingError: Can't pickle <type 'function'>: it's not found as
__builtin__.function

otherwise the error behavior is the same as decribed above.

Cheers, Lars

(as far as i have read and tested a bit, deepcopy allows more to be
copied then pickle, but cPickle is faster, so i have my own deepcopy:

def deepcopy(obj):
     try:
         return cPickle.loads(cPickle.dumps(obj, -1))
     except PicklingError:
         return copy.deepcopy(obj)
)


On Thu, Oct 3, 2013 at 5:19 AM, Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>
<mailto:richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>>> wrote:

    I'm a little confused.  Are you pickling the tasklets, or copying
    them?  These are different things to Python.

    As far as I know using copy or deepcopy has never been supported for
    tasklets and therefore channels.

    Pickling however, is another matter.

    Cheers,
    Richard.

    _______________________________________________
    Stackless mailing list
    Stackless at stackless.com<mailto:Stackless at stackless.com> <mailto:Stackless at stackless.com<mailto:Stackless at stackless.com>>

    http://www.stackless.com/mailman/listinfo/stackless




--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com> <mailto:lars at rational-it.com<mailto:lars at rational-it.com>>
+31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039> <tel:%2B31%206%2026%2088%2055%2039>

====================================



--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com> <mailto:lars at rational-it.com<mailto:lars at rational-it.com>>

+31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039>
====================================
_______________________________________________
Stackless mailing list
Stackless at stackless.com<mailto:Stackless at stackless.com>
http://www.stackless.com/mailman/listinfo/stackless

--
 Dipl. Phys. Anselm Kruis                       science + computing ag
 Senior Solution Architect                      Ingolstädter Str. 22
 email A.Kruis at science-computing.de<mailto:A.Kruis at science-computing.de>             80807 München, Germany
 phone +49 89 356386 874<tel:%2B49%2089%20356386%20874>  fax 737               www.science-computing.de<http://www.science-computing.de>
--
Vorstandsvorsitzender/Chairman of the board of management:
Gerd-Lothar Leonhart
Vorstand/Board of Management:
Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Arno Steitz, Dr. Ingrid Zech
Vorsitzender des Aufsichtsrats/
Chairman of the Supervisory Board:
Philippe Miltin
Sitz/Registered Office: Tuebingen
Registergericht/Registration Court: Stuttgart
Registernummer/Commercial Register No.: HRB 382196



_______________________________________________
Stackless mailing list
Stackless at stackless.com<mailto:Stackless at stackless.com>
http://www.stackless.com/mailman/listinfo/stackless



--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com>
+31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039>
====================================



--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com>
+31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039>
====================================

_______________________________________________
Stackless mailing list
Stackless at stackless.com<mailto:Stackless at stackless.com>
http://www.stackless.com/mailman/listinfo/stackless



--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com>
+31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039>
====================================

_______________________________________________
Stackless mailing list
Stackless at stackless.com<mailto:Stackless at stackless.com>
http://www.stackless.com/mailman/listinfo/stackless



--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com>
+31 6 26 88 55 39
====================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20131012/78d2debb/attachment-0001.html>


More information about the Stackless mailing list