[Stackless] Fwd: deepcopying (pickling) channels
lars van Gemerden
lars at rational-it.com
Fri Oct 11 20:33:04 CEST 2013
So i can pickle tasklets/channels, but then i cannot debug my program ;-)
I don't know how debuggers work, but it seems still odd that channels that
are sending/wating to receive cannot be pickled, but empty channels can (if
you let the loops in the little demo end at the same count the object can
be pickled in wing).
Is there an easy way to clear a channel that is sending/waiting?
Anyway, for now i just set create all channels before starting
stackless.run and set them to None afterwards.
Cheers, Lars
On Fri, Oct 11, 2013 at 5:44 PM, Kristján Valur Jónsson <
kristjan at ccpgames.com> wrote:
> Ok, this makes sense. What you are seeing is the failure to pickle
> inaccessible functions inside the wingware debugger. It is the debug
> callback frame that is failing, and this frame contains local variables,
> including “has_key” that point to anonymous functions. Anonymous functions
> cannot (currently) be pickled, since all functions are (currently) pikled
> by name.****
>
> I suppose we _*could*_ add function recreation (i.e. pickle the code
> objects) in such cases….****
>
> But anyway, the reason you are failing is because you are running inside
> the debugger J****
>
> ** **
>
> K****
>
> ** **
>
> *From:* stackless-bounces at stackless.com [mailto:
> stackless-bounces at stackless.com] *On Behalf Of *Kristján Valur Jónsson
> *Sent:* 11. október 2013 14:48
>
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] Fwd: deepcopying (pickling) channels****
>
> ** **
>
> Ok, running this in the wingware debugger, I see this is happening because
> it is trying to pickle the debugging frame!****
>
> I need to investigate more what Is happening, but functions cannot be
> pickled if they are not accessible by name, and some of the debugging code
> internal functions might be anonymous.****
>
> Stackless can pickle frames and functions, as long as those functions are
> pickleable by name, IIRC.****
>
> ****
>
> K****
>
> ****
>
> *From:* stackless-bounces at stackless.com [
> mailto:stackless-bounces at stackless.com <stackless-bounces at stackless.com>]
> *On Behalf Of *Kristján Valur Jónsson
> *Sent:* 11. október 2013 11:21
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] Fwd: deepcopying (pickling) channels****
>
> ****
>
> “I run this with the default settings in wing IDE. If i run it in IDLE,
> there is no exception.”****
>
> That sounds like the problem.****
>
> Boy, do I hate how hard it is to debug pickling problems. for “pickle” we
> really should have a flag that prints out the hierarchy as it is pickling.
> ****
>
> Maybe someone has already done that?****
>
> ****
>
> Anyway, I’ll install Wingware and see if I can reproduce it there. I’m
> sure wingware is messing with pickling in some unexpected way.****
>
> (I’m unfamiliar with wingware, btw. If you feel up to it, try putting
> debugging code inside pickle.py to dump out what it is doing, that might be
> faster. I would guess that it is doing something with the copy_reg module
> and this is messing with our function pickling.****
>
> ****
>
> K****
>
> ****
>
> ****
>
> ****
>
> *From:* stackless-bounces at stackless.com [
> mailto:stackless-bounces at stackless.com <stackless-bounces at stackless.com>]
> *On Behalf Of *lars van Gemerden
> *Sent:* 10. október 2013 15:24
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] Fwd: deepcopying (pickling) channels****
>
> ****
>
> the callstack is:****
>
> ****
>
> pickle.PicklingError: Can't pickle <type 'function'>: it's not found as
> __builtin__.function****
>
> File "d:\Documents\Code\python\floware\toolshed\tests\stackless_test.py",
> line 34, in <module>****
>
> print pickle.dumps(t) #ERROR****
>
> File "C:\Python27\lib\pickle.py", line 1411, in dumps****
>
> Pickler(file, protocol).dump(obj)****
>
> File "C:\Python27\lib\pickle.py", line 231, in dump****
>
> self.save(obj)****
>
> File "C:\Python27\lib\pickle.py", line 338, in save****
>
> self.save_reduce(obj=obj, *rv)****
>
> File "C:\Python27\lib\pickle.py", line 426, in save_reduce****
>
> save(state)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 661, in save_dict****
>
> self._batch_setitems(obj.iteritems())****
>
> File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems****
>
> save(v)****
>
> File "C:\Python27\lib\pickle.py", line 338, in save****
>
> self.save_reduce(obj=obj, *rv)****
>
> File "C:\Python27\lib\pickle.py", line 426, in save_reduce****
>
> save(state)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 569, in save_tuple****
>
> save(element)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 607, in save_list****
>
> self._batch_appends(iter(obj))****
>
> File "C:\Python27\lib\pickle.py", line 622, in _batch_appends****
>
> save(x)****
>
> File "C:\Python27\lib\pickle.py", line 338, in save****
>
> self.save_reduce(obj=obj, *rv)****
>
> File "C:\Python27\lib\pickle.py", line 426, in save_reduce****
>
> save(state)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 569, in save_tuple****
>
> save(element)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 607, in save_list****
>
> self._batch_appends(iter(obj))****
>
> File "C:\Python27\lib\pickle.py", line 622, in _batch_appends****
>
> save(x)****
>
> File "C:\Python27\lib\pickle.py", line 338, in save****
>
> self.save_reduce(obj=obj, *rv)****
>
> File "C:\Python27\lib\pickle.py", line 426, in save_reduce****
>
> save(state)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 569, in save_tuple****
>
> save(element)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 661, in save_dict****
>
> self._batch_setitems(obj.iteritems())****
>
> File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems****
>
> save(v)
> <== THIS LINE****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 802, in save_function****
>
> return self.save_reduce(obj=obj, *rv)****
>
> File "C:\Python27\lib\pickle.py", line 426, in save_reduce****
>
> save(state)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 569, in save_tuple****
>
> save(element)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 661, in save_dict****
>
> self._batch_setitems(obj.iteritems())****
>
> File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems****
>
> save(v)****
>
> File "C:\Python27\lib\pickle.py", line 338, in save****
>
> self.save_reduce(obj=obj, *rv)****
>
> File "C:\Python27\lib\pickle.py", line 426, in save_reduce****
>
> save(state)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 661, in save_dict****
>
> self._batch_setitems(obj.iteritems())****
>
> File "C:\Python27\lib\pickle.py", line 675, in _batch_setitems****
>
> save(v)****
>
> File "C:\Python27\lib\pickle.py", line 293, in save****
>
> f(self, obj) # Call unbound method with explicit self****
>
> File "C:\Python27\lib\pickle.py", line 760, in save_global****
>
> (obj, module, name))****
>
> ****
>
> Probably not that usefull without values ... in THIS LINE above, v is
> module C:\python27\lib\new.pyc.****
>
> ****
>
> I don't think there is a framework in place that would influence this
> script, the imports (pickle, stackless) are unaltered.****
>
> ****
>
> I run this with the default settings in wing IDE. If i run it in IDLE,
> there is no exception.****
>
> ****
>
> On Thu, Oct 10, 2013 at 3:22 PM, Kristján Valur Jónsson <
> kristjan at ccpgames.com> wrote:****
>
> The error is weird. It seems to be trying to pickle the “function” type.
> It shouldn’t try doing that.****
>
> Now, I know that pickling channels with tasklets on them works. This is
> what stackless does.****
>
> I tried saving your code into a module and just running it from the
> command line.****
>
> I wonder if there is some copy_reg magic that is interfering with the
> code. How are you running it? Is this when running the code as a module,
> or when typing it in from the console?****
>
> Are you trying this in a stand-alone environment or in some application
> framework?****
>
> ****
>
> Finally, can I see the full traceback for the pickle exception. If you
> are using pickle (and not cPickle) a traceback can help us find out why it
> is failing.****
>
> ****
>
> K****
>
> ****
>
> *From:* stackless-bounces at stackless.com [mailto:
> stackless-bounces at stackless.com] *On Behalf Of *lars van Gemerden
> *Sent:* 9. október 2013 12:48****
>
>
> *To:* The Stackless Python Mailing List****
>
> *Subject:* Re: [Stackless] Fwd: deepcopying (pickling) channels****
>
> ****
>
> Hi Kristjan,****
>
> ****
>
> the version i am using is: python 2.7.5 Stackless 3.1b3 (from binary).****
>
> ****
>
> I think the stackless mailsystem was a little off, i tried with
> pickle.dumps (see mail history below), with the error: pickle.PicklingError:
> Can't pickle <type 'function'>: it's not found as __builtin__.function.***
> *
>
> ****
>
> "stackless.run() returns when there are no *runnable* tasklets ...", ok,
> clear, thanks.****
>
> ****
>
> Cheers, Lars****
>
> ****
>
> On Wed, Oct 9, 2013 at 2:36 PM, lars van Gemerden <lars at rational-it.com>
> wrote:****
>
> I am reasonably familiar with pickle (e.g. i know you cant pickle most
> functions), but this is pretty stackless specific: why can i pickle an
> empty channel, but not a sending/receiving channel?****
>
> ****
>
> Cheers, Lars****
>
> ****
>
> On Mon, Oct 7, 2013 at 7:50 PM, Anselm Kruis <a.kruis at science-computing.de>
> wrote:****
>
> Hi Lars,
>
> you are observing well known deficiencies of the pickle/cPickle
> implementation of Python. Perhaps sPickle (
> http://pypi.python.org/pypi/sPickle) can pickle the channels.
>
> Regards
> Anselm
>
>
> Am 06.10.2013 08:58, schrieb lars van Gemerden:****
>
> maybe double but i got a mail bounce ..
>
> ---------- Forwarded message ----------
> From: lars van Gemerden <lars at rational-it.com>
> Date: Thu, Oct 3, 2013 at 6:15 PM
> Subject: Re: [Stackless] deepcopying (pickling) channels
> To: The Stackless Python Mailing List <stackless at stackless.com>
>
>
> actually both give an error. The only thing different is the error message.
> with pickle the code would be:
>
> ---------------------------------------------------------------------------
> import stackless, pickle
>
> class Test(object):
> def __init__(self):
> self.channel = stackless.channel()
>
> def run(self):
> stackless.tasklet(self.sender)()
> stackless.tasklet(self.receiver)()
> stackless.run()
>
> def sender(self):
> counter = 0
> while counter < 10:
> self.channel.send(counter)
> counter += 1
>
> def receiver(self):
> counter = 0
> while counter < 12:
> counter = self.channel.receive()
> print counter,
> print "done"
>
> if __name__ == "__main__":
>
> t = Test()
> t.run()
> t.channel = None
> s = pickle.dumps(t) #OK
>
> t = Test()
> t.run()
> s = pickle.dumps(t) #ERROR
>
> -----------------------------------------------------------------------------
> and the error:
>
> pickle.PicklingError: Can't pickle <type 'function'>: it's not found as
> __builtin__.function
>
> otherwise the error behavior is the same as decribed above.
>
> Cheers, Lars
>
> (as far as i have read and tested a bit, deepcopy allows more to be copied
> then pickle, but cPickle is faster, so i have my own deepcopy:
>
> def deepcopy(obj):
> try:
> return cPickle.loads(cPickle.dumps(obj, -1))
> except PicklingError:
> return copy.deepcopy(obj)
> )
>
>
> On Thu, Oct 3, 2013 at 5:19 AM, Richard Tew <richard.m.tew at gmail.com>
> wrote:****
>
> I'm a little confused. Are you pickling the tasklets, or copying
> them? These are different things to Python.
>
> As far as I know using copy or deepcopy has never been supported for
> tasklets and therefore channels.
>
> Pickling however, is another matter.
>
> Cheers,
> Richard.
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless****
>
>
>
>
>
>
> maybe double but i got a mail bounce ..
>
> ---------- Forwarded message ----------****
>
> From: *lars van Gemerden* <lars at rational-it.com
> <mailto:lars at rational-it.com>>
> Date: Thu, Oct 3, 2013 at 6:15 PM
> Subject: Re: [Stackless] deepcopying (pickling) channels
> To: The Stackless Python Mailing List <stackless at stackless.com****
>
> <mailto:stackless at stackless.com>>
>
>
> actually both give an error. The only thing different is the error
> message. with pickle the code would be:
>
> ---------------------------------------------------------------------------
> import stackless, pickle
>
> class Test(object):
> def __init__(self):
> self.channel = stackless.channel()
> def run(self):
> stackless.tasklet(self.sender)()
> stackless.tasklet(self.receiver)()
> stackless.run()
> def sender(self):
> counter = 0
> while counter < 10:
> self.channel.send(counter)
> counter += 1
> def receiver(self):
> counter = 0
> while counter < 12:
> counter = self.channel.receive()
> print counter,
> print "done"
> if __name__ == "__main__":
> t = Test()
> t.run()
> t.channel = None
> s = pickle.dumps(t) #OK
> t = Test()
> t.run()
> s = pickle.dumps(t) #ERROR
>
> -----------------------------------------------------------------------------
> and the error:
>
> pickle.PicklingError: Can't pickle <type 'function'>: it's not found as
> __builtin__.function
>
> otherwise the error behavior is the same as decribed above.
>
> Cheers, Lars
>
> (as far as i have read and tested a bit, deepcopy allows more to be
> copied then pickle, but cPickle is faster, so i have my own deepcopy:
>
> def deepcopy(obj):
> try:
> return cPickle.loads(cPickle.dumps(obj, -1))
> except PicklingError:
> return copy.deepcopy(obj)
> )
>
>
> On Thu, Oct 3, 2013 at 5:19 AM, Richard Tew <richard.m.tew at gmail.com****
>
> <mailto:richard.m.tew at gmail.com>> wrote:
>
> I'm a little confused. Are you pickling the tasklets, or copying
> them? These are different things to Python.
>
> As far as I know using copy or deepcopy has never been supported for
> tasklets and therefore channels.
>
> Pickling however, is another matter.
>
> Cheers,
> Richard.
>
> _______________________________________________
> Stackless mailing list****
>
> Stackless at stackless.com <mailto:Stackless at stackless.com>****
>
>
> http://www.stackless.com/mailman/listinfo/stackless
>
>
>
>
> --
> ====================================
> Lars van Gemerden****
>
> lars at rational-it.com <mailto:lars at rational-it.com>
> +31 6 26 88 55 39 <tel:%2B31%206%2026%2088%2055%2039<%2B31%206%2026%2088%2055%2039>
> >****
>
>
> ====================================
>
>
>
> --
> ====================================
> Lars van Gemerden****
>
> lars at rational-it.com <mailto:lars at rational-it.com>****
>
>
> +31 6 26 88 55 39
> ====================================****
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless****
>
>
> --
> Dipl. Phys. Anselm Kruis science + computing ag
> Senior Solution Architect Ingolstädter Str. 22
> email A.Kruis at science-computing.de 80807 München, Germany
> phone +49 89 356386 874 fax 737 www.science-computing.de
> --
> Vorstandsvorsitzender/Chairman of the board of management:
> Gerd-Lothar Leonhart
> Vorstand/Board of Management:
> Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Arno Steitz, Dr. Ingrid Zech
> Vorsitzender des Aufsichtsrats/
> Chairman of the Supervisory Board:
> Philippe Miltin
> Sitz/Registered Office: Tuebingen
> Registergericht/Registration Court: Stuttgart
> Registernummer/Commercial Register No.: HRB 382196****
>
>
>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless****
>
>
>
> ****
>
> ****
>
> --
> ====================================
> Lars van Gemerden
> lars at rational-it.com
> +31 6 26 88 55 39
> ==================================== ****
>
>
>
> ****
>
> ****
>
> --
> ====================================
> Lars van Gemerden
> lars at rational-it.com
> +31 6 26 88 55 39
> ==================================== ****
>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless****
>
>
>
> ****
>
> ****
>
> --
> ====================================
> Lars van Gemerden
> lars at rational-it.com
> +31 6 26 88 55 39
> ==================================== ****
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>
--
====================================
Lars van Gemerden
lars at rational-it.com
+31 6 26 88 55 39
====================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20131011/7656261e/attachment-0001.html>
More information about the Stackless
mailing list