[Stackless] Pickling and recursion

[Lars van Gemerden]

Thank you,

I'll do nothing in my code for the moment (working on a prototype). By the time i get closer to production i'll do some calculation and set recursion limit accordingly. 

Do you know by chance whether it is possible to change the python recursionlimit dynamically (mid-run)? I could then catch the exception, log the exception, double the limit, call pickle again ( for now).

Cheers, lars

====================================
Lars van Gemerden
lars at rational-it.com
+31 6 26 88 55 39
====================================

On 3 okt. 2012, at 15:25, Kristján Valur Jónsson <kristjan at ccpgames.com> wrote:

> Hello Lars.
> First off, deep recursion isn’t really a problem if you are using a single thread, since most operating systems will just allocate more stack as you need it.
> In the unlikely situation that you get into OS problems, though, this is when Stackless will help more than regular python will.
>  
> 1)      for pickle, just make sure that your sys.setrecursionlimit() is high enough.  for cPickle you don’t need any such thing.
> 2)      Nothing mitigates pytyon’s recursionlimit.  Stackless’ stack spilling is a behind-the-scenes way of trimming the c stack, that is all.  But python’s recursionlimit only applies to regular python code.  C code is unarffercted by it and cPickle is c code.
> 3)      python 2 and 3 are the same, except that in 3, “pickle” is implemented in C and “_pickle” is a pure python reference implementation.
> 4)      Nothing special, it is the same as using the regular (non-C) pickle module.  Make sure that sys.setrecursionlimit() is set high.
>  
> So, to be clear:  sys.*recursionlimit is really an artificial construct in python, there to help avoid infinite recursion.  Normally, you just increase it to help with your deep recursion, the OS then allocates stack space.  Stackless is the same, but it will try to use less OS stack space.  This will help on “threads” (that have fixed stacks, normally) and on systems that run out of c stack easily.
>  
> K
>  
> From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of lars van Gemerden
> Sent: 3. október 2012 11:44
> To: The Stackless Python Mailing List
> Subject: Re: [Stackless] Pickling and recursion
>  
> Hi Kristjan,
>  
> Thanks for your response, it is somewhat clearer now. However I have some more questions about the subject (because some of my data structures are very recursive and redesigning them would be a lot of work):
>  
> - under what conditions can stackless mitigate these recursion limits (for pickle and cPickle, what do i need to avoid)?
> - does this mitigation circumvent pythons recursion limit or does it only kick in after the python recursion limit has been disabled?
> - i am working on stackless python 2.7, what are the differences with stackless python 3 in this respect?
> - What happens if i write my own highly recursive persistence module (i wrote one that converts most python objects to XML via the etree module)? 
>  
> This is far from the only reason I use stackless, but it would be a great bonus if i could be reasonable sure that i won't run into recursion problems in the near future (this has been one of those nagging uncertainties in the back of my mind).
>  
> Cheers, Lars
>  
>  
>  
> On Tue, Oct 2, 2012 at 11:34 AM, Kristján Valur Jónsson <kristjan at ccpgames.com> wrote:
> Not necessarily.
> the term "stackless" is historic in this respect.  Originally, the attempt was made to have pure python code work independently of the C stack.
> In practice, this change was simply too big to be maintainable and so, nowadays, stackless does try to do this when possible, but otherwise relents.
> Pickling in version 3 of python is by default done using C code, and C code normally also uses the c stack for recursion.
> 
> Two opposite caveats exists:
> 1) Stackless does attempt to mitigate C recursion by using "stack spilling" where very deep c invocation stacks are stored away on the heap.  This process is not foolproof and can become degenerated, but it is designed to work with regular python function invocation and pickling.
> 2) Python imposes its own recursion limit.  This has to be disabled to go very deep.  But again, this does not apply to c pickling.
> 
> So no, stackless python does not promise stackless python evaluation in the literal sense, but it might help you for very deep pickling.  Since it is not the primary purpose of Stackless, your mileage may vary.
> K
> 
> > -----Original Message-----
> > From: stackless-bounces at stackless.com [mailto:stackless-
> > bounces at stackless.com] On Behalf Of Lars van Gemerden
> > Sent: 1. október 2012 08:16
> > To: stackless at stackless.com
> > Subject: [Stackless] Pickling and recursion
> >
> > Hello,
> >
> > I was wandering whether using stackless also means that there is (practically)
> > no limit to the recursion depth of highly recursive objects when using
> > pickle/cPickle?
> >
> > Cheers, Lars
> >
> > ====================================
> > Lars van Gemerden
> > lars at rational-it.com
> > +31 6 26 88 55 39
> > ====================================
> > _______________________________________________
> > Stackless mailing list
> > Stackless at stackless.com
> > http://www.stackless.com/mailman/listinfo/stackless
> 
> 
> 
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
> 
> 
>  
> -- 
> ====================================
> Lars van Gemerden
> lars at rational-it.com
> +31 6 26 88 55 39
> ====================================
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20121007/e97694d3/attachment.html>