[Stackless] Windows 2.7.6 stability

Kristján Valur Jónsson kristjan at ccpgames.com
Thu Feb 6 11:40:50 CET 2014


I've updated the code.
Flags are now set only when required, not on all types wholesale.
It is possible to disable this with a  new macro, STACKLESS_NO_TYPEINFO
See how this affects your crash scenario

(still runs all the stackless unittests.  with the STACKLESS_NO_TYPEINFO == 1 flag, it fails in the expected places when checking for type flags.
K

> -----Original Message-----
> From: stackless-bounces at stackless.com [mailto:stackless-
> bounces at stackless.com] On Behalf Of Kristján Valur Jónsson
> Sent: 6. febrúar 2014 09:28
> To: The Stackless Python Mailing List
> Subject: Re: [Stackless] Windows 2.7.6 stability
> 
> I'm trying to familiarize myself with the code too :)
> 
> There are two inheritance scenarios:
> 1) when a non-heap class inherits from a base class.  It then gets copies of
> the slots from the base.
> 2) When a heap class inherits from a base class.  It then gets copies of the
> slots from the base, but after any local slots have been filled with custom
> __next__ etc.
> 
> Having looked at the code some more and tried to understand all this
> "stackless" flag passing, I've come to the conclusion that
> 1) It is okay if we miss the 'can do stackless' flag on a slot.  if a stackles-aware
> function is called with 'stackless==0' it just behaves as normal.  So, passing
> the correct stackless flags for the slots is good, since it enables the stackless
> performance enhancement, but it is not strictly necessary.  In other words,
> we would get away with never copying this property.  On the other hand we
> can also be more clever in copying the flags, particulary for external types
> that inherit from one of the builtins....
> 3) there is only one flag per method so we might use a bitfield.
> 4) Only a handful of types actually have the stackless property, that is, have
> slots that can do stackless.  It is possible to optimize stuff greatly by adding
> the stackless flag only to those particular ones during initialization.
> 
> 5) This change causes a binary incompatibility with external modules that
> were compiled using the stackless headers previously, since they would use
> an extended type object.
> 
> I think 5 is the only possible cause for concern here.  For that reason, we
> _may_ want to employ a new flag, one that means 'extensions are in the
> Mapping member', and ignore the old one, but only if you think this is a
> concern.
> 
> I'M going to make a new version of this now, one that isn't as intrusive by
> selectively patching only those types that we deem matter....  Using that
> paradigm, we may, for example, disable the stackless flag feature altogether
> at compile time, and then we can see if it has anything to do at all with the
> PySide crashes.
> 
> K
> 
> 
> 
> > -----Original Message-----
> > From: stackless-bounces at stackless.com [mailto:stackless-
> > bounces at stackless.com] On Behalf Of Richard Tew
> > Sent: 6. febrúar 2014 01:46
> > To: The Stackless Python Mailing List
> > Subject: Re: [Stackless] Windows 2.7.6 stability
> >
> > Where do types inherit a bunch of slots from the builtin type?  It's
> > been a while since I had to be familiar with that part of the code.
> >
> > Should your changes have resulted in something that crashes harder and
> > faster (and more consistently) than the old heap type version?
> >
> > Cheers,
> > Richard.
> >
> > On 2/6/14, Kristján Valur Jónsson <kristjan at ccpgames.com> wrote:
> > > I think that the problem may be a bit more insidious than just the
> > > size of the PyTypeObject.
> > > Imagine this scenario:
> > > An extension creates a C extension of a builtin type.  This
> > > extension is compiled against vanilla Python.  So, its own default
> > > flags don't have the STACKLESS flag.  But it does inherit a bunch of
> > > slots from the builtin stackless object, and those are defined to be
> > > STACKLESS aware.
> > >
> > > so, stackless will call those inherited slots thinking they don't
> > > adhere to the STACKLESS protocol.
> > >
> > > Is it safe to call "stackless" type calls using the old method?  Or
> > > is this purely an "added extra", so that the new, stackless aware,
> > > slot methods _can_ work with the old calling mechanism?  I don't
> > > know.  Hm, perhaps they do....  In which case, the stackless flag is
> > > an opt-in thing.  If it is missing, even for a stackless method, then it would
> be ok.....
> > >
> > >
> > > I think that with the method here, of having the STACKLESS flag
> > > indicate that there is extra data in the PyMappingMethods structure,
> > > we can in fact apply the STACKLESS flag to any inherited type, which
> > > calls PyType_Ready().  If the parent type has the STACKLESS flag, we
> > > can modify its tp_as_mapping member to point to a dynamically
> > > allocated, larger, structure, and then proceed to do inheritance correctly.
> > >
> > > So, we can inherit the stackless flag for subtypes that were
> > > compiled unaware of stackless.  If PySide were for example to create
> > > a subtype of PyFunctionType, then we could decorate it with the
> > > STACKLESS flag and have it inherit the stackless slot flags, by
> > > giving the subtype a new tp_as_mapping member.
> > >
> > > K
> > >
> > >
> > > ________________________________________
> > > From: stackless-bounces at stackless.com
> > > [stackless-bounces at stackless.com] on behalf of Richard Tew
> > > [richard.m.tew at gmail.com]
> > > Sent: Wednesday, February 05, 2014 19:58
> > > To: The Stackless Python Mailing List
> > > Subject: Re: [Stackless] Windows 2.7.6 stability
> > >
> > > On one hand I wonder why we just didn't do this long ago.
> > >
> > > On the other, it crashes :-)  I'll look into it later as I am
> > > celebrating Waitangi Day in the traditional way of sitting around
> > > doing nothing.
> > >
> > > When this is working, I'll rerelease 2.7.6 and try and get people to
> > > rebuild all the binaries.
> > >
> > >>       python27.dll!getset_get(PyGetSetDescrObject *
> > >> descr=0x0056a1e8, _object * obj=0x03848b98, _object *
> > >> type=0x00000000)  Line 147 + 0xa
> > bytes
> > >>      C
> > >         python27.dll!PyObject_Call(_object * func=0x0056a1e8, _object *
> > > arg=0x03848b98, _object * kw=0x00000000)  Line 2539 + 0x1c bytes        C
> > >         python27.dll!PyObject_CallFunctionObjArgs(_object *
> > > callable=0x0056a1e8, ...)  Line 2786 + 0x13 bytes       C
> > >         python27.dll!build_class(_object * methods=0x0384c390, _object *
> > > bases=0x1e0efddf, _object * name=0x037ba270)  Line 5117 + 0xf bytes
> C
> > >         python27.dll!PyEval_EvalFrame_value(_frame * f=0x00502428, int
> > > throwflag=58434160, _object * retval=0x0384c390)  Line 3394     C
> > >         python27.dll!PyEval_EvalFrameEx_slp(_frame * f=0x00502428, int
> > > throwflag=0, _object * retval=0x1e234ff4)  Line 964 + 0x17 bytes        C
> > >         python27.dll!slp_eval_frame_newstack(_frame * f=0x00502428, int
> > > exc=0, _object * retval=0x1e234ff4)  Line 521 + 0x1b bytes      C
> > >         python27.dll!PyEval_EvalFrameEx_slp(_frame * f=0x00502428, int
> > > throwflag=0, _object * retval=0x1e234ff4)  Line 964     C
> > >         python27.dll!slp_frame_dispatch_top(_object * retval=0x1e234ff4)
> > > Line 810 + 0x9 bytes    C
> > >         python27.dll!slp_run_tasklet(_frame * f=0x00502428)  Line 1504  C
> > >         python27.dll!slp_eval_frame(_frame * f=0x00502428)  Line 317 + 0xa
> > > bytes        C
> > >         python27.dll!climb_stack_and_eval_frame(_frame *
> > > f=0x00502428) Line
> > > 274 + 0x9 bytes C
> > >         python27.dll!slp_eval_frame(_frame * f=0x00502428)  Line 303 + 0x6
> > > bytes        C
> > >         python27.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x022faa40,
> > > _object
> > > * globals=0x00502428, _object * locals=0x004bc930, _object * *
> > > args=0x00000000, int argcount=0, _object * * kws=0x00000000, int
> > > kwcount=0, _object * * defs=0x00000000, int defcount=0, _object *
> > > closure=0x00000000)  Line 3697 + 0x6 bytes      C
> > >         python27.dll!PyEval_EvalCode(PyCodeObject * co=0x022faa40,
> > > _object
> > > *
> > > globals=0x004bc930, _object * locals=0x004bc930)  Line 674 + 0x22
> > > bytes   C
> > >         python27.dll!run_mod(_mod * mod=0x023613e8, const char *
> > > filename=0x00000000, _object * globals=0x004bc930, _object *
> > > locals=0x004bc930, PyCompilerFlags * flags=0x00000000, _arena *
> > > arena=0x00000000)  Line 1408 + 0x16 bytes       C
> > >         python27.dll!PyRun_FileExFlags(_iobuf * fp=0x73247408, const
> > > char * filename=0x00354be4, int start=257, _object *
> > > globals=0x004bc930, _object * locals=0x004bc930, int closeit=1,
> > PyCompilerFlags *
> > > flags=0x0027ff00)  Line 1393    C
> > >         python27.dll!PyRun_SimpleFileExFlags(_iobuf * fp=0x73247408,
> > > const char * filename=0x00354be4, int closeit=1, PyCompilerFlags *
> > > flags=0x0027ff00)  Line 967 + 0x18 bytes        C
> > >         python27.dll!PyRun_AnyFileExFlags(_iobuf * fp=0x73247408,
> > > const char
> > > * filename=0x00354be4, int closeit=1, PyCompilerFlags *
> > > flags=0x0027ff00)  Line 770 + 0x11 bytes        C
> > >         python27.dll!Py_Main(int argc=2, char * * argv=0x00354ba0)
> > > Line
> > > 643
> > > + 0x27 bytes    C
> > >         python.exe!__tmainCRTStartup()  Line 582 + 0x17 bytes   C
> > >
> > > From the output window...
> > >
> > > HEAP[python.exe]: HEAP: Free Heap block 3812418 modified at 3812438
> > > after it was freed Windows has triggered a breakpoint in python.exe.
> > >
> > > This may be due to a corruption of the heap, which indicates a bug
> > > in python.exe or any of the DLLs it has loaded.
> > >
> > > This may also be due to the user pressing F12 while python.exe has focus.
> > >
> > > The output window may have more diagnostic information.
> > >
> > > Cheers,
> > > Richard.
> > >
> > > On 2/6/14, Kristján Valur Jónsson <kristjan at ccpgames.com> wrote:
> > >> Ok, I have a proposed patch in a branch on
> > >> https://bitbucket.org/krisvale/stackless-scratch
> > >>
> > >> The idea is to move the flags from the PyTypeObject into the
> > >> PyMappingMethods structure, which is unlikely to be extended by
> > >> third party apps.
> > >> This runs the stackless testsuite.
> > >>
> > >> K
> > >>
> > >> From: stackless-bounces at stackless.com
> > >> [mailto:stackless-bounces at stackless.com] On Behalf Of lars van
> > >> Gemerden
> > >> Sent: 5. febrúar 2014 12:17
> > >> To: The Stackless Python Mailing List
> > >> Subject: Re: [Stackless] Windows 2.7.6 stability
> > >>
> > >> I would like to see better PySide compatibility, since my code
> > >> relies on stackless and PySide.
> > >>
> > >> The crashes have become less though since maybe a year ago; maybe
> > >> some improvements on the PySide?
> > >>
> > >> I am using stackless python 2.7.5.
> > >>
> > >> CL
> > >>
> > >> On Wed, Feb 5, 2014 at 9:55 AM, Kristján Valur Jónsson
> > >> <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:
> > >> At one point I had a patch going in stackless, which changed the
> > >> way we extended PyHeapType, IIRC.
> > >> I think this is the way to go, make sure we stay compatible.
> > >>
> > >>> -----Original Message-----
> > >>> From:
> > >>> stackless-bounces at stackless.com<mailto:stackless-
> > bounces at stackless.c
> > >>> om> [mailto:stackless-<mailto:stackless->
> > >>> bounces at stackless.com<mailto:bounces at stackless.com>] On Behalf
> Of
> > >>> Richard Tew
> > >>> Sent: 4. febrúar 2014 20:11
> > >>> To: The Stackless Python Mailing List
> > >>> Subject: Re: [Stackless] Windows 2.7.6 stability
> > >>>
> > >>> Well, I think Christian was deep into it, and trying to get the
> > >>> PySide people to do a compatibility patch or something.  Last I
> > >>> recall, from several months ago.
> > >>>
> > >>> It is indeed a pity we can't say we're compatible with all Python
> > >>> extensions, and it would be good to get it fixed.
> > >>>
> > >>> Cheers,
> > >>> Richard.
> > >>>
> > >>> On 2/4/14, Kristján Valur Jónsson
> > >>> <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:
> > >>> > This is so annoying.  Time for another stab at this problem, perhaps?
> > >>> >
> > >>> >> -----Original Message-----
> > >>> >> From:
> > >>> >> stackless-bounces at stackless.com<mailto:stackless-
> > bounces at stackles
> > >>> >> s.com> [mailto:stackless-<mailto:stackless->
> > >>> >> bounces at stackless.com<mailto:bounces at stackless.com>] On
> Behalf
> > Of
> > >>> >> Richard Tew
> > >>> >> Sent: 3. febrúar 2014 18:53
> > >>> >> To: The Stackless Python Mailing List
> > >>> >> Subject: Re: [Stackless] Windows 2.7.6 stability
> > >>> >>
> > >>> >> No, thanks for the suggestion.  But this is definitely PySide
> > >>> >> and it's modification of the base object type that stackless
> > >>> >> also modifies.
> > >>> >>
> > >>> >> Cheers,
> > >>> >> Richard.
> > >>> >>
> > >>> >> On 2/4/14, Anselm Kruis
> > >>> >> <a.kruis at science-computing.de<mailto:a.kruis at science-
> > computing.de
> > >>> >> >>
> > >>> >> wrote:
> > >>> >> > Could it be related to the PGO optimised build?
> > >>> >> > As far as I know, the mainline python installer is build
> > >>> >> > without PGO.
> > >>> >> > Our installer is build with PGO optimisation.
> > >>> >> >
> > >>> >> > Another question: can you try our 2.7.5 build? It is PGO
> > >>> >> > optimized too.
> > >>> >> >
> > >>> >> > Cheers
> > >>> >> >    Anselm
> > >>> >> >
> > >>> >> >
> > >>> >> > Am 03.02.2014 12<tel:03.02.2014%2012>:13, schrieb Kristján
> > >>> >> > Valur
> > >>> >> > Jónsson:
> > >>> >> >> Hi.
> > >>> >> >> Do you get this only with the installed version?
> > >>> >> >> Could you try replacing it with your own build?  If so,
> > >>> >> >> could you go into the source code and disable stack spilling ?
> > >>> >> >> You have to nerf the macro CSTACK_SAVE_NOW.
> > >>> >> >>
> > >>> >> >> I saw some mysterious crashes recently in a live build in
> > >>> >> >> Shanghai that went away when I disabled this.
> > >>> >> >>
> > >>> >> >> K
> > >>> >> >>
> > >>> >> >>> -----Original Message-----
> > >>> >> >>> From:
> > >>> >> >>> stackless-bounces at stackless.com<mailto:stackless-
> > bounces at stac
> > >>> >> >>> kless.com> [mailto:stackless-<mailto:stackless->
> > >>> >> >>> bounces at stackless.com<mailto:bounces at stackless.com>] On
> > >>> >> >>> Behalf Of Richard Tew
> > >>> >> >>> Sent: 3. febrúar 2014 03:04
> > >>> >> >>> To: stackless at stackless.com<mailto:stackless at stackless.com>
> > >>> >> >>> Subject: [Stackless] Windows 2.7.6 stability
> > >>> >> >>>
> > >>> >> >>> Hi,
> > >>> >> >>>
> > >>> >> >>> I've got the installer we provide for 2.7.6, on Windows.
> > >>> >> >>> And I've been getting lots of non-crashing premature exits:
> > >>> >> >>>
> > >>> >> >>> SystemError: unknown opcode XXX lineno: 314, opcode: 0
> > >>> >> >>>
> > >>> >> >>> It's not consistently reproducible using running the same
> > >>> >> >>> code, but can be sometimes, and I'm not using any Stackless
> > features.
> > >>> >> >>>
> > >>> >> >>> If I take the mainline python repo and sync to v2.7.6 and
> > >>> >> >>> generate a dll, and put it in c:\python27, all the problems
> > >>> >> >>> go away.
> > >>> >> >>>
> > >>> >> >>> I thought it might be pyside 1.1.2 which I was using, but
> > >>> >> >>> upgraded that and the problem remained with pyside 1.2.1.
> > >>> >> >>> That's the only external dependency my code uses.
> > >>> >> >>>
> > >>> >> >>> Anyone else using this installer?
> > >>> >> >>>
> > >>> >> >>> Cheers,
> > >>> >> >>> Richard.
> > >>> >> >>>
> > >>> >> >>> _______________________________________________
> > >>> >> >>> Stackless mailing list
> > >>> >> >>> Stackless at stackless.com<mailto:Stackless at stackless.com>
> > >>> >> >>> http://www.stackless.com/mailman/listinfo/stackless
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> _______________________________________________
> > >>> >> >> Stackless mailing list
> > >>> >> >> Stackless at stackless.com<mailto:Stackless at stackless.com>
> > >>> >> >> http://www.stackless.com/mailman/listinfo/stackless
> > >>> >> >>
> > >>> >> >
> > >>> >> > --
> > >>> >> >   Dipl. Phys. Anselm Kruis                       science +
> > >>> >> > computing
> > >>> >> > ag
> > >>> >> >   Senior Solution Architect                      Ingolstädter Str.
> > >>> >> > 22
> > >>> >> >   email
> > >>> >> > A.Kruis at science-computing.de<mailto:A.Kruis at science-
> > computing.de>
> > >>> >> >         80807 München, Germany
> > >>> >> >   phone +49 89 356386 874<tel:%2B49%2089%20356386%20874>
> fax
> > >>> >> > 737 www.science-computing.de<http://www.science-
> > computing.de>
> > >>> >> > --
> > >>> >> > Vorstandsvorsitzender/Chairman of the board of management:
> > >>> >> > Gerd-Lothar Leonhart
> > >>> >> > Vorstand/Board of Management:
> > >>> >> > Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Arno Steitz, Dr.
> > >>> >> > Ingrid Zech Vorsitzender des Aufsichtsrats/ Chairman of the
> > >>> >> > Supervisory
> > >>> >> > Board:
> > >>> >> > Philippe Miltin
> > >>> >> > Sitz/Registered Office: Tuebingen
> > >>> >> > Registergericht/Registration
> > >>> >> > Court: Stuttgart Registernummer/Commercial Register No.: HRB
> > >>> >> > 382196
> > >>> >> >
> > >>> >> >
> > >>> >> > _______________________________________________
> > >>> >> > Stackless mailing list
> > >>> >> > Stackless at stackless.com<mailto:Stackless at stackless.com>
> > >>> >> > http://www.stackless.com/mailman/listinfo/stackless
> > >>> >> >
> > >>> >>
> > >>> >> _______________________________________________
> > >>> >> Stackless mailing list
> > >>> >> Stackless at stackless.com<mailto:Stackless at stackless.com>
> > >>> >> http://www.stackless.com/mailman/listinfo/stackless
> > >>> >
> > >>> >
> > >>> >
> > >>> > _______________________________________________
> > >>> > Stackless mailing list
> > >>> > Stackless at stackless.com<mailto:Stackless at stackless.com>
> > >>> > http://www.stackless.com/mailman/listinfo/stackless
> > >>> >
> > >>>
> > >>> _______________________________________________
> > >>> Stackless mailing list
> > >>> Stackless at stackless.com<mailto:Stackless at stackless.com>
> > >>> http://www.stackless.com/mailman/listinfo/stackless
> > >>
> > >>
> > >>
> > >> _______________________________________________
> > >> Stackless mailing list
> > >> Stackless at stackless.com<mailto:Stackless at stackless.com>
> > >> http://www.stackless.com/mailman/listinfo/stackless
> > >>
> > >>
> > >>
> > >> --
> > >> ====================================
> > >> Lars van Gemerden
> > >> lars at rational-it.com<mailto:lars at rational-it.com>
> > >> +31 6 26 88 55 39
> > >> ====================================
> > >>
> > >
> > > _______________________________________________
> > > Stackless mailing list
> > > Stackless at stackless.com
> > > http://www.stackless.com/mailman/listinfo/stackless
> > >
> > > _______________________________________________
> > > Stackless mailing list
> > > Stackless at stackless.com
> > > http://www.stackless.com/mailman/listinfo/stackless
> > >
> >
> > _______________________________________________
> > Stackless mailing list
> > Stackless at stackless.com
> > http://www.stackless.com/mailman/listinfo/stackless
> 
> 
> 
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless





More information about the Stackless mailing list