[Stackless] Windows 2.7.6 stability

Kristján Valur Jónsson kristjan at ccpgames.com
Thu Feb 6 10:27:32 CET 2014


I'm trying to familiarize myself with the code too :)

There are two inheritance scenarios:
1) when a non-heap class inherits from a base class.  It then gets copies of the slots from the base.
2) When a heap class inherits from a base class.  It then gets copies of the slots from the base, but after any local slots have been filled with custom __next__ etc.

Having looked at the code some more and tried to understand all this "stackless" flag passing, I've come to the conclusion that
1) It is okay if we miss the 'can do stackless' flag on a slot.  if a stackles-aware function is called with 'stackless==0' it just behaves as normal.  So, passing the correct stackless flags for the slots is good, since it enables the stackless performance enhancement, but it is not strictly necessary.  In other words, we would get away with never copying this property.  On the other hand we can also be more clever in copying the flags, particulary for external types that inherit from one of the builtins....
3) there is only one flag per method so we might use a bitfield.
4) Only a handful of types actually have the stackless property, that is, have slots that can do stackless.  It is possible to optimize stuff greatly by adding the stackless flag only to those particular ones during initialization.

5) This change causes a binary incompatibility with external modules that were compiled using the stackless headers previously, since they would use an extended type object.

I think 5 is the only possible cause for concern here.  For that reason, we _may_ want to employ a new flag, one that means 'extensions are in the Mapping member', and ignore the old one, but only if you think this is a concern.

I'M going to make a new version of this now, one that isn't as intrusive by selectively patching only those types that we deem matter....  Using that paradigm, we may, for example, disable the stackless flag feature altogether at compile time, and then we can see if it has anything to do at all with the PySide crashes.

K



> -----Original Message-----
> From: stackless-bounces at stackless.com [mailto:stackless-
> bounces at stackless.com] On Behalf Of Richard Tew
> Sent: 6. febrúar 2014 01:46
> To: The Stackless Python Mailing List
> Subject: Re: [Stackless] Windows 2.7.6 stability
> 
> Where do types inherit a bunch of slots from the builtin type?  It's been a
> while since I had to be familiar with that part of the code.
> 
> Should your changes have resulted in something that crashes harder and
> faster (and more consistently) than the old heap type version?
> 
> Cheers,
> Richard.
> 
> On 2/6/14, Kristján Valur Jónsson <kristjan at ccpgames.com> wrote:
> > I think that the problem may be a bit more insidious than just the
> > size of the PyTypeObject.
> > Imagine this scenario:
> > An extension creates a C extension of a builtin type.  This extension
> > is compiled against vanilla Python.  So, its own default flags don't
> > have the STACKLESS flag.  But it does inherit a bunch of slots from
> > the builtin stackless object, and those are defined to be STACKLESS
> > aware.
> >
> > so, stackless will call those inherited slots thinking they don't
> > adhere to the STACKLESS protocol.
> >
> > Is it safe to call "stackless" type calls using the old method?  Or is
> > this purely an "added extra", so that the new, stackless aware, slot
> > methods _can_ work with the old calling mechanism?  I don't know.  Hm,
> > perhaps they do....  In which case, the stackless flag is an opt-in
> > thing.  If it is missing, even for a stackless method, then it would be ok.....
> >
> >
> > I think that with the method here, of having the STACKLESS flag
> > indicate that there is extra data in the PyMappingMethods structure,
> > we can in fact apply the STACKLESS flag to any inherited type, which
> > calls PyType_Ready().  If the parent type has the STACKLESS flag, we
> > can modify its tp_as_mapping member to point to a dynamically
> > allocated, larger, structure, and then proceed to do inheritance correctly.
> >
> > So, we can inherit the stackless flag for subtypes that were compiled
> > unaware of stackless.  If PySide were for example to create a subtype
> > of PyFunctionType, then we could decorate it with the STACKLESS flag
> > and have it inherit the stackless slot flags, by giving the subtype a
> > new tp_as_mapping member.
> >
> > K
> >
> >
> > ________________________________________
> > From: stackless-bounces at stackless.com
> > [stackless-bounces at stackless.com] on behalf of Richard Tew
> > [richard.m.tew at gmail.com]
> > Sent: Wednesday, February 05, 2014 19:58
> > To: The Stackless Python Mailing List
> > Subject: Re: [Stackless] Windows 2.7.6 stability
> >
> > On one hand I wonder why we just didn't do this long ago.
> >
> > On the other, it crashes :-)  I'll look into it later as I am
> > celebrating Waitangi Day in the traditional way of sitting around
> > doing nothing.
> >
> > When this is working, I'll rerelease 2.7.6 and try and get people to
> > rebuild all the binaries.
> >
> >>       python27.dll!getset_get(PyGetSetDescrObject * descr=0x0056a1e8,
> >> _object * obj=0x03848b98, _object * type=0x00000000)  Line 147 + 0xa
> bytes
> >>      C
> >         python27.dll!PyObject_Call(_object * func=0x0056a1e8, _object *
> > arg=0x03848b98, _object * kw=0x00000000)  Line 2539 + 0x1c bytes        C
> >         python27.dll!PyObject_CallFunctionObjArgs(_object *
> > callable=0x0056a1e8, ...)  Line 2786 + 0x13 bytes       C
> >         python27.dll!build_class(_object * methods=0x0384c390, _object *
> > bases=0x1e0efddf, _object * name=0x037ba270)  Line 5117 + 0xf bytes     C
> >         python27.dll!PyEval_EvalFrame_value(_frame * f=0x00502428, int
> > throwflag=58434160, _object * retval=0x0384c390)  Line 3394     C
> >         python27.dll!PyEval_EvalFrameEx_slp(_frame * f=0x00502428, int
> > throwflag=0, _object * retval=0x1e234ff4)  Line 964 + 0x17 bytes        C
> >         python27.dll!slp_eval_frame_newstack(_frame * f=0x00502428, int
> > exc=0, _object * retval=0x1e234ff4)  Line 521 + 0x1b bytes      C
> >         python27.dll!PyEval_EvalFrameEx_slp(_frame * f=0x00502428, int
> > throwflag=0, _object * retval=0x1e234ff4)  Line 964     C
> >         python27.dll!slp_frame_dispatch_top(_object * retval=0x1e234ff4)
> > Line 810 + 0x9 bytes    C
> >         python27.dll!slp_run_tasklet(_frame * f=0x00502428)  Line 1504  C
> >         python27.dll!slp_eval_frame(_frame * f=0x00502428)  Line 317 + 0xa
> > bytes        C
> >         python27.dll!climb_stack_and_eval_frame(_frame * f=0x00502428)
> > Line
> > 274 + 0x9 bytes C
> >         python27.dll!slp_eval_frame(_frame * f=0x00502428)  Line 303 + 0x6
> > bytes        C
> >         python27.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x022faa40,
> > _object
> > * globals=0x00502428, _object * locals=0x004bc930, _object * *
> > args=0x00000000, int argcount=0, _object * * kws=0x00000000, int
> > kwcount=0, _object * * defs=0x00000000, int defcount=0, _object *
> > closure=0x00000000)  Line 3697 + 0x6 bytes      C
> >         python27.dll!PyEval_EvalCode(PyCodeObject * co=0x022faa40,
> > _object
> > *
> > globals=0x004bc930, _object * locals=0x004bc930)  Line 674 + 0x22
> > bytes   C
> >         python27.dll!run_mod(_mod * mod=0x023613e8, const char *
> > filename=0x00000000, _object * globals=0x004bc930, _object *
> > locals=0x004bc930, PyCompilerFlags * flags=0x00000000, _arena *
> > arena=0x00000000)  Line 1408 + 0x16 bytes       C
> >         python27.dll!PyRun_FileExFlags(_iobuf * fp=0x73247408, const
> > char * filename=0x00354be4, int start=257, _object *
> > globals=0x004bc930, _object * locals=0x004bc930, int closeit=1,
> PyCompilerFlags *
> > flags=0x0027ff00)  Line 1393    C
> >         python27.dll!PyRun_SimpleFileExFlags(_iobuf * fp=0x73247408,
> > const char * filename=0x00354be4, int closeit=1, PyCompilerFlags *
> > flags=0x0027ff00)  Line 967 + 0x18 bytes        C
> >         python27.dll!PyRun_AnyFileExFlags(_iobuf * fp=0x73247408,
> > const char
> > * filename=0x00354be4, int closeit=1, PyCompilerFlags *
> > flags=0x0027ff00)  Line 770 + 0x11 bytes        C
> >         python27.dll!Py_Main(int argc=2, char * * argv=0x00354ba0)
> > Line
> > 643
> > + 0x27 bytes    C
> >         python.exe!__tmainCRTStartup()  Line 582 + 0x17 bytes   C
> >
> > From the output window...
> >
> > HEAP[python.exe]: HEAP: Free Heap block 3812418 modified at 3812438
> > after it was freed Windows has triggered a breakpoint in python.exe.
> >
> > This may be due to a corruption of the heap, which indicates a bug in
> > python.exe or any of the DLLs it has loaded.
> >
> > This may also be due to the user pressing F12 while python.exe has focus.
> >
> > The output window may have more diagnostic information.
> >
> > Cheers,
> > Richard.
> >
> > On 2/6/14, Kristján Valur Jónsson <kristjan at ccpgames.com> wrote:
> >> Ok, I have a proposed patch in a branch on
> >> https://bitbucket.org/krisvale/stackless-scratch
> >>
> >> The idea is to move the flags from the PyTypeObject into the
> >> PyMappingMethods structure, which is unlikely to be extended by third
> >> party apps.
> >> This runs the stackless testsuite.
> >>
> >> K
> >>
> >> From: stackless-bounces at stackless.com
> >> [mailto:stackless-bounces at stackless.com] On Behalf Of lars van
> >> Gemerden
> >> Sent: 5. febrúar 2014 12:17
> >> To: The Stackless Python Mailing List
> >> Subject: Re: [Stackless] Windows 2.7.6 stability
> >>
> >> I would like to see better PySide compatibility, since my code relies
> >> on stackless and PySide.
> >>
> >> The crashes have become less though since maybe a year ago; maybe
> >> some improvements on the PySide?
> >>
> >> I am using stackless python 2.7.5.
> >>
> >> CL
> >>
> >> On Wed, Feb 5, 2014 at 9:55 AM, Kristján Valur Jónsson
> >> <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:
> >> At one point I had a patch going in stackless, which changed the way
> >> we extended PyHeapType, IIRC.
> >> I think this is the way to go, make sure we stay compatible.
> >>
> >>> -----Original Message-----
> >>> From:
> >>> stackless-bounces at stackless.com<mailto:stackless-
> bounces at stackless.c
> >>> om> [mailto:stackless-<mailto:stackless->
> >>> bounces at stackless.com<mailto:bounces at stackless.com>] On Behalf Of
> >>> Richard Tew
> >>> Sent: 4. febrúar 2014 20:11
> >>> To: The Stackless Python Mailing List
> >>> Subject: Re: [Stackless] Windows 2.7.6 stability
> >>>
> >>> Well, I think Christian was deep into it, and trying to get the
> >>> PySide people to do a compatibility patch or something.  Last I
> >>> recall, from several months ago.
> >>>
> >>> It is indeed a pity we can't say we're compatible with all Python
> >>> extensions, and it would be good to get it fixed.
> >>>
> >>> Cheers,
> >>> Richard.
> >>>
> >>> On 2/4/14, Kristján Valur Jónsson
> >>> <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:
> >>> > This is so annoying.  Time for another stab at this problem, perhaps?
> >>> >
> >>> >> -----Original Message-----
> >>> >> From:
> >>> >> stackless-bounces at stackless.com<mailto:stackless-
> bounces at stackles
> >>> >> s.com> [mailto:stackless-<mailto:stackless->
> >>> >> bounces at stackless.com<mailto:bounces at stackless.com>] On Behalf
> Of
> >>> >> Richard Tew
> >>> >> Sent: 3. febrúar 2014 18:53
> >>> >> To: The Stackless Python Mailing List
> >>> >> Subject: Re: [Stackless] Windows 2.7.6 stability
> >>> >>
> >>> >> No, thanks for the suggestion.  But this is definitely PySide and
> >>> >> it's modification of the base object type that stackless also
> >>> >> modifies.
> >>> >>
> >>> >> Cheers,
> >>> >> Richard.
> >>> >>
> >>> >> On 2/4/14, Anselm Kruis
> >>> >> <a.kruis at science-computing.de<mailto:a.kruis at science-
> computing.de
> >>> >> >>
> >>> >> wrote:
> >>> >> > Could it be related to the PGO optimised build?
> >>> >> > As far as I know, the mainline python installer is build
> >>> >> > without PGO.
> >>> >> > Our installer is build with PGO optimisation.
> >>> >> >
> >>> >> > Another question: can you try our 2.7.5 build? It is PGO
> >>> >> > optimized too.
> >>> >> >
> >>> >> > Cheers
> >>> >> >    Anselm
> >>> >> >
> >>> >> >
> >>> >> > Am 03.02.2014 12<tel:03.02.2014%2012>:13, schrieb Kristján
> >>> >> > Valur
> >>> >> > Jónsson:
> >>> >> >> Hi.
> >>> >> >> Do you get this only with the installed version?
> >>> >> >> Could you try replacing it with your own build?  If so, could
> >>> >> >> you go into the source code and disable stack spilling ?
> >>> >> >> You have to nerf the macro CSTACK_SAVE_NOW.
> >>> >> >>
> >>> >> >> I saw some mysterious crashes recently in a live build in
> >>> >> >> Shanghai that went away when I disabled this.
> >>> >> >>
> >>> >> >> K
> >>> >> >>
> >>> >> >>> -----Original Message-----
> >>> >> >>> From:
> >>> >> >>> stackless-bounces at stackless.com<mailto:stackless-
> bounces at stac
> >>> >> >>> kless.com> [mailto:stackless-<mailto:stackless->
> >>> >> >>> bounces at stackless.com<mailto:bounces at stackless.com>] On
> >>> >> >>> Behalf Of Richard Tew
> >>> >> >>> Sent: 3. febrúar 2014 03:04
> >>> >> >>> To: stackless at stackless.com<mailto:stackless at stackless.com>
> >>> >> >>> Subject: [Stackless] Windows 2.7.6 stability
> >>> >> >>>
> >>> >> >>> Hi,
> >>> >> >>>
> >>> >> >>> I've got the installer we provide for 2.7.6, on Windows.  And
> >>> >> >>> I've been getting lots of non-crashing premature exits:
> >>> >> >>>
> >>> >> >>> SystemError: unknown opcode
> >>> >> >>> XXX lineno: 314, opcode: 0
> >>> >> >>>
> >>> >> >>> It's not consistently reproducible using running the same
> >>> >> >>> code, but can be sometimes, and I'm not using any Stackless
> features.
> >>> >> >>>
> >>> >> >>> If I take the mainline python repo and sync to v2.7.6 and
> >>> >> >>> generate a dll, and put it in c:\python27, all the problems
> >>> >> >>> go away.
> >>> >> >>>
> >>> >> >>> I thought it might be pyside 1.1.2 which I was using, but
> >>> >> >>> upgraded that and the problem remained with pyside 1.2.1.
> >>> >> >>> That's the only external dependency my code uses.
> >>> >> >>>
> >>> >> >>> Anyone else using this installer?
> >>> >> >>>
> >>> >> >>> Cheers,
> >>> >> >>> Richard.
> >>> >> >>>
> >>> >> >>> _______________________________________________
> >>> >> >>> Stackless mailing list
> >>> >> >>> Stackless at stackless.com<mailto:Stackless at stackless.com>
> >>> >> >>> http://www.stackless.com/mailman/listinfo/stackless
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> _______________________________________________
> >>> >> >> Stackless mailing list
> >>> >> >> Stackless at stackless.com<mailto:Stackless at stackless.com>
> >>> >> >> http://www.stackless.com/mailman/listinfo/stackless
> >>> >> >>
> >>> >> >
> >>> >> > --
> >>> >> >   Dipl. Phys. Anselm Kruis                       science +
> >>> >> > computing
> >>> >> > ag
> >>> >> >   Senior Solution Architect                      Ingolstädter Str.
> >>> >> > 22
> >>> >> >   email
> >>> >> > A.Kruis at science-computing.de<mailto:A.Kruis at science-
> computing.de>
> >>> >> >         80807 München, Germany
> >>> >> >   phone +49 89 356386 874<tel:%2B49%2089%20356386%20874>  fax
> >>> >> > 737 www.science-computing.de<http://www.science-
> computing.de>
> >>> >> > --
> >>> >> > Vorstandsvorsitzender/Chairman of the board of management:
> >>> >> > Gerd-Lothar Leonhart
> >>> >> > Vorstand/Board of Management:
> >>> >> > Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Arno Steitz, Dr.
> >>> >> > Ingrid Zech Vorsitzender des Aufsichtsrats/ Chairman of the
> >>> >> > Supervisory
> >>> >> > Board:
> >>> >> > Philippe Miltin
> >>> >> > Sitz/Registered Office: Tuebingen Registergericht/Registration
> >>> >> > Court: Stuttgart Registernummer/Commercial Register No.: HRB
> >>> >> > 382196
> >>> >> >
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > Stackless mailing list
> >>> >> > Stackless at stackless.com<mailto:Stackless at stackless.com>
> >>> >> > http://www.stackless.com/mailman/listinfo/stackless
> >>> >> >
> >>> >>
> >>> >> _______________________________________________
> >>> >> Stackless mailing list
> >>> >> Stackless at stackless.com<mailto:Stackless at stackless.com>
> >>> >> http://www.stackless.com/mailman/listinfo/stackless
> >>> >
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Stackless mailing list
> >>> > Stackless at stackless.com<mailto:Stackless at stackless.com>
> >>> > http://www.stackless.com/mailman/listinfo/stackless
> >>> >
> >>>
> >>> _______________________________________________
> >>> Stackless mailing list
> >>> Stackless at stackless.com<mailto:Stackless at stackless.com>
> >>> http://www.stackless.com/mailman/listinfo/stackless
> >>
> >>
> >>
> >> _______________________________________________
> >> Stackless mailing list
> >> Stackless at stackless.com<mailto:Stackless at stackless.com>
> >> http://www.stackless.com/mailman/listinfo/stackless
> >>
> >>
> >>
> >> --
> >> ====================================
> >> Lars van Gemerden
> >> lars at rational-it.com<mailto:lars at rational-it.com>
> >> +31 6 26 88 55 39
> >> ====================================
> >>
> >
> > _______________________________________________
> > Stackless mailing list
> > Stackless at stackless.com
> > http://www.stackless.com/mailman/listinfo/stackless
> >
> > _______________________________________________
> > Stackless mailing list
> > Stackless at stackless.com
> > http://www.stackless.com/mailman/listinfo/stackless
> >
> 
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless





More information about the Stackless mailing list