[Stackless] Using Stackless Python to overcome GIL issues interoperating with a C++ runtime

[Adam Preble]

What I've been striving towards is something like this, although I strayed
around quite a bit.  I have something like promises and futures and a
command stack for other threads already.  I've had to do stuff like that
for drawing systems before because a lot of them get real angry when I try
to tell them to draw something from a different thread--even if it's
synchronized.

The interfaces I can implement in Python use wrapper classes to interface
with them, so I already have this sprinkled around:

PyGILState_STATE gstate = PyGILState_Ensure();
this->get_override("TheActualMethod")(all, the, arguments);
PyGILState_Release(gstate);

I was thinking instead to do:

        scripting->addToCommandStack(function(&TheWrapper::TheMethod, this,
all, the, arguments));

Since I posted the original post, I discovered the situation isn't as dire
as I thought.  I had thought if one thread called PyGILState_Ensure() and
then called it again, it would deadlock.  So I was coming up with all this
stuff with saving thread states and moving on to other commands when one
command across the boundaries lead to another.  I wrote up a little test
program to demonstrate the issue and found it ran perfectly fine, rather
than deadlock.

An issue of mine is I call Py_Main() in the current app and doodle in the
console to check on stuff.  I have to basically replace the REPL with my
own version that comes up for air every once in awhile so I can process the
other commands.

I also have to say with a name like Gil, a little devil must appear on your
shoulder and whisper to you to troll these posts about avoiding the GIL and
all that stuff. :)


On Sun, May 6, 2012 at 7:20 PM, Gil Colgate <gcolgate at gmail.com> wrote:

> The GIL is a hard  to work around. I am wondering if you could instead of
> processing the other thread's python commands immediately, you could put
> the items into a queue for processing by a python callback on the main
> thread later, and make sure all python is called from this single thread
> (an assert will help). I know this is not super efficient for a multi-core
> machine, but neither is using python.
>
> Someone made a variation of python that supports multiple threads, it
> turns out it was very inefficient.  I imagine he put mutexes on every
> python object to prevent them from being accessed from two threads at the
> the same time.... which is the fundamental problem the GIL is trying to
> solve.
>
> On Sun, May 6, 2012 at 3:25 AM, Adam Preble <adam.preble at gmail.com> wrote:
>
>> I wondered if anybody here has used Stackless Python in some clever ways
>> to work around the global interpreter lock.  I have some C++ objects
>> wrapped with Boost.Python, and I run into some interesting stuff when I
>> start implementing interfaces defined in these wrappers in Python.  At that
>> point I have to do some work that involves acquiring the GIL to run the
>> Python version of the code whenever I'm running into it from the C++ side
>> in other threads.  This normally works okay until I get into situations
>> like this:
>>
>> 1. Python interpreter calls something with a wrapper into C++.
>> 2. C++ code calls back into a Python implementation of a C++ interface
>> (GIL involved here).
>> 3. Python implementation calls some other thing wrapped from C++ . . .
>> note that step #1 hasn't actually completed yet.
>> 4. C++ code calls back into Python.  Here we deadlock trying to get the
>> GIL.
>>
>> I'm looking at a situation here where I need to place all these pending
>> GIL-related operations on some common stack, and have something comb over
>> them.  When that comber realizes it already took the GIL, and it's being
>> told to do a new operation, it can at least do some negotiation with
>> itself.  I don't know how the GIL works entirely here since I haven't
>> implemented this logic yet, but it's clear I have to do something.  At the
>> worst I assume I will have to save the interpreter stack from the prior
>> call before moving on to the next call.
>>
>> It occurred to me there could be some nice ways to do this with
>> coroutines.  I'm already pondering some stuff Boost has for coroutines that
>> might make this a lot less ugly, but I wondered if Stackless had something
>> that could come to the rescue.  I don't know what it could possibly be, but
>> I thought I'd ask the list for advice.
>>
>>
>> _______________________________________________
>> Stackless mailing list
>> Stackless at stackless.com
>> http://www.stackless.com/mailman/listinfo/stackless
>>
>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20120507/a833a404/attachment.html>