[Stackless] Stackless Embedding/Extending & Threading Questions
bjt at pmp.com.au
Tue Mar 27 01:37:10 CEST 2007
Thanks Richard. Without delving into "trade secret stuff" - could I get
your feedback on the following idea then?
The application server is written embedding Stackless Python. A single
thread runs a script starting up the Stackless Tasklets. Other threads
(for database access, network comms, etc) are run in native code (i.e.
developed in C++). Boost, Pyrex, <whatever> can be used ignoring Python
dependent threading issues as the GIL is only ever required for the one
thread. If this is similar to your setup/experience at CCP - I can
assume that any bugs I find are related to my code and not the
underlying Stackless / Application design.
The only thing to work out then would be a nice method of waiting on
events from the other (native) threads when requesting data from the
database in Python code; most likely by putting dependent tasklets into
a list that polls the "ready" state of the "native query" object before
waking said tasklet up with a channel send.
Does this sound offbase as a method of using the Stackless environment?
I'm willing to submit any "proof of concept" code back to the Wiki if
that would be of benefit. If it works, I can see it being a good
launching point for the use of Stackless in a few research projects (my
current project is an AI / Data-mining project that can have it's
current <intermediate> state saved to disk at arbitrary points in time).
Richard Tew wrote:
> On 3/26/07, Benjamin Tolputt <bjt at pmp.com.au> wrote:
>> The issue comes from the multi-threading issues above. Firstly, to my
>> understanding Boost Python integration is not by default multi-thread
>> safe (there is apparently a method to make it so, but it is not included
>> in the official distribution). As such, the easiest method I know to
>> extend/embed Python & C++ is out the window. What methods do users on
>> this list use to extend/embed their native code with/in Stackless?
> CCP currently uses a custom method but we are moving to Boost Python.
> We ignore the thread safe problems which exist with both solutions.
> But it isn't really a problem as we run all our Python logic in the
> main thread
> and Python is never used to deal with the other threads.
>> Finally, given the cooperative nature of Stackless, I am not sure if
>> there is a Stackless method of passing information between threads such
>> that each thread can continue running. I have looked at the examples on
>> the wiki, but the code samples indicate that either a) there is to be
> The code samples are not intended to be used as documentation :-)
> At least not the ones I wrote. In most cases they were written to
> illustrate that something was possible with the expectation the
> user would adapt it to suit their Stackless environment.
>> only one thread running to make Stackless safe or b) one has to make the
> Multiple Python threads running in Stackless should work fine.
>> tasklet atomic & use "standard" threading classes/code to send
>> information between Python threads. Is this correct or am I missing
>> something fundamental?
> Well, you want to pass information between threads without
> blocking. In this case I would use the Queue class and its nowait
> methods to do so. And making the tasklet atomic doesn't come
> into it in that case because you are not blocking the tasklet
> in the thread you are running the scheduler on anyway.
> Hope this helps,
Stackless mailing list
Stackless at stackless.com
More information about the Stackless