[Stackless] Stackless Embedding/Extending & Threading Questions

[Benjamin Tolputt]

Thanks Richard. Without delving into "trade secret stuff" - could I get 
your feedback on the following idea then?

The application server is written embedding Stackless Python. A single 
thread runs a script starting up the Stackless Tasklets. Other threads 
(for database access, network comms, etc) are run in native code (i.e. 
developed in C++). Boost, Pyrex, <whatever> can be used ignoring Python 
dependent threading issues as the GIL is only ever required for the one 
thread. If this is similar to your setup/experience at CCP - I can 
assume that any bugs I find are related to my code and not the 
underlying Stackless / Application design.

The only thing to work out then would be a nice method of waiting on 
events from the other (native) threads when requesting data from the 
database in Python code; most likely by putting dependent tasklets into 
a list that polls the "ready" state of the "native query" object before 
waking said tasklet up with a channel send.

Does this sound offbase as a method of using the Stackless environment? 
I'm willing to submit any "proof of concept" code back to the Wiki if 
that would be of benefit. If it works, I can see it being a good 
launching point for the use of Stackless in a few research projects (my 
current project is an AI / Data-mining project that can have it's 
current <intermediate> state saved to disk at arbitrary points in time).

Regards,
B.J.Tolputt


Richard Tew wrote:
> On 3/26/07, Benjamin Tolputt <bjt at pmp.com.au> wrote:
>> The issue comes from the multi-threading issues above. Firstly, to my
>> understanding Boost Python integration is not by default multi-thread
>> safe (there is apparently a method to make it so, but it is not included
>> in the official distribution). As such, the easiest method I know to
>> extend/embed Python & C++ is out the window. What methods do users on
>> this list use to extend/embed their native code with/in Stackless?
>
> CCP currently uses a custom method but we are moving to Boost Python.
> We ignore the thread safe problems which exist with both solutions.
> But it isn't really a problem as we run all our Python logic in the 
> main thread
> and Python is never used to deal with the other threads.
>
>> Finally, given the cooperative nature of Stackless, I am not sure if
>> there is a Stackless method of passing information between threads such
>> that each thread can continue running. I have looked at the examples on
>> the wiki, but the code samples indicate that either a) there is to be
>
> The code samples are not intended to be used as documentation :-)
> At least not the ones I wrote.  In most cases they were written to
> illustrate that something was possible with the expectation the
> user would adapt it to suit their Stackless environment.
>
>> only one thread running to make Stackless safe or b) one has to make the
>
> Multiple Python threads running in Stackless should work fine.
>
>> tasklet atomic & use "standard" threading classes/code to send
>> information between Python threads. Is this correct or am I missing
>> something fundamental?
>
> Well, you want to pass information between threads without
> blocking.  In this case I would use the Queue class and its nowait
> methods to do so.  And making the tasklet atomic doesn't come
> into it in that case because you are not blocking the tasklet
> in the thread you are running the scheduler on anyway.
>
> Hope this helps,
> Richard.
>
>



_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://www.stackless.com/mailman/listinfo/stackless