[Stackless] stackless and threads
Christian Tismer
tismer at tismer.com
Mon Jul 14 01:58:27 CEST 2003
Hi Richard,
> On Sun, 13 Jul 2003, Vladimir Vukicevic wrote:
>
>
>>On Sat, 2003-07-12 at 14:44, Christian Tismer wrote:
>>
>>
>>>At the moment, under circumstances, it will crash.
>>>I had that complaint before. What I have to do at least
>>>is to avoid switches between tasklets from different threads.
>>>I don't know whether it makes sense to build real thread
>>>switching into tasklets, or just let them raise an exception.
>>
>>Mm. I was hoping to have the ability to migrate tasklets between
>>threads.. i.e. create a tasklet in a thread that does, say, network
>>processing, and then add it to the tasklet runnable queue of another OS
>>thread. Thus the network thread could do natural poll() or whatnot
>>based waiting for input, and the tasklets could be cooperatively
>>scheduled on another thread.
>
>
> In a similar vien, I was wanting to migrate tasklets between different
> interpreters running in same process space. I would create the threads in
> C using os primitives and create an independent python interpreter in each
> one. Attempting to solve via running in completely seperate process
> spaces, you just move the problem else where having to poll some io for
> the communicating processes.
Every tasklet has a chain of frames. These frames have their thread
state. There is a different thread state for every thread, in every
interpreter.
At the moment, there is one way:
- if a tasklet does not involve any C stack
- then a tasklet can be pickled and re-created elsewhere.
> However, I am making the gross assumption that (a) the python interpreters
> running in seperate threads are thread safe and (b) don't suffer from GIL.
> Christian : can we get away with tasklet switches in seperate interpreters
> or do we have the same problem?
We have a problem with tasklets which have C state. These cannot
be transferred. We can transfer tasklets with no C state.
I also can make this easier and more explicit:
It might be thinkable to make some migrate function that
grabs a tasklet and puts it into a different thread,
which also might belong to a different interpreter.
But this is exactly then possible when the pickling trick works.
> An alternative solution would be to use nonblocking IO and create our own
> poll like functions via a tasklet and a scheduler object. It wouldn't be
> as efficient - but it would mean we could run the likes of networking code
> in one thread. A high level library such as Pyro (stackless-ified ;-))
> could be used to farm out work to other processes if the box has more than
> processor and migrate pickled tasklets to other processes.
Why would you want to use poll at all?
IronPort has made a very efficient solution
using BSD kernel queues, so-called kqueues.
And I know there are (while less efficient) ways
to simulate this.
Basically, the idea is to wait for an action on a large
number of connections, and then to feed the result/request
to as many tasklets as necessary.
> The second solution also allows us have very large number of connections
> open without grinding the OS to halt with excessive amounts of threads &
> context switching.
I believe, this should be doable with at most one extra
real thread, unless, the OS is Windows. On the latter, I
think to remember there is an upper limit for the concurrent
arguments in one WaitForMultipleObjects call.
ciao - chris
--
Christian Tismer :^) <mailto:tismer at tismer.com>
Mission Impossible 5oftware : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/
_______________________________________________
Stackless mailing list
Stackless at www.tismer.com
http://www.tismer.com/mailman/listinfo/stackless
More information about the Stackless
mailing list