[Stackless] Real Threading for multiprocessors
Breton M. Saunders
breton.saunders at ntlworld.com
Wed Jun 11 00:04:11 CEST 2003
Thanks for the reply.
I should have been more clear with my mail in the first place. I
would prefer to have manual access controls on critical data for maximum
efficiency (like one would write using a C++/pthreads app). However,
there is a major tradeoff against interpreter safety. I suspect that
modifying a python list in two separate threads at the same time without
access restriction could crash the interpreter - hence the 30%
performance loss because locking is instated on each python object.
I like your idea of tasklets - but am wondering how they could be
implemented. One id4ea I'm tossing around is a TIL (replacing the
GIL). Each Thread Interpreter Lock would be acquired and held by an
executing thread. Objects created in threads would automatically be
assigned ownership to the specific creating thread requiring no TIL
locking to access. However, for other threads requiring access to
objects owned in other threads the TIL would need to be acquired.
Hence, multi-threading but automatic locking on object sharing.
However, I'm concerned about the access costs to check permission on
objects - it could degrade performance to the lock-per-object model.
I'm probably going to have to chase the CS literature for help on this
one. Anyone know any good papers on threading in interpreted languages?
Just a thought -
On Tue, 2003-06-10 at 18:28, Christian Tismer wrote:
> Breton M. Saunders wrote:
> > Guys,
> > I've been designing a large-scale server at work that could benefit
> > from running multiple threads in python simultaneously - aka real
> > threading, not the sequential-execution threading provided by the
> > standard python. Currently I implement threading at the C-code level
> > only.
> > I have two questions:
> > 1) Does stackless provide a real threading model providing N
> > simultaneously executing threads on N processors? (I suspect not AFAIK)
> Right, it doesn't, yet.
> > 2) How complex would it be to make the interpreter _really_ thread
> > safe?
> From former experiements of Greg Stein, I have the memory of about
> 30 percent extra overhead, for making every mutable thread-safe.
> > I've been looking at the stackless code; but have really only just
> > started.
> I have been told of applications which would benefit from free
> threading with general shared objects, regardless the rather high costs.
> My personal taste is a bit different: I think it is more efficient
> to have free-running, disjoint interpreters, and to make object
> access between threads explicit, involving locks. This would allow
> to let the available CPUs run at full speed as long as their
> computation is local, and only to involve some special communication
> wrappers run when information is interchanged.
> I was pointed to an implementation by Stephan Diehl, which does this,
> it is named "POSH", available at http://poshmodule.sourceforge.net
> In combination with Stackless, this might become an interesting
> combination. (Didn't investigate further, yet)
> My vision is about tasklets, which are run by a number of
> Python processes, one for each processor. These tasklets
> should be able to migrate, like "nomad tasks", to the
> processor with the smallest work-load.
> Something like this might become my next challenge after
> Stackless 3.0 ist presented on EuroPy.
> cheers - chris
Breton M. Saunders <breton.saunders at ntlworld.com>
Brett Saunders Inc
More information about the Stackless