[Stackless] Multi-CPU Actor Based Python

Michael ms at cerenity.org
Wed Nov 19 12:10:22 CET 2008


On Tuesday 18 November 2008 21:41:36 Timothy Baldridge wrote:
> The issue with Stackless is that you are still limited by the GIL.
> Therefore you can never use more than 100% of one CPU. 

1 hardware thread can only give you 100% of itself. (I'm using that rather 
than saying core or CPU since it's less ambiguous)

I presume this means you are stating that you are limited to the capacity of a 
single CPU even on a multicore machine. This is not generally true. If all 
your processing is CPU bound inside python itself then this is true, however 
if you use decent C libraries to offload work into that release the GIL, this 
rapidly becomes false.

If you are desperately worried about this, use IronPython of Jython. Neither 
of which have a GIL.

> You can pickle/unpickle messages and pass them via pipes or shared memory,
> but that takes time. 

Yes, we do do that. It works. You just have to be intelligent about how you 
partition your application. But then I'm much more likely to know what's 
going to be expensive to pass between processes than python is anyway.

Lots of people like the idea of doing all this automatically, but forget that
actually conciously partitioning across CPUs is a very good idea, simply
because of things like cache invalidation, etc. When we start have thousands
of cores on a CPU, then the rules change again.

> Simply copying a pointer would be much faster. 

Indeed it would. If you're serious about improving CPython in this regard, 
making posh reliable, and _maintained_ , would be really useful.
    * http://poshmodule.sourceforge.net/

That is however an optimisation. Building something that works first and then
optimising will let you know where to optimise. Mind you, if you want to do 
that optimisation, the poshmodule would be the place to start, since that 
would help the most projects IMO.


Michael.
-- 
http://yeoldeclue.com/blog
http://www.kamaelia.org/Home




More information about the Stackless mailing list