[Stackless] stackless python in a multicore environment

[Chris Lee]

Hi Everyone,

I suspect that this question has come up before, but a search of the 
archive revealed nothing so please forgive me if you are tired of answering.
Basically I need some hints on using python (or stackless python) on a 
multi core CPU.
Let me give some details of the project:
I am running simulations on light traveling through a very disordered 
medium. I do this using a ballistic approximation, which essentially 
means that I assume the light consists of particles and then generate a 
bunch of random numbers for each particle. I use the random numbers to 
determine the path taken by the particle.

In practice, I simulate between 1e4 and 1e6 particles at a time taking 
advantage of numpy and python to keep the code clean while still getting 
good speed from a single CPU core. However, the simulations are becoming 
more sophisticated and I would like to be able to take advantage of 
multiple core CPUs and maybe even multiple computers.

My first attempt was a disaster. I used the threading module and simple 
split the task into two threads. The interpreter put each thread on a 
separate CPU and bus deadlock ensued (each CPU was trying to access 
136*5e5 bytes of memory simultaneously).

I realized that I would need finer grained control over how the data was 
apportioned between threads, but doing this using the python queue and 
events starts to look a bit messy again. That was when I happened about 
stackless python and tasklets. With tasklets I get the finegrained 
control over data access that allows to me to ease the bus contention 
... but all the tasklets run on a single core. Even if I take the 
trouble to spawn python threads and the threads run method invokes a 
tasklet, they all run on the same core.

Can someone give me some advice towards making use of multicores, 
preferable with stackless--the code is sooo much nicer--but I'll take 
any python based solution at this point.

Cheers
Chris

-- 
**********************************************
*  Chris Lee                                 *
*  Laser physics and nonlinear optics group  *
*  MESA+ Institute                           *
*  University of Twente                      *
*  Phone: ++31 (0)53 489 3968                *
*  fax: ++31 (0) 53 489 1102                 *
**********************************************


_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://stackless.com/cgi-bin/mailman/listinfo/stackless