[Stackless] Newbie questions

Konrad Hinsen hinsen at cnrs-orleans.fr
Sun Feb 16 10:23:56 CET 2003

On Sunday 16 February 2003 01:39, Christian Tismer wrote:

> Fascinating to have *you* in this. I'm pretty sure
> you will force me to implement MPI and other stuff, soon.

No! I have switched from MPI to BSP (see www.bsp-worldwide.org, or a recent 
copy of Scientific Python), and I intend to implement it in Stackless myself 

> Yes. With 2.0, there is an issue, which didn't exist in 1.0,
> because it wasn't able to switch extensions at all.

I could even live with that, as long as I can use my extensions at all.

> problem in callbacks. This is also the reason why 2.0 refuses
> to use pre-emptive multitasking, since a switch could happen
> inside of an extension that cannot stand it.

So when do task switches occur? Only when channel operations are called?

> In other words: If your extension does not call back into
> Python, or if it does, but there are no switches intended,
> you have no problem.

Fine. So if I make sure that all callbacks set the "atomic" flag, then 
everything should be fine.

> The sent objects of channels are identical to the
> sent ones. This might change, since I want to use

But the object ids are not the same as the ones of the sent objects - as I 
just discovered experimentally.

> the channel protocol over different machines as well,
> but for now, this is implemented in the cheapest possible way.

Fine, then I make copies myself if needed.

> Very much interested!

OK, so here is a brief description. I work on high-level parallelization using 
the BSP model. Python/BSP is explained in 
http://starship.python.net/~hinsen/ScientificPython/BSP_Tutorial.pdf, which 
for the moment is all the available documentation.  BSP divides a computation 
in consecutive "compute" and "communicate" steps. One of the advantages is 
that communication is simple to implement, and easy to optimize.

My next project is automatic load balancing. The idea is to have many more 
"virtual" processors than real ones. Each real CPU measures the time spent on 
dealing with each "virtual" CPU, and once in a while the real CPUs exchange 
virtual CPUs among each other to equalize CPU load.

The obvious solution for each "virtual" CPU would be a thread. But there are 
two problems: Many low-level communications libraries don't coexist well with 
threads, and there is a significant overhead, which is completely unnecessary 
because pseudo-simultaneous execution is not even required, only the 
possiblity of task switches. Which is where Stackless comes in. I plan to 
make each virtual CPU a tasklet, which runs atomically most of the time, 
until it reaches the communication step and waits.

BTW, I saw hints about thread pickling, is that a reality already? It could be 
useful in moving threads between CPUs.


Stackless mailing list
Stackless at www.tismer.com

More information about the Stackless mailing list