[Stackless] Newbie questions
hinsen at cnrs-orleans.fr
Sun Feb 16 10:23:56 CET 2003
On Sunday 16 February 2003 01:39, Christian Tismer wrote:
> Fascinating to have *you* in this. I'm pretty sure
> you will force me to implement MPI and other stuff, soon.
No! I have switched from MPI to BSP (see www.bsp-worldwide.org, or a recent
copy of Scientific Python), and I intend to implement it in Stackless myself
> Yes. With 2.0, there is an issue, which didn't exist in 1.0,
> because it wasn't able to switch extensions at all.
I could even live with that, as long as I can use my extensions at all.
> problem in callbacks. This is also the reason why 2.0 refuses
> to use pre-emptive multitasking, since a switch could happen
> inside of an extension that cannot stand it.
So when do task switches occur? Only when channel operations are called?
> In other words: If your extension does not call back into
> Python, or if it does, but there are no switches intended,
> you have no problem.
Fine. So if I make sure that all callbacks set the "atomic" flag, then
everything should be fine.
> The sent objects of channels are identical to the
> sent ones. This might change, since I want to use
But the object ids are not the same as the ones of the sent objects - as I
just discovered experimentally.
> the channel protocol over different machines as well,
> but for now, this is implemented in the cheapest possible way.
Fine, then I make copies myself if needed.
> Very much interested!
OK, so here is a brief description. I work on high-level parallelization using
the BSP model. Python/BSP is explained in
for the moment is all the available documentation. BSP divides a computation
in consecutive "compute" and "communicate" steps. One of the advantages is
that communication is simple to implement, and easy to optimize.
My next project is automatic load balancing. The idea is to have many more
"virtual" processors than real ones. Each real CPU measures the time spent on
dealing with each "virtual" CPU, and once in a while the real CPUs exchange
virtual CPUs among each other to equalize CPU load.
The obvious solution for each "virtual" CPU would be a thread. But there are
two problems: Many low-level communications libraries don't coexist well with
threads, and there is a significant overhead, which is completely unnecessary
because pseudo-simultaneous execution is not even required, only the
possiblity of task switches. Which is where Stackless comes in. I plan to
make each virtual CPU a tasklet, which runs atomically most of the time,
until it reaches the communication step and waits.
BTW, I saw hints about thread pickling, is that a reality already? It could be
useful in moving threads between CPUs.
Stackless mailing list
Stackless at www.tismer.com
More information about the Stackless