[Stackless] stackless python in a multicore environment

[Johan Carlsson]

(I'm sending this back to the list if that's ok, I have trouble
remembering to reply-all)

> Johan Carlsson wrote:
> > What did you try to do with pp and stackless?

On 9/21/07, Chris Lee <c.j.lee at tnw.utwente.nl> wrote:
> I tried a number of things
> 1) Use tasklets to initiate pp jobs. This didn't work, but I suspect I
> could get it to work now that I am more familiar with pp
> 2) Use channels to communicated data between different pp jobs. This
> certainly won't work because pp sets up different python instances,
> though pickling the communication might get around this
> 3) Use channels and tasklets to efficiently switch between setting up
> jobs, starting jobs, and processing the results

Sounds like os threads might be more appropriate,
at least for the job management?

> The problem with this is that the tasklets tend to block while waiting
> for pp jobs to return so it runs as if you were using pp without
> stackless or threading.

Well since stackless threading is cooperative, if the calls to pp
block so will the tasklet.
Hence, that's why I think preemptive os threads would work better in
that situation.

One thing you didn't try, which I would find more interesting is to run
the jobs with stackless, e.g. each process would have it's own schedule.

The issue of synchronization over process bounds is interesting.
I think you need to create a new type of channel class that can
synchronize with the remote processes.

I don't know much about pp, I haven't had the time for it yet,
but another framework comes to mind, the Spread framework.
Spread provides multicast to channel that are reliable and ordered.
The send of a spread multicast doesn't block, and the receive can be
made non-blocking by polling its status.

How is synchronization usually done in pp?

/Johan

-- 
Johan Carlsson
Colliberty Easy Publisher
http://www.easypublisher.com