[Stackless] Creating a relation between tasklets to forward exceptions

Gabriel Lavoie glavoie at gmail.com
Mon Sep 6 15:31:59 CEST 2010


Hello Richard,
     you made a good point by saying Stackless is meta-framework. I also see
my layer as a meta-layer for distributed applications on top of which
schedulers or monitoring tools could be written (and distributed
applications themselves). About the parent/child relationship, I already
made my own way of managing this (example from the first post). It works
pretty well and parent tasklet migration is managed automatically.

About making my code available, it already is but unfortunately it only
works on top of latest trunk PyPy (because of coroutine.throw() that was
added to it). I also used the PyPy Proxy Object Space for distributed
objects. Here's the code:
https://w3.mutehq.net/websvn/wildchild/dstackless/trunk/

A description of the API is available there but it's only in french for now:
https://w3.mutehq.net/wiki/maitrise/API_DStackless

Most of the API are used exactly like the Stackless counterpart. Here's what
I added:
dstackless.identify(("hostname", 6473)): Identify an interpreter so others
can connect to it for tasklet migration. It takes a tuple composed of an IP
address (hostname too) and a port on which the interpreter can be contacted.
The interpreter will listen to this port.

dstackless.connect() is shown but is now transparent. You can ignore it.

dstackless.tasklet(): Used like stackless.tasklet() but three parameters
were added. "label" to name a tasklet. It was already there with
stackless.py from PyPy and I use it for debugging purpose. Right now, every
"print" from tasklets are also prefixed with this label. "ioconfig" to
specify where to redirect stdout for the tasklet (for now only a node
identifier can be used). Finally "parent" to create a parent/child
relationship. This parameters receives directly the dstackless.tasklet()
object of the parent and if the child dies with an exception raised, it will
be forwarded to this parent.

To simplify migration, I create a new module with its own global dictionnary
for every new tasklets created. This module is populated with the content of
the same .py file from where the function used for the tasklet entry point
was found. This may sound strange but with this I wanted to discourage the
use of global memory to share data between tasklets. It also looks a bit
like the behaviour of fork() that creates two separate processes that can
share communication channels (pipe()).

ref_tasklet.move(("hostname", 6474)): Move a live tasklet to the specified
node. Channel connections are kept and all objects the tasklet refers to are
copied to the new host unless they are distributed objects. With those, only
a remote reference will be created. The global dictionnary of the tasklet
follows it.

o = []
distributed_o = dstackless.ref_object(o): Create a distributed object. If
"distributed_o "sent to a remote node through a channel of follows a
migrated tasklet, "o" will stay on the node where it was created. All
request to "distributed_o" will then be redirected to the node hosting "o".
I plan to also make add a move() API to distributed objects to move them
like tasklets... Actually, a new method on those isn't the best way of doing
it. I should make a single dstackless.move() API that could move either
tasklets or distributed objects.

I experimented a lot with every parts of this project and I know some are
not that clean. At least, I have a working proof of concept for my master
degree and I can improve after! It was also my first real project with the
Python language and I learned a lot over time. I still need to improve my
Python skills. ;)

Also, I hope to have some time soon to translate some part of this wiki in
English and to write an article, also in English. But before, I don't have
much time to left to finish my thesis and this is my priority. I'll keep you
updated!

Gabriel

2010/9/6 Richard Tew <richard.m.tew at gmail.com>

> On Mon, Sep 6, 2010 at 3:06 AM, Gabriel Lavoie <glavoie at gmail.com> wrote:
> > how to manage this problem with Stackless. On my side, it doesn't help
> that
> > much because I have to work with many tasklets executed on different
> > interpreters but still exchanging data through channels. The idea of my
>
> Your use case is of course a good example of where Stackless would
> never have provided the ability to do this.  This would always belong
> in the meta-framework you wrap around Stackless to get the
> distribution.  It, to me, highlights that keeping Stackless as a
> generic base set of functionality, was the right choice.
>
> > Master Degree project is to make Stackless Python distributed with easy
> > tasklet migration between interpreters. It's not perfect but right now I
> > have hybrid channels that can exchange data between local or remote
> > tasklets. I made some kind of distributed objets using PyPy proxy object
>
> That's the easy part :-)
>
> > space for objets that shouldn't be copied when they are sent to a remote
> > tasklet (like big objects). I also tried to manage the best I could the
> case
> > of global memory and shared object. This could still be improved.
>
> That's the hard part.
>
> > The idea of creating a parent/child relation between tasklets is manage
> the
> > case when two tasklets in relation are not executed on the same
> interpreter.
> > If the remote tasklet dies or the interpreter where it was executed dies,
> > the parent tasklet should be notified in some way. Trying to manage this
> > with a weakref would be a nightmare for the programmer if the remote
> tasklet
> > moves a lot between interpreters. I want this to be managed by my
> library.
>
> The more and more I read this, the more it sounds like your framework
> needs to be managing it.
>
> > Also trying to wrap the function used to create a tasklet isn't really
> > useful when the interpreter running it dies and the parent is elsewhere.
> In
> > any case, the way you described to do it is still some useful material
> for
> > my thesis. ;)
>
> I look forward to when you finish it and send a link to this mailing list
> :-)
>
> Richard.
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>



-- 
Gabriel Lavoie
glavoie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20100906/da3d015b/attachment.html>


More information about the Stackless mailing list