[Stackless] parallel processing

Richard Emslie rxe at ukshells.co.uk
Wed Oct 9 18:46:26 CEST 2002


(Sorry chris for abusing stackless mailing list)

Odin is an object database (or my interpretation of what an object
database is) and differs greatly from monolithic sql / relational
databases, in a way that what it is trying to achieve with respect to
scalability / clustering is much simplier... (basically because you access
objects via an interface, and objects cannot share data)

In a nutshell, Odin adopts (2) below and kind of side-steps issues
relating to (1).  The idea with Odin is basically it is built from a
number of cooperating proceses across a cluster of machines.  Within each
node there exists a central microkernel / dynamic request broker which
ultimately acts as an intelligent router for odin objects (persistent
python onjects with database constraints and/or stackless tasklets).  A
number of clients falls off the broker, including a number of stackless
python interpreters (this deals with multiple cpus), a persistence storage
engine, network layers and others.

Communication and object locking is maintained by the request broker and a
large proportion of the system memory (which is shared).  Memory
management of this is controlled roughly 70% by the broker, and the rest
is managed by the clients.  The interaction between clients, the broker
and the shared memory is via short send blocked message passes.

Odin objects / tasklets are migrated from box to box, and even from
interpreter to interpreter (given that tasklet / channels can knock the
socks off of IPC and context switching), depending on the frequency of
access.  I've got know idea how this is going to work yet.

Sorry thats a pretty poor description but I hope that makes at least some
sense...  an explanation of what it does, rather than how it works might
be better.

Cheers,
Richard





On Wed, 9 Oct 2002, Aaron Watters wrote:

> Regarding Odin:
> I'm heading towards finally doing another rewrite of Gadfly based
> in stackless, which might be kind of similar to Odin, maybe, fwiw.
>
> Regarding parallel processing I see 2 issues: (1) using multiple cpus
> in the same process and (2) distributing related tasklets across
> cpus in different processes (maybe on different boxes connected in a
> network).
>
> (1) Requires some basic changes to the Python interpreter like the
> "free threaded python" that Greg Stein played with some years back.
>
> (2) I think can be implemented a number of different ways now, with
> restrictions -- basically you've got to come up with some sort of a protocol
> to do channel communication using sockets and some sort of a notion of
> a services directory or some such...
>
> my 2c.  -- Aaron Watters
>
> Richard Emslie wrote:
>
> >
> >
> >
> >On Wed, 9 Oct 2002, Stephan Diehl wrote:
> >
> >>Hi everyone,
> >>
> >>I'm new to stackless and new to the list (hi chris). I just played aound a
> >>little so far.
> >>I've a question regarding the greater picture of all of that. The features of
> >>stackless (besides being stackless :-) come from the background of parallel
> >>computing and network computing. So far, Python seems to be more of a single
> >>processor language (GIL). Are there any plans or thoughts about building a
> >>parallel python, or some framework that dispatches tasklets to free resources
> >>(on the same machine or a remote computer)?
> >>
> >
> >
> >I'm working on an object database called Odin which effectively does
> >the above (plus a little more).  At the moment it is not even in
> >alpha and requires a rewrite of the object schema (was XML Schema) before
> >the parallel framework can be brought into operation.  Odin has a
> >microkernel / message passing design and will take care of all the object
> >/ tasklet locking, transparent communication (with unix domain sockets /
> >UDP / myrinet bindings), etc.
> >
>
>
>



_______________________________________________
Stackless mailing list
Stackless at www.tismer.com
http://www.tismer.com/mailman/listinfo/stackless



More information about the Stackless mailing list