[Stackless] parallel processing
rxe at ukshells.co.uk
Wed Oct 9 18:46:26 CEST 2002
(Sorry chris for abusing stackless mailing list)
Odin is an object database (or my interpretation of what an object
database is) and differs greatly from monolithic sql / relational
databases, in a way that what it is trying to achieve with respect to
scalability / clustering is much simplier... (basically because you access
objects via an interface, and objects cannot share data)
In a nutshell, Odin adopts (2) below and kind of side-steps issues
relating to (1). The idea with Odin is basically it is built from a
number of cooperating proceses across a cluster of machines. Within each
node there exists a central microkernel / dynamic request broker which
ultimately acts as an intelligent router for odin objects (persistent
python onjects with database constraints and/or stackless tasklets). A
number of clients falls off the broker, including a number of stackless
python interpreters (this deals with multiple cpus), a persistence storage
engine, network layers and others.
Communication and object locking is maintained by the request broker and a
large proportion of the system memory (which is shared). Memory
management of this is controlled roughly 70% by the broker, and the rest
is managed by the clients. The interaction between clients, the broker
and the shared memory is via short send blocked message passes.
Odin objects / tasklets are migrated from box to box, and even from
interpreter to interpreter (given that tasklet / channels can knock the
socks off of IPC and context switching), depending on the frequency of
access. I've got know idea how this is going to work yet.
Sorry thats a pretty poor description but I hope that makes at least some
sense... an explanation of what it does, rather than how it works might
On Wed, 9 Oct 2002, Aaron Watters wrote:
> Regarding Odin:
> I'm heading towards finally doing another rewrite of Gadfly based
> in stackless, which might be kind of similar to Odin, maybe, fwiw.
> Regarding parallel processing I see 2 issues: (1) using multiple cpus
> in the same process and (2) distributing related tasklets across
> cpus in different processes (maybe on different boxes connected in a
> (1) Requires some basic changes to the Python interpreter like the
> "free threaded python" that Greg Stein played with some years back.
> (2) I think can be implemented a number of different ways now, with
> restrictions -- basically you've got to come up with some sort of a protocol
> to do channel communication using sockets and some sort of a notion of
> a services directory or some such...
> my 2c. -- Aaron Watters
> Richard Emslie wrote:
> >On Wed, 9 Oct 2002, Stephan Diehl wrote:
> >>Hi everyone,
> >>I'm new to stackless and new to the list (hi chris). I just played aound a
> >>little so far.
> >>I've a question regarding the greater picture of all of that. The features of
> >>stackless (besides being stackless :-) come from the background of parallel
> >>computing and network computing. So far, Python seems to be more of a single
> >>processor language (GIL). Are there any plans or thoughts about building a
> >>parallel python, or some framework that dispatches tasklets to free resources
> >>(on the same machine or a remote computer)?
> >I'm working on an object database called Odin which effectively does
> >the above (plus a little more). At the moment it is not even in
> >alpha and requires a rewrite of the object schema (was XML Schema) before
> >the parallel framework can be brought into operation. Odin has a
> >microkernel / message passing design and will take care of all the object
> >/ tasklet locking, transparent communication (with unix domain sockets /
> >UDP / myrinet bindings), etc.
Stackless mailing list
Stackless at www.tismer.com
More information about the Stackless