[Stackless] Multi core/cpu?

[Michael Sparks]

On Sunday 21 October 2007 00:01, Anand Patil wrote:
> As long as I'm posting: I've also been waiting for a truly concurrent
> microthread solution for Python. What I'd really like to find is an
> implementation of the actor model, in which each actor gets its own private
> microthread of execution and memory space, but actors can pass data around
> by reference if they want to. Actors could only read data produced by
> another actor. Is there a term for this system, and does anyone know of an
> implementation in any language?

That sounds very similar to what we do with Kamaelia. Using concurrency in 
Kamaelia is second nature since everything is dataflow. (personally I use the 
term CSP-like, since that's the core inspiration we take, but we aim to wrap 
things up in friendly metaphors) The difference from "normal" is that the 
concurrency is useful to make it easier to express what the program is doing 
rather than worrying about "can I squeeze every ounce of power from this 
machine".

We don't do multiple processes at present, but using threads or microthreads
(generators, not what stackless calls microthrads) is identical (except for
the yield keyword). The interesting thing about this is that (apparently) this
should mean we automagically make use of multicore CPUs under
IronPython (under CPython the GIL gets in the way). I've not actually
tried this, but apparently that should be the case.

Various examples in the cookbook:
   * http://kamaelia.sourceforge.net/Cookbook

Most useful (to me) app is this one:
   * http://kamaelia.sourceforge.net/KamaeliaGrey
     * source: http://tinyurl.com/2sbjxl

Or perhaps the whiteboard - source:
   * http://tinyurl.com/36rtfy
   * mainfile: http://tinyurl.com/2upquw

Example that shows a variety of concepts:
    * http://tinyurl.com/37hj5j

Pycon UK presentation files:
   * http://www.slideshare.net/kamaelian/building-systems-with-kamaelia
     ( need to figure out why slideshare broke all the fonts! :-) )

Euro OSCON presentation: (walks through a substantial system)
http://www.slideshare.net/kamaelian/timeshift-everything-miss-nothing-mashup-your-pvr-with-kamaelia
Code that presentation walks through:
    * http://tinyurl.com/3aq9an

The component/pipeline nature makes it easy to evolve and compose systems
into new ones. The nice thing is you tend to focus on functionality and gain
naturally concurrent software along the way. We can measure that by plugging
an introspector onto the system after the fact. In the case of the whiteboard
IIRC there's around 150 components which run in parallel[1], whereas in the
greylisting app it's more around 5-7 components when no-one's connected
and then a handful more per connection.
   [1] The whiteboard acts as not just a whiteboard, but also as a server,
      sharing content with any client whiteboards, who can themselves be
      servers. They also handle audio in/out with local remixing, and
      paginated (synchronise) displays. (makes recording sessions easy for
      example)

Stuff we've made using this varies from log processing through handling of 
digital TV, shot change detection, video transcoding, timeshifting and 
cropping, work on mobiles, through handwriting (well, stroke) recognition, 3D 
tools, through to bit torrent distribution, through games, through to SMTP 
greylisting. I'm in the process of using this to write a simple LOGO 
interpreter for my son as well. (probably extend the language there to be 
either python like using a generic language parser I created a few years 
back, and probably hook in kamaelia concepts there as well) *ALL* of these 
systems were highly concurrent but in a natural to use & code manner.

see http://kamaelia.sourceforge.net/Components for a full list of what's 
currently sitting on the SVN HEAD. (nowhere near the complete list)

Personally I find this a more fun way of playing with code & data.  :-)
The fact it'll be trivial (bar some interesting edge cases...!) to make
multicore (just check everything works as expected in IronPython
under mono) when I get a multicore system is something I quite like
the idea of. (OK, this was a core design goal, but it looks a reasonable
proposition now)

There's also proof of concept code translations into Ruby & C++. Shedskin can 
also compile a massively simplified version from python down to C++. A brave 
soul has also done our "build your own mini-core" [2] using java instead of 
python.
   * http://kamaelia.sourceforge.net/MiniAxon/


Michael.