[Stackless] More on a Stackless Select

Andrew Francis andrewfr_ice at yahoo.com
Sat Apr 17 05:35:47 CEST 2010


Hi Christian:

--- On Fri, 4/16/10, Christian Tismer <tismer at stackless.com> wrote:

> That is good to hear. Stackless, as it is, puts a big
> impact on the developer. The changes to the core are way too big.
> Developing modules against it is not recommended now.

Fair enough. Again, through prototyping I am interested in figuring
out what the minimum number of changes that would be necessary to
implement most of a Select like mechanism in Python. 

> Not sure what you mean. Blocked is meant to be an
> indicator.
> Maybe I overloaded it.

Blocked indicates both that the tasklet is blocked and on which
operation (receive, send).

> _channel is an implementation detail, to find the channel
> if necessary. In case of multiple channels, it should become
> a set of channels. The channel structure might get
> extended, or we create a channel set. Not sure which structure would
> be best. Again this is better to model in Python.

Yes I am modeling in Python. Let me work on this. I guess the main things
is any selector implementation needs to know

1) If it is blocked, what was the <channel, operation, value> tuple that
became active. 

2) what channels is the select waiting on. 

Right now, my selector is an object.

> I think for prototyping you should try to implement
> multi-channels on top of channels. You need to use a tasklet 
>per channelthat does this communication. The point is to wake up when any >of the channels becomes active. Then all other waiting helper
> tasklet  get an exception to quit.

A tasklet per channel is not necessary. I posted in go-language-nuts
and got responses from Rob Pike, Russ Cox and Ian Taylor. Internally Stackless, Newsqueak, Limbo, and Go are different. However Russ gave me enough information for me to figure out what I was doing wrong and
implement a solution that does not involve playing with priorites (i.e., forcing the selector tasklet to immediately schedule). So far it is looking good but I need to run more tests to make sure I am out of the
woods. 

> To make this efficient is not trivial. Select suffers from
> the call  overhead. If you want to wait on 10000 channels, then 
>when one fires, the select is done, and for the next operation, 
>you will have to select the 10000
> channels again. You need a sophisticated data structure, a
> channelset, that stays intact after a call.
> The channel action would then just remove the tasklet and
> the channel from the set.

For this reason, I don't implement select the same way as Newsqueak.
I don't remove the tasklet from the unactivated channels. I am not sure what Go does. I need to look more closely at its code. I make the assumption that the common select paradigm is:

for flag {
   select {
   }
}

why tear down only to set things up again? 

> I never got my thought about this clear enough to implement
> it. Again, model this in Python.

And yes my data structures are starting to get complex. Hopefully by Sunday, I can post a new stackless.py with examples. Stackless.py has
bugs but it is really useful. 

AF> 4) To avoid race conditions or really really
AF> complicated data structures, the tasklet with the selector
AF> has to be scheduled immediately. This implies that anything
AF> using a select like mechanism will run with high priority.
AF> Then again, it is acting like a dispatcher in an event
AF> loop.
   
> Not really. The action becomes two-fold. The reaction on
> the channel activity is one thing. The tasklet gets freed from the
> channel set. But it does not have to run immediately. I would implement
> this with a helper tasklet.

You are right: I don't need to schedule immediately. From reading 
Russ's remarks, I need to do two things 1) Store the result (channel, operation, value) from each peer tasklet in a separate place in the selector tasklet. 2) traverse these results in order (Russ talks 
about traversing locks in order but that does not apply to Stackless). I have tested this quick-and-dirty and it seems to work. I need to adapt my existing structures.

> Once again: I highly dis-recommend writing any C extensions for
> Stackless. Not only because stackless is going to change internally a
> lot, but also because of upcoming Psyco support.

Thanks for the head's up. Besides, my C Kung Fu is pretty rusty :-)
Again, my plan is to implement this in Python and figure out the 
minimum amount of changes in Stackless to accommodate a select mechanism.

> See my other post on that, to come right now.
> Whatever you write in Python is subject to compilation.
> It will most probably accelerated by Psyco. And if it's
> not,then I will put effort in to support it, because that
> bringsmuch more benefits than optimizing C code, which is
> very limited and not flexible.

Once again Christian, thanks for your input! And I am looking forward
to the Stackless/Psyco fusion (I see your post!)

Cheers,
Andrew 


      



More information about the Stackless mailing list