[Stackless] Thoughts on I/O Driven Scheduler

Andrew Francis andrewfr_ice at yahoo.com
Mon Mar 28 18:04:28 CEST 2011


Hi Kristjan:

Message: 1
Date: Mon, 28 Mar 2011 19:53:19 +0800
From: Kristj?n Valur J?nsson <kristjan at ccpgames.com>
To: The Stackless Python Mailing List <stackless at stackless.com>
Subject: Re: [Stackless] Thoughts on I/O Driven Scheduler
Message-ID:
    <2E034B571A5CE44E949B9FCC3B6D24EE6CD05A87 at exchcn.ccp.ad.local>
Content-Type: text/plain; charset="iso-8859-1"

>I actually think the discussion about the performance of soft switching >vs hard switching to be irrelevant. Tasklet switching in any meaningful >application happens so rarely that it doesn't show up on the radar.  If >your application spends any measurable time just switching contexts, then >you ought to think about changing your algorithm.

Depends on the architecture and the type of application one is building. For the most part, I think meaningful applications sit around and wait. Again, I speak from the highly theoretical world of BPEL like applications - something I have only written small examples to test features rather than performance. 

However if one has a lot of small agents communicating over channels, then context switching ought to matter. When the numbers are big enough, non-value producing activity like context switches will add up. Maybe an extreme example is the sieve of Eratosthenes.

>Now, stackless however was supposed to be able to use its soft switching >algorithm to simplify execution of regular python code (by being >stackless).  I have no idea whether this turned out to be useful in >practice.

Well it seems to me what you are really questioning is whether synchronous message passing is useful. I say yes but I am biased. In my PyCon 2011 talk, I use select() and channels to write the Santa Claus problem. If I threw in join patterns, the solution would be shorter yet.  In contrast, look at 'solutions' that use mutexes and semaphores. And we are dealing with only twenty-one threads. The rub is that they are interacting in a non-trivial fashion. 

I read Go-Lang Nuts. A lot of  people don't get this paradigm. You still see demands for low level stuff like mutexes and what not.

I believe the applications that will most benefit from Stackless are still in their infancy....


>As for real world data, I added instrumentation to our Stackless python >branch a while back.  I was trying to optimize this time by modifying the >stack 'cache'.

>Below is a snapshot from one of our game proxies.  We have some 10 times >more hard switches than soft switches.  Despite this, hard switching >accounts for 0.3% of wallclock time.

>The process is running at some 13%cpu, so that means that hard switching >for us is consuming some 0.03% of application time.

My silly mailer scrubs the png. But I have no doubt about the validity of your data. I suspect most of the time is spent waiting. Also it would be interesting to know how many tasklets were running. 

Still, if anything, I wish I was far more competent at instrumenting Stackless code and interpreting results. 

Cheers,
Andrew


      



More information about the Stackless mailing list