[Stackless] Thoughts on I/O Driven Scheduler
andrewfr_ice at yahoo.com
Mon Mar 28 18:04:28 CEST 2011
Date: Mon, 28 Mar 2011 19:53:19 +0800
From: Kristj?n Valur J?nsson <kristjan at ccpgames.com>
To: The Stackless Python Mailing List <stackless at stackless.com>
Subject: Re: [Stackless] Thoughts on I/O Driven Scheduler
<2E034B571A5CE44E949B9FCC3B6D24EE6CD05A87 at exchcn.ccp.ad.local>
Content-Type: text/plain; charset="iso-8859-1"
>I actually think the discussion about the performance of soft switching >vs hard switching to be irrelevant. Tasklet switching in any meaningful >application happens so rarely that it doesn't show up on the radar. If >your application spends any measurable time just switching contexts, then >you ought to think about changing your algorithm.
Depends on the architecture and the type of application one is building. For the most part, I think meaningful applications sit around and wait. Again, I speak from the highly theoretical world of BPEL like applications - something I have only written small examples to test features rather than performance.
However if one has a lot of small agents communicating over channels, then context switching ought to matter. When the numbers are big enough, non-value producing activity like context switches will add up. Maybe an extreme example is the sieve of Eratosthenes.
>Now, stackless however was supposed to be able to use its soft switching >algorithm to simplify execution of regular python code (by being >stackless). I have no idea whether this turned out to be useful in >practice.
Well it seems to me what you are really questioning is whether synchronous message passing is useful. I say yes but I am biased. In my PyCon 2011 talk, I use select() and channels to write the Santa Claus problem. If I threw in join patterns, the solution would be shorter yet. In contrast, look at 'solutions' that use mutexes and semaphores. And we are dealing with only twenty-one threads. The rub is that they are interacting in a non-trivial fashion.
I read Go-Lang Nuts. A lot of people don't get this paradigm. You still see demands for low level stuff like mutexes and what not.
I believe the applications that will most benefit from Stackless are still in their infancy....
>As for real world data, I added instrumentation to our Stackless python >branch a while back. I was trying to optimize this time by modifying the >stack 'cache'.
>Below is a snapshot from one of our game proxies. We have some 10 times >more hard switches than soft switches. Despite this, hard switching >accounts for 0.3% of wallclock time.
>The process is running at some 13%cpu, so that means that hard switching >for us is consuming some 0.03% of application time.
My silly mailer scrubs the png. But I have no doubt about the validity of your data. I suspect most of the time is spent waiting. Also it would be interesting to know how many tasklets were running.
Still, if anything, I wish I was far more competent at instrumenting Stackless code and interpreting results.
More information about the Stackless