[Stackless] Thoughts on I/O Driven Scheduler

[Kristján Valur Jónsson]

I actually think the discussion about the performance of soft switching vs hard switching to be irrelevant.
Tasklet switching in any meaningful application happens so rarely that it doesn't show up on the radar.  If your application spends any measurable time just switching contexts, then you ought to think about changing your algorithm.

Now, stackless however was supposed to be able to use its soft switching algorithm to simplify execution of regular python code (by being stackless).  I have no idea whether this turned out to be useful in practice.

As for real world data, I added instrumentation to our Stackless python branch a while back.  I was trying to optimize this time by modifying the stack 'cache'.
Below is a snapshot from one of our game proxies.  We have some 10 times more hard switches than soft switches.  Despite this, hard switching accounts for 0.3% of wallclock time.
The process is running at some 13%cpu, so that means that hard switching for us is consuming some 0.03% of application time.

K

[cid:image001.png at 01CBED3E.471FE920]


From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of Richard Tew
Sent: 28. mars 2011 02:24
To: The Stackless Python Mailing List
Subject: Re: [Stackless] Thoughts on I/O Driven Scheduler

On Sun, Mar 27, 2011 at 8:23 AM, Andrew Francis <andrewfr_ice at yahoo.com<mailto:andrewfr_ice at yahoo.com>> wrote:
At both PyCon 2011 and EuroPython 2010, I was somewhat surprised by the popularity of greenlet based solutions (eventlet, gevent). This is despite the fact that a Stackless Python based solution ought to outperform the aforementioned.

Greenlet is where the attention and interest is, and I do not find it surprising.  It is a lot more approachable to use an extension and gain the bulk of the benefits of Stackless, than to use a custom Python interpreter.  To be honest, I see this as a positive thing that speaks for how token and cumbersome generator coroutines are as a solution for the same sorts of problems.

During PyCon 2011, there were two discussions of particular interest. The first was with Christian Tismer about what makes stackless "stackless" and when does hard switching occur. This is important since soft switching is ten times faster than hard switching.

Of interest to me is how many times in the typically networking scenario does hard switches occur? And maybe alter Stackless to occur these cases.

...

I have a few more ideas but this all for now. Comments?

No idea about Twisted related topics.  I have little understanding of, and no interest in the subject.

Cheers,
Richard.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20110328/554e1626/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 36128 bytes
Desc: image001.png
URL: <http://www.stackless.com/pipermail/stackless/attachments/20110328/554e1626/attachment-0001.png>