[Stackless] Thoughts on I/O Driven Scheduler
andrewfr_ice at yahoo.com
Sun Mar 27 01:23:44 CET 2011
At both PyCon 2011 and EuroPython 2010, I was somewhat surprised by the popularity of greenlet based solutions (eventlet, gevent). This is despite the fact that a Stackless Python based solution ought to outperform the aforementioned.
During PyCon 2011, there were two discussions of particular interest. The first was with Christian Tismer about what makes stackless "stackless" and when does hard switching occur. This is important since soft switching is ten times faster than hard switching.
Of interest to me is how many times in the typically networking scenario does hard switches occur? And maybe alter Stackless to occur these cases.
The second talk was with the Twisted team. The subject of a Stackless Twisted Reactor was mentioned and I was told, "why bother - eventlets has Twisted support!" From what I have seen, Twisted support is in the form of blockOns, a technique used by Christopher Armstrong. I use blockOns.
BlockOn is a very effective technique. The bulk of my talk, "Adventures in Stackless Python / Twisted Integration" deal with handling what for now are rather obscure edge cases where dead lock would occur. Since that talk,
I have been interested in making tasklets and callbacks work together more naturally. In part I believe callbacks become unnecessary.
Now this echos back to a conversation I had at EuroPython 2010. Things boiled down to "why do you need a scheduler?" Well in the case of both gevent and twisted, there is a still a "scheduler" that is essentially non pre-emptive and can only schedule I/O bound "tasks." However this may be good enough for most people's needs.
The Twisted conversation ended with me sitting with the Twisted team and learning the basics of writing a custom reactor. Glyph and J.P kindly pointed out the pit-falls.
Using stackless.py as a test bed, I would write a Stackless Reactor. However instead of using CPython with greenlets. I would use PyPy-c which to the best of my knowledge is solely soft-switching. I would replace the stackless.run() with a new stackless.run that would be a modified Twisted reactor. Of course tasklets would not know the difference.
Amongst the things I would do, is replace the blockOn, with a lighter "channel" class that could transfer data into the tasklet and schedule it. Also appropriate Twisted class methods could be transparently called, i.e. client.getPage().
And to cover edge cases, protocol instances would run in a tasklet (after all I want to build stuff that I can use for my own work).
I have a few more ideas but this all for now. Comments?
More information about the Stackless