[Stackless] irc threads
andrewfr_ice at yahoo.com
Sat Jul 18 02:10:07 CEST 2009
Date: Wed, 15 Jul 2009 09:42:58 +0000
From: Kristj?n Valur J?nsson <kristjan at ccpgames.com>
To: Kristj?n Valur J?nsson <kristjan at ccpgames.com>, Larry Dickson
<ldickson at cuttedge.com>
Cc: Henning Diedrich <hd at authentic-internet.de>,
"stackless at stackless.com" <stackless at stackless.com>
Subject: Re: [Stackless] irc threads
<930F189C8A437347B80DF2C156F7EC7F057FE8557D at exchis.ccp.ad.local>
Content-Type: text/plain; charset="iso-8859-1"
>select() as such is recognized as scaling badly, even on unix() which is >why (I've been told) there are now alternatives for asynchronous IO >commonly available on linux.
My understanding is that select() is slow when it comes to large number of descriptors due to the copying of file descriptors and excessive scanning of descriptors. I believe on Linux, epoll() is a preferred alternative. To me, signals just looks messy....
>Also, I'd like to add that we don't generally want to block, even for >select(). In our framework, the application may have numerous tasks to
>do when it is not servicing IO. This necessitates using select to poll
>all the sockets, with the associated setup overhead.
This was a part of the problem I encountered two years ago - the reactor tasklet (which just happened to be Twisted) would block the application until it received input. Consequently other tasklets that could run, did not run. Luckily this problem was relatively easy to solve....
That said Kristjan, I think it is a balancing act to effectively do network I/O and use a single OS thread. From the tests I have so far conducted, I think one of the main tradeoffs is between response time and transactions per some time unit.
One of these days, I will conduct more structured tests to see how sensitive performance is, all other things being equal, to the following factors:
1) The I/O system call (i.e., select(), epoll())
2) The number of context switches (controlled through channel preferences)
3) The number of times a particular I/O call is made (how frequently do we call select() - this addresses setup overhead).
4) The use of an optimizer such as Psyco on the network (read reactor tasklet).
More information about the Stackless