[Stackless] irc threads

Richard Tew richard.m.tew at gmail.com
Mon Jul 20 14:03:34 CEST 2009


On Mon, Jul 20, 2009 at 8:42 PM, Rach<rachie at oasiz.net> wrote:
> Hi Richard, thanks for the reply.
> Unfortunately the stacklesssocket allows even less (511 concurrent
> connections):
> CONN: 506
> Client 127.0.0.1:56879 connected...
> CONN: 507
> Client 127.0.0.1:56880 connected...
> CONN: 508
> Client 127.0.0.1:56881 connected...
> CONN: 509
> Client 127.0.0.1:56882 connected...
> CONN: 510
> Client 127.0.0.1:56883 connected...
> CONN: 511
> Traceback (most recent call last):
>   File "install.py", line 109, in <module>
>     stackless.run()
>   File "C:\Users\Rob\Desktop\stacklesssocket.py", line 77, in ManageSockets
>     asyncore.poll(0.05)
>   File "C:\stackless26\lib\asyncore.py", line 130, in poll
>     r, w, e = select.select(r, w, e, timeout)
> ValueError: too many file descriptors in select()
>
> Any idea where to go from here? (running Windows Vista)

Bringing this back to the list, this is not a limitation with
'stacklesssocket', it is a limitation with the implementation of the
'select' module either because that is how Windows rolls, or because
Python's wrapping of it adds this constraint.  This is interesting to
find out, it implies that 'stacklesssocket' needs to be rewritten at
some point to either aggregate sockets into lumps of less than 512 to
pass to select, or to be rewritten to use IOCP on Windows.  But that
is a tangential note, not something which helps you.

Given that your goal is not to use Stackless in a scalable way but
rather to have your Windows computer accept what you consider to be a
reasonable number of connections, you would get the best help if you
went to forums which covered that aspect of development.

Reading your posts, it occurs to me that with this in mind what makes
sense is to fob you off onto Twisted :-)  Twisted is a polished
networking solution, and if you want to get a solution which through a
convolution of design patterns, functionality, abstraction and
boilerplate achieves a level of quality 'stacklesssocket' will most
likely lack for the forseeable future, then it is what you should try
to see how far your networking project can scale.

It has been a while since I looked at Twisted, but I seem to recall
that it has different "reactors"  which can choose between different
socket IO mechanisms.  And indeed, after a quick web search, this page
backs that up:

  http://docs.huihoo.com/python/twisted/howto/choosing-reactor.html

As you can see in that page, if you build a solution on Twisted, it
already has the low-level support needed to use something other than
the 'select' module.  And indeed, again after a quick web search, this
page backs that up:

  http://twistedmatrix.com/pipermail/twisted-python/2009-March/019273.html

In that post, you can see one of the Twisted brain trust confirming
that the limit for "select" is 512.  And he also recommends trying
alternate IO mechanisms (i.e. IOCP).

So, there's something for you to try:

Install Twisted and have it use the IOCP reactor (or one of the others
it might provide for Windows), then see how many connections IOCP can
hold.  Whether you are intentionally not using Twisted does not
matter, if IOCP holds as many connections as you want to be held, then
it is indicative that an IOCP based solution will work for you, in
Twisted or not.  Maybe you can then choose to use Twisted, maybe you
can look for an IOCP based solution with less dependencies if that's a
concern, maybe you can write your own... or maybe you can pay someone
to write one for you.  Maybe you'll still have connection limits, even
with IOCP.  I don't know.

Cheers,
Richard.




More information about the Stackless mailing list