richard.m.tew at gmail.com
Sun Aug 22 09:20:45 CEST 2010
For anyone who is using stacklesssocket.py, I've just upgraded it so
that it passes the non-UDP Python standard library socket unit tests
(tested on 2.6 and 2.7). Note that it actually passes all of the UDP
unit tests except for two, so that's a good sign that the changes I
have made make it inherently more correct. I would also add that the
tests do not pass on 2.5, but this looks to be due to minor
exception/behaviour related changes and it should work as well as it
ever did with 2.5.
I still have to do stress testing work on these changes.
Some relevant things you might want to know about the changes:
- The way it worked with the socket module and asyncore and garbage
collection was to have a dictionary ('asyncore.socket_map') which weak
references to tasklets were stored in. This was bad, as it meant that
dead weak references would sit in the dictionary until events
propagated and they were removed. This in turn would cause calls to
'asyncore.poll(seconds)' to have reference errors raised out of it
under load as it choked on those dead weak references. Now it
replaces 'asyncore.socket_map' with a WeakValueDictionary and when
weak references die, they are removed immediately and there no more
reference errors happen in the described way.
- If you set 'stacklesssocket._sleep_func' to your sleep function,
then the socket timeout mechanism will become enabled and take
advantage of it (e.g. 'my_socket.settimeout(num_seconds'). You can
also implement your own timeout mechanism with a
'stacklesssocket._timeout_func', but that is a little more complex and
suits in the situation where you doing interthread communication with
channels and wish to avoid the concerns that come with tasklets
belonging to different threads and channel operations.
- You can now call 'my_socket.setblocking(0)' and it will work as expected.
- A wider swatch of the socket object API is now supported, with
regard to weird functions like recv_into and the like.
- No internal buffering of data, which means it should be more
optimal. You get the data the socket provides directly (well, as
direct as an asynchronous operation can be made to look synchronous
Most of the work in this change was in contriving a script
('stacklesssocket_unittest.py') that would set up an environment where
the Python unit tests could be run with the stacklesssocket module
monkey patched into place. This involved writing a module
('stacklessthread.py') that turns a range of thread blocking
operations into tasklet blocking operations, when they happen on the
main thread (where the expectation is that the scheduler is only being
run on that thread). While over the years we have fleshed out
Christian's threading support for special Stackless functionality,
there are still a few inconsistencies and when you are dealing with
Python threads and Stackless they tend to pop up. This is the reason
why 'stacklesssocket_unittest.py' supports avoiding creating tasklets
on Python threads other than the main one, and why
'stacklesssocket.py' needs a '_timeout_func' to access that.
Note that if you are using the 3.x version of the module, this has not
been updated. But if I recall correctly, the 3.x version is simply
the 2.x version with required structural changes to match how things
have changed. It should be an easy project to take the 3.x version
and apply the 2.x changes. And to adapt the new unit test script that
will run the 3.x socket unit tests against it.
Author: richard.m.tew at gmail.com
Date: Sat Aug 21 23:42:50 2010
Log: Minor changes that allow the unit tests to run against Python
2.5, which does not have 'some_thread.ident'.
The unit tests pass in Stackless Python 2.7 exactly the same as
Stackless Python 2.6.
Author: richard.m.tew at gmail.com
Date: Sat Aug 21 23:26:45 2010
Log: Fleshed out stacklesssocket.py:
- Calls to asyncore.poll() and similar functions no longer error
occasionally under heavy use with reference errors when a garbage
collected socket is still in its socket map. This is the difference
between storing weak references in a dictionary manually, and using
- If the user of the module sets a value for '_sleep_func' or
'_timeout_func', then socket timeout support is enabled via the
standard function 'mysocket.settimeout(seconds)'.
- Stackless sockets are "blocking", but block the calling tasklet
rather than the current thread. Now they can be toggled to be
non-blocking via the standard function 'mysocket.setblocking(0)',
which will mean that if they will behave like normal non-blocking
- All non-UDP standard Python socket unit tests pass when it is run on
Python 2.6, and only two of the UDP unit tests fail or error. The
unit tests are run against stacklesssocket.py with
stacklesssocket_unittest.py. The two failing UDP-related tests are
BasicUDPTest.testSendtoAndRecv and UDPTimeoutTest.testUDPTimeout.
- The internal stacklesssocket.py have been tested to pass against
Python 2.5 and 2.6.
Fleshed out stacklessthread.py:
- This covers monkey patching a variety of thread blocking operations,
so that if they are happening in the main thread, they are turned into
tasklet blocking operations (and farmed out to a new custom created
More information about the Stackless