[Stackless] stacklesssocket.py

Alejandro Castillo pyalec at gmail.com
Sun Aug 22 10:12:13 CEST 2010


Hello,

Good news for stacklesssocket :)

I would like to know more about the inconsistencies when dealing with
Threads in Stackless.
I think I have bumped into one or more of them.

Cheers,
Alejandro

2010/8/22 Richard Tew <richard.m.tew at gmail.com>:
> Hi,
>
> For anyone who is using stacklesssocket.py, I've just upgraded it so
> that it passes the non-UDP Python standard library socket unit tests
> (tested on 2.6 and 2.7).  Note that it actually passes all of the UDP
> unit tests except for two, so that's a good sign that the changes I
> have made make it inherently more correct.  I would also add that the
> tests do not pass on 2.5, but this looks to be due to minor
> exception/behaviour related changes and it should work as well as it
> ever did with 2.5.
>
>  http://code.google.com/p/stacklessexamples/source/browse/trunk/examples/networking/stacklesssocket.py
>
> I still have to do stress testing work on these changes.
>
> Some relevant things you might want to know about the changes:
> - The way it worked with the socket module and asyncore and garbage
> collection was to have a dictionary ('asyncore.socket_map') which weak
> references to tasklets were stored in.  This was bad, as it meant that
> dead weak references would sit in the dictionary until  events
> propagated and they were removed.  This in turn would cause calls to
> 'asyncore.poll(seconds)' to have reference errors raised out of it
> under load as it choked on those dead weak references.  Now it
> replaces 'asyncore.socket_map' with a WeakValueDictionary and when
> weak references die, they are removed immediately and there no more
> reference errors happen in the described way.
> - If you set 'stacklesssocket._sleep_func' to your sleep function,
> then the socket timeout mechanism will become enabled and take
> advantage of it (e.g. 'my_socket.settimeout(num_seconds').  You can
> also implement your own timeout mechanism with a
> 'stacklesssocket._timeout_func', but that is a little more complex and
> suits in the situation where you doing interthread communication with
> channels and wish to avoid the concerns that come with tasklets
> belonging to different threads and channel operations.
> - You can now call 'my_socket.setblocking(0)' and it will work as expected.
> - A wider swatch of the socket object API is now supported, with
> regard to weird functions like recv_into and the like.
> - No internal buffering of data, which means it should be more
> optimal.  You get the data the socket provides directly (well, as
> direct as an asynchronous operation can be made to look synchronous
> using channels).
>
> Most of the work in this change was in contriving a script
> ('stacklesssocket_unittest.py') that would set up an environment where
> the Python unit tests could be run with the stacklesssocket module
> monkey patched into place.  This involved writing a module
> ('stacklessthread.py') that turns a range of thread blocking
> operations into tasklet blocking operations, when they happen on the
> main thread (where the expectation is that the scheduler is only being
> run on that thread).  While over the years we have fleshed out
> Christian's threading support for special Stackless functionality,
> there are still a few inconsistencies and when you are dealing with
> Python threads and Stackless they tend to pop up.  This is the reason
> why 'stacklesssocket_unittest.py' supports avoiding creating tasklets
> on Python threads other than the main one, and why
> 'stacklesssocket.py' needs a '_timeout_func' to access that.
>
>  http://code.google.com/p/stacklessexamples/source/browse/trunk/examples/networking/stacklesssocket_unittest.py
>  http://code.google.com/p/stacklessexamples/source/browse/trunk/examples/threading/stacklessthread.py
>
> Note that if you are using the 3.x version of the module, this has not
> been updated.  But if I recall correctly, the 3.x version is simply
> the 2.x version with required structural changes to match how things
> have changed.  It should be an easy project to take the 3.x version
> and apply the 2.x changes.  And to adapt the new unit test script that
> will run the 3.x socket unit tests against it.
>
>  http://code.google.com/p/stacklessexamples/source/browse/trunk/examples/networking/stacklesssocket30.py
>
> Cheers,
> Richard.
>
> --
>
> Revision: 143
> Author: richard.m.tew at gmail.com
> Date: Sat Aug 21 23:42:50 2010
> Log: Minor changes that allow the unit tests to run against Python
> 2.5, which does not have 'some_thread.ident'.
>
> The unit tests pass in Stackless Python 2.7 exactly the same as
> Stackless Python 2.6.
> http://code.google.com/p/stacklessexamples/source/detail?r=143
>
> Modified:
>  /trunk/examples/networking/stacklesssocket_unittest.py
>  /trunk/examples/threading/stacklessthread.py
>
> Revision: 142
> Author: richard.m.tew at gmail.com
> Date: Sat Aug 21 23:26:45 2010
> Log: Fleshed out stacklesssocket.py:
> - Calls to asyncore.poll() and similar functions no longer error
> occasionally under heavy use with reference errors when a garbage
> collected socket is still in its socket map.  This is the difference
> between storing weak references in a dictionary manually, and using
> WeakValueDictionary instead.
> - If the user of the module sets a value for '_sleep_func' or
> '_timeout_func', then socket timeout support is enabled via the
> standard function 'mysocket.settimeout(seconds)'.
> - Stackless sockets are "blocking", but block the calling tasklet
> rather than the current thread.  Now they can be toggled to be
> non-blocking via the standard function 'mysocket.setblocking(0)',
> which will mean that if they will behave like normal non-blocking
> sockets.
> - All non-UDP standard Python socket unit tests pass when it is run on
> Python 2.6, and only two of the UDP unit tests fail or error.  The
> unit tests are run against stacklesssocket.py with
> stacklesssocket_unittest.py. The two failing UDP-related tests are
> BasicUDPTest.testSendtoAndRecv and UDPTimeoutTest.testUDPTimeout.
> - The internal stacklesssocket.py have been tested to pass against
> Python 2.5 and 2.6.
>
> Fleshed out stacklessthread.py:
> - This covers monkey patching a variety of thread blocking operations,
> so that if they are happening in the main thread, they are turned into
> tasklet blocking operations (and farmed out to a new custom created
> thread).
>
> http://code.google.com/p/stacklessexamples/source/detail?r=142
>
> Modified:
>  /trunk/examples/networking/stacklesssocket.py
>  /trunk/examples/networking/stacklesssocket_unittest.py
>  /trunk/examples/threading/stacklessthread.py
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>



More information about the Stackless mailing list