[Stackless] multiprocessing module and stackless threads

Kristján Valur Jónsson kristjan at ccpgames.com
Fri Jan 3 14:15:13 CET 2014


threading._Event is the actual event class.
threading.Event is a factory function.  I think the intention of the library designers was to use factory functions rather than classes directly to have the freedom to change implementation later.
Why Django is doing it this way, I don't know, but for the purposes of monkeypatching, you should set
threading._Event = threading.Event
so that you get your stackless-aware event object in there.

From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of Robert Babiak
Sent: 3. janúar 2014 07:07
To: The Stackless Python Mailing List
Subject: Re: [Stackless] multiprocessing module and stackless threads

hmm.

I am seeing some strange things (using mac os 10.9)

When i do a patch all I got a error from the Django app, they apparently use the threading._Event class that isn't in the monkeyed version. I worked around that with a threading._Event = threading.real_threading._Event

All but the web server ran fine (wx python, and several concurrent tasklets).

The web server part is the problem, I dropped out my Django app and used the wsgiref.simple_server and demo app.

breaking the patch_app() function out into
mp.patch_misc()

    mp.patch_thread()
    mp.patch_threading()
    import threading
    threading._Event = threading.real_threading._Event

    mp.patch_select()
    #mp.patch_socket()
    #mp.patch_ssl()

I found that if the sockets and ssl are not patched then the web server works as expected, but very slow (5-50 seconds for request responses) and longer if it was transferring a file like a image or CSS file. It seems like it is possibly spending a lot of time doing other tasklets, I am not sure. Kristjan had a nice tasklet profiling tool in Eve, is there something similar in the standard stackless?

Then switched the thread over to a tasklet for handling the web requests, to see if the separate thread was causing problem.

This made things worse. It could be because the sockets are not patched properly, but if I include the patch socket and patch ssl then no web requests come through at all. This might just be a problem with the mac sockets and not on windows, i don't have my Win dev enviroment set up for this project so can't comment if it is happening on both OS.


At this point I am not seeing the advantage of going this way, and the cross thread channels might be the best way to handle things.

Any suggestions I should try before i drop this branch and move on?

- rob


On Thu, Jan 2, 2014 at 8:38 PM, Robert Babiak <rbabiak at gmail.com<mailto:rbabiak at gmail.com>> wrote:
Thanks.

FYI I am a ex CCPer. :)

On 1/2/14, Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>> wrote:
> Monkey-patching is simply replacing the low level operations (like the
> socket module and the select.select function and the time.sleep
> function and so forth).  These normally block the whole thread (and
> therefore scheduler) until the operation is complete.
>
> So you write some code to import these modules and replace attributes
> and objects with one's that just block the calling tasklet.  The
> tasklet is either awakened by some non-blocking method or callback or
> simply farming the work off to another thread transparently, when the
> operation completes.
>
> Only you don't need to do this monkey patching yourself, because
> Kristjan has released and updates the source code to a small library
> that CCP uses in some projects - stacklesslib.
>
> You can find it on bitbucket:
>
> https://bitbucket.org/stackless-dev/stacklesslib
>
> Most third party code will just work if you do the monkeypatching
> first, then import the django or whatever, and then just go about your
> business.
>
> There are some cases where the code won't work, and you need to muck
> around a little to get it to work.  Graphical frameworks, perhaps like
> tkinter, pyqt, pygame and so forth often have main loops.  In those
> cases they need to run a call to the stackless scheduler, or have the
> scheduler do a secondary poll call of some sort, rather than running
> the main loop.
>
> I'm sure someone has already covered how to get django running with
> stackless somewhere.  I think CCP did it at some point.
>
> There's also a wikipedia page on monkeypatching if you are not
> familiar with the general concept:
>
> http://en.wikipedia.org/wiki/Monkey_patch
>
> Cheers,
> Richard.
>
> On 1/3/14, Robert Babiak <rbabiak at gmail.com<mailto:rbabiak at gmail.com>> wrote:
>> Thanks Richard.
>>
>> but what do you mean by "monkeypatch it via stacklesslib" how/ where can
>> i
>> learn about MonkeyPatching?
>>
>> Then I would assume I would just run it as any other tasklet, which I
>> would
>> suspect would solve several problems i have with passing events, and SQL
>> calls on the wrong thread.
>>
>> - Rob
>>
>> ---------- Forwarded message ----------
>> From: Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>>
>> To: The Stackless Python Mailing List <stackless at stackless.com<mailto:stackless at stackless.com>>
>> Cc:
>> Date: Thu, 2 Jan 2014 11:58:39 +1300
>> Subject: Re: [Stackless] multiprocessing module and stackless threads
>> If your web server uses non-blocking IO, then it shouldn't be
>> necessary to have two schedulers.  I imagine with Django, you would
>> monkeypatch it via stacklesslib and it would then work in a
>> stackless-compatible non-blocking fashion.
>>
>> Stackless supports per-thread schedulers, and explicitly supports
>> interthread channel communication.  There is no explicit Stackless
>> support for multiprocessing.  If it opaquely uses threads, then it
>> works in the manner mentioning threads above.  If interthread channel
>> usage is not being used, then channels switch between tasklets on the
>> same Python thread.
>>
>> Cheers,
>> Richard.
>>
>> On 1/2/14, Robert Babiak <rbabiak at gmail.com<mailto:rbabiak at gmail.com>> wrote:
>>> I have a question.
>>>
>>> Is it possible to use the multi processing module and stackless
>>> together?
>>>
>>> What i would like to do is have my application start a second process to
>>> handle the web server side of things, and the main thread control the
>>> main
>>> display.
>>>
>>> Can the two communicate via channels, or is the separate process
>> completely
>>> independant?
>>>
>>> If I don't use the multiprocessing and just use the threading module to
>>> make a dedicated web server thread. Would it be better to run two
>> stackless
>>> schedulers and have the one only run a single tasklet running the web
>>> server request loop?
>>>
>>> FYI: I am using DJANGO for the web server support, and the main display
>>> is
>>> a Open GL display. The main interaction to the display will be by web
>>> requests, but only some web calls will interact with the GL display.
>>>
>>> Suggestions and wisdom welcome.
>>>
>>> - robert
>>> --
>>> Life: Bah, I will worry about it when it is over.
>>>
>>
>> --
>> Life: Bah, I will worry about it when it is over.
>>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com<mailto:Stackless at stackless.com>
> http://www.stackless.com/mailman/listinfo/stackless
>


--
Life: Bah, I will worry about it when it is over.



--
Life: Bah, I will worry about it when it is over.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20140103/533bcf35/attachment-0001.html>


More information about the Stackless mailing list