[Stackless] monkeypatching

lars van Gemerden lars at rational-it.com
Tue Dec 3 17:22:09 CET 2013


I have tried the real_thread approach as below, but it seems to get stuck
in a loop that calls stackless.schedule().  Isn't the webserver getting
scheduled (anymore, it runs at the start) ?

----------------------------------------------------------------

Where the program gets stuck (if i don't use this class, all runs well,
although pretty slow compared to without the real thread):

class Trigger(BaseTrigger):                      #part of flowmodel, see
below

    def start(self):                                       #called from
flowmodel.start() see MonkeyTasklets.run below
        super(Trigger, self).start()
        stackless.tasklet(self.run)()

    def condition(self):
        raise NotImplementedError()

    def run(self):
        while True:                                  #gets stuck here
            if self.condition():                    #returns False
                self.trigger()
            else:
                stackless.schedule()


where i create the real thread:

    def run_server(self, config = None, threaded = False):
# in this case threaded = True
        from flow.tasklets import MonkeyTasklets
        clone = self.create_run_clone(config)                  #deepcopies
all models in which tasklets are created (except webserver tasklet)
        clone.tasklets = MonkeyTasklets()                       #see below
        if threaded:
            import thread
            def run_function(world):
                clone.tasklets.run(world = world,
                                             webserver = webserver,
                                             maxcount =
world.config.RUN_LIMIT)
            thread.real_thread.start_new_thread(run_function, (clone,))
             #create real thread
        else:
            clone.tasklets.run(clone, webserver, maxcount =
clone.config.RUN_LIMIT)
        return clone
                                                #clone is returned to be
able to stop the thread via clone.tasklets.stop = True


This is where the main loop runs:

class MonkeyTasklets(object):

 stacklesslib.app.install_stackless()

    def __init__(self):
        self.stop = None                              #used to stop the
loop and therefore the thread

    def run(self, world, webserver = None, maxcount = None):
        self.stop = False
        world.flowmodel.start()
        if webserver:
            webserver.start_server(world_model = world)
   #creates a tasklet for the webserver
        counter = 0
        while not self.stop or counter > maxcount:
            counter += 1
            try:
                stacklesslib.main.mainloop.loop()
            except Exception as e:
                import asyncore
                if isinstance(e, ReferenceError):
                    print("run:EXCEPTION", str(e), asyncore.socket_map)
                else:
                    print("run:EXCEPTION", asyncore.socket_map)
                    traceback.print_exc()
                sys.exc_clear()
        if webserver:
                webserver.stop()
        world.flowmodel.stop()


monkeypatching takes place at the start of the program.

I would like to get this running (because i am into it right now), but it
isn't crucial.

Cheers, Lars




On Mon, Dec 2, 2013 at 9:52 AM, Kristján Valur Jónsson <
kristjan at ccpgames.com> wrote:

>  Well, that’s the thing, the way we “currently” do monkeypatching, it is
> an all-or-nothing approach.
>
> Either you replace your entire application’s “threading” usage with fake
> threads from stackless, or you don’t.
>
> Although, as a convention, I have put in the old modules as _*real*_*
> attributes inside the fake ones.  This helps, if your code is aware of this
> trick.
>
>
>
> A few years ago, Jack Diedrich had a PyCon talk about using context
> managers to temporarily monkeypatch code.
>
> This would be useful, e.g. if you wanted to start up part of your
> application in such a way.
>
> Then you could do something like:
>
>
>
> with stackless.monkeypatch.patch_all():
>
>   import flask
>
>   import mywebserver
>
>
>
> import myGUI # using regular threads.
>
>
>
> I’m sure we could re-engineer stacklesslib.monkeypatch to make use of this
> pattern.
>
>
>
> K
>
>
>
> *From:* stackless-bounces at stackless.com [mailto:
> stackless-bounces at stackless.com] *On Behalf Of *lars van Gemerden
> *Sent:* 2. desember 2013 01:26
>
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] monkeypatching
>
>
>
> thank you for the help,
>
>
>
> With the fix it works now; i have one problem left, is that when i start
> the server from the gui, the server runs, but the (Qt/PySide) gui becomes
> unresponsive. Before, i started the web app and server in a separate thread
> (and had the scheduler run in that), but that probably  wont work if
> threads are replaced with tasklets (if i understood the monkeypatching
> somewhat).
>
>
>
> Does anyone have any ideas how to solve that (to keep the gui responsive,
> also to be able to stop the server)?  Before i tried multiprocessing, but
> that led to a lot of servers running :-)
>
>
>
> In another experiment I have overridden the server loop serve_forever() to
> call stackless.schedule() (or rather tasklets.schedule() in the code
> above). This seems to work apart from an occasional crash (weird attribute
> error for (my subclass of) WSGIServer).
>
>
>
> Also thanks a lot for putting up the 2.7.5 64 bit binaries!! It finally
> let me run the demo on my Windows 8 laptop (no idea why, something to do
> with PySide 32bit and win8 probably)
>
>
>
> Cheers, Lars
>
>
>
>
>
> On Mon, Dec 2, 2013 at 12:36 AM, Kristján Valur Jónsson <
> kristjan at ccpgames.com> wrote:
>
> A bug!
>
> in threadpool.py, replace the final line with:
>
>
>
> return tasklet_call(wrapped, dispatcher=dispatcher, timeout=timeout,
> onOrphaned=onOrphaned)
>
> Needs more unittests :)
>
>
>
> K
>   ------------------------------
>
> *From:* stackless-bounces at stackless.com [stackless-bounces at stackless.com]
> on behalf of lars van Gemerden [lars at rational-it.com]
> *Sent:* Sunday, December 01, 2013 1:58 PM
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] monkeypatching
>
> OK, thanks for the input,
>
>
>
> i am trying to get this to work, but i might have found a bug (which i
> find hard to debug):
>
>
>
> i have reimplemented the code above as:
>
> --------------------------------------------------------------
>
> import asyncore, traceback, sys, logging
>
>
>
> import stacklesslib.main
>
> import stacklesslib.app
>
>
>
> # the unittests use time.time() for various time tests.
>
> # Therefore, we must make sure that main uses this
>
> from time import time as elapsed_time
>
> stacklesslib.main.elapsed_time = elapsed_time
>
>
>
> class MonkeyTasklets(object):
>
>
>
>     _stop = False
>
>
>
>     stacklesslib.app.install_stackless()
>
>
>
>     @classmethod
>
>     def run(cls, world, webserver = None, maxcount = None): #param killer
> is ignored
>
>         world.flowmodel.start()
>
>         if webserver:
>
>             cls.server = webserver.start_server(world_model = world)
> #starts a tasklet for running the webserver
>
>         counter = 0
>
>         while not cls._stop or counter > maxcount:
>
>             counter += 1
>
>             tick_time = elapsed_time()
>
>             try:
>
>                 stacklesslib.main.mainloop.loop()
>
>             except Exception as e:
>
>                 import asyncore
>
>                 if isinstance(e, ReferenceError):
>
>                     print("run:EXCEPTION", str(e), asyncore.socket_map)
>
>                 else:
>
>                     print("run:EXCEPTION", asyncore.socket_map)
>
>                     traceback.print_exc()
>
>                 sys.exc_clear()
>
>         world.flowmodel.stop()
>
>
>
>     @classmethod
>
>     def schedule(cls):
>
>         stackless.schedule()
>
>
>
>     @classmethod
>
>     def tasklet(cls, func):
>
>         return stackless.tasklet(func)
>
>
>
>     @classmethod
>
>     def stop(cls):
>
>         if hasattr(cls, "server"):
>
>             cls.server.stop()
>
>         cls._stop = True
>
>
>
>
>
> tasklets = MonkeyTasklets
>
>
>
> --------------------------------------------------------------
>
>  and i get the exception:
>
>
>
> File *"d:\Documents\Code\python\floware\server\bottle.py"*, line 641, in
> *run*
>   *run(self, **kwargs)*
> File *"d:\Documents\Code\python\floware\server\bottle.py"*, line 2720, in
> *run*
>   *server.run(app)*
> File *"d:\Documents\Code\python\floware\server\webserver.py"*, line 329,
> in *run*
>   *self.server.serve_forever()*
> File *"C:\Python27\lib\SocketServer.py"*, line 236, in *serve_forever*
>   *poll_interval)*
> File *"C:\Python27\lib\SocketServer.py"*, line 155, in *_eintr_retry*
>   *return func(*args)*
> File
> *"d:\Documents\Code\python\floware\stacklesslib\replacements\select.py"*,
> line 27, in *select*
>   *return stacklesslib.threadpool.call_on_thread(real_select.select,
> args, kwargs)*
> File *"d:\Documents\Code\python\floware\stacklesslib\threadpool.py"*,
> line 123, in *call_on_thread*
>   *return tasklet_call(wrapped, dispatcher, timeout=timeout,
> onOrphaned=onOrphaned)*
> File *"d:\Documents\Code\python\floware\stacklesslib\util.py"*, line 209,
> in *tasklet_call*
>   *return channel_wait(chan, timeout)*
> File *"d:\Documents\Code\python\floware\stacklesslib\util.py"*, line 53,
> in *channel_wait*
>   *return chan.receive()*
> File *"d:\Documents\Code\python\floware\stacklesslib\util.py"*, line 192,
> in *helper*
>   *result = function(*args, **kwargs)*
>
> *TypeError: wrapped() argument after * must be a sequence, not function*
>
> --------------------------------------------------------------
>
> is this something i am doing wrong or indeed a buggy in stacklesslib and
> if so, how can it be fixed?
>
>
>
> Cheers, Lars
>
>
>
> On Sun, Dec 1, 2013 at 4:05 AM, Richard Tew <richard.m.tew at gmail.com>
> wrote:
>
> latest stacklesslib, i mean.
>
>
> On 12/1/13, Richard Tew <richard.m.tew at gmail.com> wrote:
> > Hi Lars,
> >
> > stacklessio is internal ccp stuff.
> >
> > main.py in stacklesslib is pretty much all you need to understand,
> > beyond calling patch_all().  It is a scheduler, and makes your Open
> > scheduler redundant.
> >
> > I suggest you get the latest stacklessio from:
> >
> > https://bitbucket.org/krisvale/stacklesslib
> >
> > And then you check out test / teststdlibunittests.py, which is a short
> > example of monkey patching and running the stacklesslib scheduler.
> >
> > Cheers,
> > Richard.
> >
> > On 12/1/13, lars van Gemerden <lars at rational-it.com> wrote:
> >> sorry,
> >>
> >> patch_all seemed to simple, question was a bit rethoric ;-) tried it 2
> >> mins
> >> after pressing send just in case and indeed no cigar.
> >>
> >> I am trying to get this demo running and all kinds of last minute ####
> is
> >> popping up.
> >>
> >> I tried reading the source but it doesn't click; i am not familiar with
> >> using sockets etc. at all.
> >>
> >> e.g.
> >>
> >> - do i need 'stacklessio'?
> >> - can i (essentially) just put a stackless.schedule() (or
> >> OpenTasklets.schedule() in the code below) in the serve_forever loop?
> >> - will this work at all with an adapted scheduler like this:
> >>
> >> class OpenTasklets(object):
> >>
> >>     _stop = False
> >>
> >>     @classmethod
> >>     def run(cls, maxcount = None):
> >>         cls.schedule_channel = stackless.channel()
> >>         cls.schedule_channel.preference = 1
> >>         counter = 0
> >>         OpenTasklets._stop = False
> >>         while stackless.getruncount() != 1:
> >>             stackless.run()
> >>             cls.reschedule(cls._stop or (maxcount and counter >
> >> maxcount))
> >>             counter += 1
> >>
> >>     @classmethod
> >>     def schedule(cls):
> >>         if cls.schedule_channel.receive():
> >>             raise TaskletExit
> >>
> >>     @classmethod
> >>     def reschedule(cls, stop = False):
> >>         while cls.schedule_channel.balance < 0:
> >>             cls.schedule_channel.send(stop)
> >>
> >> Any help is appreciated ..
> >>
> >> Lars
> >>
> >>
> >>
> >>
> >> On Sat, Nov 30, 2013 at 8:53 PM, Richard Tew
> >> <richard.m.tew at gmail.com>wrote:
> >>
> >>> Lars :-)
> >>>
> >>> Yes, there is no documentation for stacklesslib yet.  To use it, you
> >>> really need to be willing to read the source code.
> >>>
> >>> You should already know whether calling patch_all would be enough.  It
> >>> would have been something you could have tried immediately, rather
> >>> than asking the list and waiting.
> >>>
> >>> Cheers,
> >>> Richard.
> >>>
> >>> On 11/30/13, lars van Gemerden <lars at rational-it.com> wrote:
> >>> > Hi all,
> >>> >
> >>> > I though i could avoid it to run my demo but it seems i need to let
> >>> > the
> >>> > webserver (simple server?, single thread, 'bottle' micro framework)
> >>> > yield
> >>> > to the scheduler i am using. I have downloaded stacklesslib 1.0.3,
> but
> >>> > can't find any documentation to help with monkeypatching.
> >>> >
> >>> > - Do i just run patch_all at the start of the program?
> >>> >
> >>> > Any help is very welcome ..
> >>> >
> >>> > Cheers, Lars
> >>> >
> >>> > --
> >>> > ====================================
> >>> > Lars van Gemerden
> >>> > lars at rational-it.com
> >>> > +31 6 26 88 55 39
> >>> > ====================================
> >>> >
> >>>
> >>> _______________________________________________
> >>> Stackless mailing list
> >>> Stackless at stackless.com
> >>> http://www.stackless.com/mailman/listinfo/stackless
> >>>
> >>
> >>
> >>
> >> --
> >> ====================================
> >> Lars van Gemerden
> >> lars at rational-it.com
> >> +31 6 26 88 55 39
> >> ====================================
> >>
> >
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>
>
>
>
>
> --
> ====================================
> Lars van Gemerden
> lars at rational-it.com
>
> +31 6 26 88 55 39
> ==================================== <#142b282cef7bdad7_142b0afb3d212348_>
>
>
> _______________________________________________
> Stackless mailing list
> *Stackless at stackless.com*
> *http://www.stackless.com/mailman/listinfo/stackless*<#142b282cef7bdad7_142b0afb3d212348_>
>
>
>
>  <#142b282cef7bdad7_142b0afb3d212348_>
>
>   <#142b282cef7bdad7_142b0afb3d212348_>
>
> --
> ====================================
> Lars van Gemerden
> *lars at rational-it.com*
> +31 6 26 88 55 39
> ==================================== <#142b282cef7bdad7_142b0afb3d212348_>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>



-- 
====================================
Lars van Gemerden
lars at rational-it.com
+31 6 26 88 55 39
====================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20131203/a031da0d/attachment-0001.html>


More information about the Stackless mailing list