[Stackless] monkeypatching

lars van Gemerden lars at rational-it.com
Thu Dec 5 22:07:20 CET 2013


Sure, no problem, i'll try to keep it short;

I have a (stackless) python design gui application built with Qt/PySide.
With this a designer can graphically (drag & drop, blocks & arrows) design:

 - a datamodel (sqlalchemy, currently sqlite),
 - process flow, with steps that can basically execute any python functon,
but e.g. also generate dynamic web page elements (push & pull, execution
using tasklets),
 - end user web interfaces (implemented in bottle/html/ajax, python simple
server for now) and
 - role model(determines what users get access to which web interfaces).

And with one click (for demo purposes) deploy the whole shebang locally as
a web application server.

Some process flow steps generate web interfaces for the end users. The the
application server part that executes the process flow, uses tasklets to be
able to run process steps asynchronously (e.g. not stopping while waiting
for user input)

What i would like to do is start and be able to stop the web application,
including the web server from the designer gui and have the process
execution part of the server not block while waiting on http requests.

previously i deep copied the combined model (data, process, role/web gui)
and started the app server server based on that model in a separate thread
(oh, i created a tasklet for the bottle microframework run() method, which
includes the mainn server loop()) ; I only called schedule() on each web
server request to be able to do anything outside the server mainloop.
However that had the effect that process steps where not executed as long
as the end user sent no requests (some processes do not have any web
component).

After monkeypatching, the server did what it was supposed to (run both
internal processes as well as processes with a web component), however
because the server thread was now a tasklet, the designer gui did not
respond anymore.

After using thread.real_thread.start_new_thread to fix this, the server
seemed to run until the first http request, after which the server app got
stuck in Trigger.run (see last mail).

What i have also tried is to put stackless,schedule() in the main webserver
loop. That seemed to work pretty well, apart from the occasional crash :-(;
any ideas what that might be?

Hope this helps, it is a somewhat complicated project (but most of it runs
pretty smoothly by now).

Cheers, Lars


On Wed, Dec 4, 2013 at 11:49 AM, Kristján Valur Jónsson <
kristjan at ccpgames.com> wrote:

>  I’m sorry Lars,
>
> but I have (sort-of) lost track of what it is that you are trying to do J
>
> Could you try to summarize what it is you are trying to achieve? What are
> the components of your application?
>
>
>
> K
>
>
>
> *From:* stackless-bounces at stackless.com [mailto:
> stackless-bounces at stackless.com] *On Behalf Of *lars van Gemerden
> *Sent:* 3. desember 2013 16:22
>
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] monkeypatching
>
>
>
> I have tried the real_thread approach as below, but it seems to get stuck
> in a loop that calls stackless.schedule().  Isn't the webserver getting
> scheduled (anymore, it runs at the start) ?
>
>
>
> ----------------------------------------------------------------
>
>
>
> Where the program gets stuck (if i don't use this class, all runs well,
> although pretty slow compared to without the real thread):
>
>
>
>  class Trigger(BaseTrigger):                      #part of flowmodel, see
> below
>
>
>
>     def start(self):                                       #called from
> flowmodel.start() see MonkeyTasklets.run below
>
>         super(Trigger, self).start()
>
>         stackless.tasklet(self.run)()
>
>
>
>     def condition(self):
>
>         raise NotImplementedError()
>
>
>
>     def run(self):
>
>         while True:                                  #gets stuck here
>
>             if self.condition():                    #returns False
>
>                 self.trigger()
>
>             else:
>
>                 stackless.schedule()
>
>
>
> where i create the real thread:
>
>
>
>      def run_server(self, config = None, threaded = False):
>   # in this case threaded = True
>
>         from flow.tasklets import MonkeyTasklets
>
>         clone = self.create_run_clone(config)                  #deepcopies
> all models in which tasklets are created (except webserver tasklet)
>
>         clone.tasklets = MonkeyTasklets()                       #see below
>
>         if threaded:
>
>             import thread
>
>             def run_function(world):
>
>                 clone.tasklets.run(world = world,
>
>                                              webserver = webserver,
>
>                                              maxcount =
> world.config.RUN_LIMIT)
>
>             thread.real_thread.start_new_thread(run_function, (clone,))
>                #create real thread
>
>         else:
>
>             clone.tasklets.run(clone, webserver, maxcount =
> clone.config.RUN_LIMIT)
>
>         return clone
>                                                 #clone is returned to be
> able to stop the thread via clone.tasklets.stop = True
>
>
>
> This is where the main loop runs:
>
>
>
>  class MonkeyTasklets(object):
>
>
>
>  stacklesslib.app.install_stackless()
>
>
>
>     def __init__(self):
>
>         self.stop = None                              #used to stop the
> loop and therefore the thread
>
>
>
>     def run(self, world, webserver = None, maxcount = None):
>
>         self.stop = False
>
>         world.flowmodel.start()
>
>         if webserver:
>
>             webserver.start_server(world_model = world)
>      #creates a tasklet for the webserver
>
>         counter = 0
>
>         while not self.stop or counter > maxcount:
>
>             counter += 1
>
>             try:
>
>                 stacklesslib.main.mainloop.loop()
>
>             except Exception as e:
>
>                 import asyncore
>
>                 if isinstance(e, ReferenceError):
>
>                     print("run:EXCEPTION", str(e), asyncore.socket_map)
>
>                 else:
>
>                     print("run:EXCEPTION", asyncore.socket_map)
>
>                     traceback.print_exc()
>
>                 sys.exc_clear()
>
>         if webserver:
>
>                 webserver.stop()
>
>         world.flowmodel.stop()
>
>
>
> monkeypatching takes place at the start of the program.
>
>
>
> I would like to get this running (because i am into it right now), but it
> isn't crucial.
>
>
>
> Cheers, Lars
>
>
>
>
>
>
>
> On Mon, Dec 2, 2013 at 9:52 AM, Kristján Valur Jónsson <
> kristjan at ccpgames.com> wrote:
>
> Well, that’s the thing, the way we “currently” do monkeypatching, it is an
> all-or-nothing approach.
>
> Either you replace your entire application’s “threading” usage with fake
> threads from stackless, or you don’t.
>
> Although, as a convention, I have put in the old modules as _*real*_*
> attributes inside the fake ones.  This helps, if your code is aware of this
> trick.
>
>
>
> A few years ago, Jack Diedrich had a PyCon talk about using context
> managers to temporarily monkeypatch code.
>
> This would be useful, e.g. if you wanted to start up part of your
> application in such a way.
>
> Then you could do something like:
>
>
>
> with stackless.monkeypatch.patch_all():
>
>   import flask
>
>   import mywebserver
>
>
>
> import myGUI # using regular threads.
>
>
>
> I’m sure we could re-engineer stacklesslib.monkeypatch to make use of this
> pattern.
>
>
>
> K
>
>
>
> *From:* stackless-bounces at stackless.com [mailto:
> stackless-bounces at stackless.com] *On Behalf Of *lars van Gemerden
> *Sent:* 2. desember 2013 01:26
>
>
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] monkeypatching
>
>
>
> thank you for the help,
>
>
>
> With the fix it works now; i have one problem left, is that when i start
> the server from the gui, the server runs, but the (Qt/PySide) gui becomes
> unresponsive. Before, i started the web app and server in a separate thread
> (and had the scheduler run in that), but that probably  wont work if
> threads are replaced with tasklets (if i understood the monkeypatching
> somewhat).
>
>
>
> Does anyone have any ideas how to solve that (to keep the gui responsive,
> also to be able to stop the server)?  Before i tried multiprocessing, but
> that led to a lot of servers running :-)
>
>
>
> In another experiment I have overridden the server loop serve_forever() to
> call stackless.schedule() (or rather tasklets.schedule() in the code
> above). This seems to work apart from an occasional crash (weird attribute
> error for (my subclass of) WSGIServer).
>
>
>
> Also thanks a lot for putting up the 2.7.5 64 bit binaries!! It finally
> let me run the demo on my Windows 8 laptop (no idea why, something to do
> with PySide 32bit and win8 probably)
>
>
>
> Cheers, Lars
>
>
>
>
>
> On Mon, Dec 2, 2013 at 12:36 AM, Kristján Valur Jónsson <
> kristjan at ccpgames.com> wrote:
>
> A bug!
>
> in threadpool.py, replace the final line with:
>
>
>
> return tasklet_call(wrapped, dispatcher=dispatcher, timeout=timeout,
> onOrphaned=onOrphaned)
>
> Needs more unittests :)
>
>
>
> K
>   ------------------------------
>
> *From:* stackless-bounces at stackless.com [stackless-bounces at stackless.com]
> on behalf of lars van Gemerden [lars at rational-it.com]
> *Sent:* Sunday, December 01, 2013 1:58 PM
> *To:* The Stackless Python Mailing List
> *Subject:* Re: [Stackless] monkeypatching
>
> OK, thanks for the input,
>
>
>
> i am trying to get this to work, but i might have found a bug (which i
> find hard to debug):
>
>
>
> i have reimplemented the code above as:
>
> --------------------------------------------------------------
>
> import asyncore, traceback, sys, logging
>
>
>
> import stacklesslib.main
>
> import stacklesslib.app
>
>
>
> # the unittests use time.time() for various time tests.
>
> # Therefore, we must make sure that main uses this
>
> from time import time as elapsed_time
>
> stacklesslib.main.elapsed_time = elapsed_time
>
>
>
> class MonkeyTasklets(object):
>
>
>
>     _stop = False
>
>
>
>     stacklesslib.app.install_stackless()
>
>
>
>     @classmethod
>
>     def run(cls, world, webserver = None, maxcount = None): #param killer
> is ignored
>
>         world.flowmodel.start()
>
>         if webserver:
>
>             cls.server = webserver.start_server(world_model = world)
> #starts a tasklet for running the webserver
>
>         counter = 0
>
>         while not cls._stop or counter > maxcount:
>
>             counter += 1
>
>             tick_time = elapsed_time()
>
>             try:
>
>                 stacklesslib.main.mainloop.loop()
>
>             except Exception as e:
>
>                 import asyncore
>
>                 if isinstance(e, ReferenceError):
>
>                     print("run:EXCEPTION", str(e), asyncore.socket_map)
>
>                 else:
>
>                     print("run:EXCEPTION", asyncore.socket_map)
>
>                     traceback.print_exc()
>
>                 sys.exc_clear()
>
>         world.flowmodel.stop()
>
>
>
>     @classmethod
>
>     def schedule(cls):
>
>         stackless.schedule()
>
>
>
>     @classmethod
>
>     def tasklet(cls, func):
>
>         return stackless.tasklet(func)
>
>
>
>     @classmethod
>
>     def stop(cls):
>
>         if hasattr(cls, "server"):
>
>             cls.server.stop()
>
>         cls._stop = True
>
>
>
>
>
> tasklets = MonkeyTasklets
>
>
>
> --------------------------------------------------------------
>
>  and i get the exception:
>
>
>
> File *"d:\Documents\Code\python\floware\server\bottle.py"*, line 641, in
> *run*
>   *run(self, **kwargs)*
> File *"d:\Documents\Code\python\floware\server\bottle.py"*, line 2720, in
> *run*
>   *server.run(app)*
> File *"d:\Documents\Code\python\floware\server\webserver.py"*, line 329,
> in *run*
>   *self.server.serve_forever()*
> File *"C:\Python27\lib\SocketServer.py"*, line 236, in *serve_forever*
>   *poll_interval)*
> File *"C:\Python27\lib\SocketServer.py"*, line 155, in *_eintr_retry*
>   *return func(*args)*
> File
> *"d:\Documents\Code\python\floware\stacklesslib\replacements\select.py"*,
> line 27, in *select*
>   *return stacklesslib.threadpool.call_on_thread(real_select.select,
> args, kwargs)*
> File *"d:\Documents\Code\python\floware\stacklesslib\threadpool.py"*,
> line 123, in *call_on_thread*
>   *return tasklet_call(wrapped, dispatcher, timeout=timeout,
> onOrphaned=onOrphaned)*
> File *"d:\Documents\Code\python\floware\stacklesslib\util.py"*, line 209,
> in *tasklet_call*
>   *return channel_wait(chan, timeout)*
> File *"d:\Documents\Code\python\floware\stacklesslib\util.py"*, line 53,
> in *channel_wait*
>   *return chan.receive()*
> File *"d:\Documents\Code\python\floware\stacklesslib\util.py"*, line 192,
> in *helper*
>   *result = function(*args, **kwargs)*
>
> *TypeError: wrapped() argument after * must be a sequence, not function*
>
> --------------------------------------------------------------
>
> is this something i am doing wrong or indeed a buggy in stacklesslib and
> if so, how can it be fixed?
>
>
>
> Cheers, Lars
>
>
>
> On Sun, Dec 1, 2013 at 4:05 AM, Richard Tew <richard.m.tew at gmail.com>
> wrote:
>
> latest stacklesslib, i mean.
>
>
> On 12/1/13, Richard Tew <richard.m.tew at gmail.com> wrote:
> > Hi Lars,
> >
> > stacklessio is internal ccp stuff.
> >
> > main.py in stacklesslib is pretty much all you need to understand,
> > beyond calling patch_all().  It is a scheduler, and makes your Open
> > scheduler redundant.
> >
> > I suggest you get the latest stacklessio from:
> >
> > https://bitbucket.org/krisvale/stacklesslib
> >
> > And then you check out test / teststdlibunittests.py, which is a short
> > example of monkey patching and running the stacklesslib scheduler.
> >
> > Cheers,
> > Richard.
> >
> > On 12/1/13, lars van Gemerden <lars at rational-it.com> wrote:
> >> sorry,
> >>
> >> patch_all seemed to simple, question was a bit rethoric ;-) tried it 2
> >> mins
> >> after pressing send just in case and indeed no cigar.
> >>
> >> I am trying to get this demo running and all kinds of last minute ####
> is
> >> popping up.
> >>
> >> I tried reading the source but it doesn't click; i am not familiar with
> >> using sockets etc. at all.
> >>
> >> e.g.
> >>
> >> - do i need 'stacklessio'?
> >> - can i (essentially) just put a stackless.schedule() (or
> >> OpenTasklets.schedule() in the code below) in the serve_forever loop?
> >> - will this work at all with an adapted scheduler like this:
> >>
> >> class OpenTasklets(object):
> >>
> >>     _stop = False
> >>
> >>     @classmethod
> >>     def run(cls, maxcount = None):
> >>         cls.schedule_channel = stackless.channel()
> >>         cls.schedule_channel.preference = 1
> >>         counter = 0
> >>         OpenTasklets._stop = False
> >>         while stackless.getruncount() != 1:
> >>             stackless.run()
> >>             cls.reschedule(cls._stop or (maxcount and counter >
> >> maxcount))
> >>             counter += 1
> >>
> >>     @classmethod
> >>     def schedule(cls):
> >>         if cls.schedule_channel.receive():
> >>             raise TaskletExit
> >>
> >>     @classmethod
> >>     def reschedule(cls, stop = False):
> >>         while cls.schedule_channel.balance < 0:
> >>             cls.schedule_channel.send(stop)
> >>
> >> Any help is appreciated ..
> >>
> >> Lars
> >>
> >>
> >>
> >>
> >> On Sat, Nov 30, 2013 at 8:53 PM, Richard Tew
> >> <richard.m.tew at gmail.com>wrote:
> >>
> >>> Lars :-)
> >>>
> >>> Yes, there is no documentation for stacklesslib yet.  To use it, you
> >>> really need to be willing to read the source code.
> >>>
> >>> You should already know whether calling patch_all would be enough.  It
> >>> would have been something you could have tried immediately, rather
> >>> than asking the list and waiting.
> >>>
> >>> Cheers,
> >>> Richard.
> >>>
> >>> On 11/30/13, lars van Gemerden <lars at rational-it.com> wrote:
> >>> > Hi all,
> >>> >
> >>> > I though i could avoid it to run my demo but it seems i need to let
> >>> > the
> >>> > webserver (simple server?, single thread, 'bottle' micro framework)
> >>> > yield
> >>> > to the scheduler i am using. I have downloaded stacklesslib 1.0.3,
> but
> >>> > can't find any documentation to help with monkeypatching.
> >>> >
> >>> > - Do i just run patch_all at the start of the program?
> >>> >
> >>> > Any help is very welcome ..
> >>> >
> >>> > Cheers, Lars
> >>> >
> >>> > --
> >>> > ====================================
> >>> > Lars van Gemerden
> >>> > lars at rational-it.com
> >>> > +31 6 26 88 55 39
> >>> > ====================================
> >>> >
> >>>
> >>> _______________________________________________
> >>> Stackless mailing list
> >>> Stackless at stackless.com
> >>> http://www.stackless.com/mailman/listinfo/stackless
> >>>
> >>
> >>
> >>
> >> --
> >> ====================================
> >> Lars van Gemerden
> >> lars at rational-it.com
> >> +31 6 26 88 55 39
> >> ====================================
> >>
> >
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>
>
>
>
>
> --
> ====================================
> Lars van Gemerden
> lars at rational-it.com
>
> +31 6 26 88 55 39
> ====================================
> <#142bd3b0dfe237d2_142b282cef7bdad7_142b0afb3d212348_>
>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless<#142bd3b0dfe237d2_142b282cef7bdad7_142b0afb3d212348_>
>
>
>
>  <#142bd3b0dfe237d2_142b282cef7bdad7_142b0afb3d212348_>
>
>   <#142bd3b0dfe237d2_142b282cef7bdad7_142b0afb3d212348_>
>
> --
> ====================================
> Lars van Gemerden
> lars at rational-it.com
> +31 6 26 88 55 39
> ====================================
> <#142bd3b0dfe237d2_142b282cef7bdad7_142b0afb3d212348_>
>
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>
>
>
>
>
> --
> ====================================
> Lars van Gemerden
> lars at rational-it.com
> +31 6 26 88 55 39
> ====================================
>
> _______________________________________________
> Stackless mailing list
> Stackless at stackless.com
> http://www.stackless.com/mailman/listinfo/stackless
>



-- 
====================================
Lars van Gemerden
lars at rational-it.com
+31 6 26 88 55 39
====================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20131205/ff9faaa1/attachment-0001.html>


More information about the Stackless mailing list