[Stackless] monkeypatching

Kristján Valur Jónsson kristjan at ccpgames.com
Wed Dec 4 11:49:47 CET 2013


I'm sorry Lars,
but I have (sort-of) lost track of what it is that you are trying to do :)
Could you try to summarize what it is you are trying to achieve? What are the components of your application?

K

From: stackless-bounces at stackless.com [mailto:stackless-bounces at stackless.com] On Behalf Of lars van Gemerden
Sent: 3. desember 2013 16:22
To: The Stackless Python Mailing List
Subject: Re: [Stackless] monkeypatching

I have tried the real_thread approach as below, but it seems to get stuck in a loop that calls stackless.schedule().  Isn't the webserver getting scheduled (anymore, it runs at the start) ?

----------------------------------------------------------------

Where the program gets stuck (if i don't use this class, all runs well, although pretty slow compared to without the real thread):

class Trigger(BaseTrigger):                      #part of flowmodel, see below

    def start(self):                                       #called from flowmodel.start() see MonkeyTasklets.run below
        super(Trigger, self).start()
        stackless.tasklet(self.run)()

    def condition(self):
        raise NotImplementedError()

    def run(self):
        while True:                                  #gets stuck here
            if self.condition():                    #returns False
                self.trigger()
            else:
                stackless.schedule()

where i create the real thread:

    def run_server(self, config = None, threaded = False):                 # in this case threaded = True
        from flow.tasklets import MonkeyTasklets
        clone = self.create_run_clone(config)                  #deepcopies all models in which tasklets are created (except webserver tasklet)
        clone.tasklets = MonkeyTasklets()                       #see below
        if threaded:
            import thread
            def run_function(world):
                clone.tasklets.run(world = world,
                                             webserver = webserver,
                                             maxcount = world.config.RUN_LIMIT)
            thread.real_thread.start_new_thread(run_function, (clone,))                  #create real thread
        else:
            clone.tasklets.run(clone, webserver, maxcount = clone.config.RUN_LIMIT)
        return clone                                                                                                       #clone is returned to be able to stop the thread via clone.tasklets.stop = True

This is where the main loop runs:

class MonkeyTasklets(object):

 stacklesslib.app.install_stackless()

    def __init__(self):
        self.stop = None                              #used to stop the loop and therefore the thread

    def run(self, world, webserver = None, maxcount = None):
        self.stop = False
        world.flowmodel.start()
        if webserver:
            webserver.start_server(world_model = world)                        #creates a tasklet for the webserver
        counter = 0
        while not self.stop or counter > maxcount:
            counter += 1
            try:
                stacklesslib.main.mainloop.loop()
            except Exception as e:
                import asyncore
                if isinstance(e, ReferenceError):
                    print("run:EXCEPTION", str(e), asyncore.socket_map)
                else:
                    print("run:EXCEPTION", asyncore.socket_map)
                    traceback.print_exc()
                sys.exc_clear()
        if webserver:
                webserver.stop()
        world.flowmodel.stop()

monkeypatching takes place at the start of the program.

I would like to get this running (because i am into it right now), but it isn't crucial.

Cheers, Lars



On Mon, Dec 2, 2013 at 9:52 AM, Kristján Valur Jónsson <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:
Well, that's the thing, the way we "currently" do monkeypatching, it is an all-or-nothing approach.
Either you replace your entire application's "threading" usage with fake threads from stackless, or you don't.
Although, as a convention, I have put in the old modules as _real_* attributes inside the fake ones.  This helps, if your code is aware of this trick.

A few years ago, Jack Diedrich had a PyCon talk about using context managers to temporarily monkeypatch code.
This would be useful, e.g. if you wanted to start up part of your application in such a way.
Then you could do something like:

with stackless.monkeypatch.patch_all():
  import flask
  import mywebserver

import myGUI # using regular threads.

I'm sure we could re-engineer stacklesslib.monkeypatch to make use of this pattern.

K

From: stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com> [mailto:stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com>] On Behalf Of lars van Gemerden
Sent: 2. desember 2013 01:26

To: The Stackless Python Mailing List
Subject: Re: [Stackless] monkeypatching

thank you for the help,

With the fix it works now; i have one problem left, is that when i start the server from the gui, the server runs, but the (Qt/PySide) gui becomes unresponsive. Before, i started the web app and server in a separate thread (and had the scheduler run in that), but that probably  wont work if threads are replaced with tasklets (if i understood the monkeypatching somewhat).

Does anyone have any ideas how to solve that (to keep the gui responsive, also to be able to stop the server)?  Before i tried multiprocessing, but that led to a lot of servers running :-)

In another experiment I have overridden the server loop serve_forever() to call stackless.schedule() (or rather tasklets.schedule() in the code above). This seems to work apart from an occasional crash (weird attribute error for (my subclass of) WSGIServer).

Also thanks a lot for putting up the 2.7.5 64 bit binaries!! It finally let me run the demo on my Windows 8 laptop (no idea why, something to do with PySide 32bit and win8 probably)

Cheers, Lars


On Mon, Dec 2, 2013 at 12:36 AM, Kristján Valur Jónsson <kristjan at ccpgames.com<mailto:kristjan at ccpgames.com>> wrote:

A bug!

in threadpool.py, replace the final line with:



return tasklet_call(wrapped, dispatcher=dispatcher, timeout=timeout, onOrphaned=onOrphaned)

Needs more unittests :)



K

________________________________
From: stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com> [stackless-bounces at stackless.com<mailto:stackless-bounces at stackless.com>] on behalf of lars van Gemerden [lars at rational-it.com<mailto:lars at rational-it.com>]
Sent: Sunday, December 01, 2013 1:58 PM
To: The Stackless Python Mailing List
Subject: Re: [Stackless] monkeypatching
OK, thanks for the input,

i am trying to get this to work, but i might have found a bug (which i find hard to debug):

i have reimplemented the code above as:
--------------------------------------------------------------
import asyncore, traceback, sys, logging

import stacklesslib.main
import stacklesslib.app

# the unittests use time.time() for various time tests.
# Therefore, we must make sure that main uses this
from time import time as elapsed_time
stacklesslib.main.elapsed_time = elapsed_time

class MonkeyTasklets(object):

    _stop = False

    stacklesslib.app.install_stackless()

    @classmethod
    def run(cls, world, webserver = None, maxcount = None): #param killer is ignored
        world.flowmodel.start()
        if webserver:
            cls.server = webserver.start_server(world_model = world) #starts a tasklet for running the webserver
        counter = 0
        while not cls._stop or counter > maxcount:
            counter += 1
            tick_time = elapsed_time()
            try:
                stacklesslib.main.mainloop.loop()
            except Exception as e:
                import asyncore
                if isinstance(e, ReferenceError):
                    print("run:EXCEPTION", str(e), asyncore.socket_map)
                else:
                    print("run:EXCEPTION", asyncore.socket_map)
                    traceback.print_exc()
                sys.exc_clear()
        world.flowmodel.stop()

    @classmethod
    def schedule(cls):
        stackless.schedule()

    @classmethod
    def tasklet(cls, func):
        return stackless.tasklet(func)

    @classmethod
    def stop(cls):
        if hasattr(cls, "server"):
            cls.server.stop()
        cls._stop = True


tasklets = MonkeyTasklets

--------------------------------------------------------------
 and i get the exception:

File "d:\Documents\Code\python\floware\server\bottle.py", line 641, in run
  run(self, **kwargs)
File "d:\Documents\Code\python\floware\server\bottle.py", line 2720, in run
  server.run(app)
File "d:\Documents\Code\python\floware\server\webserver.py", line 329, in run
  self.server.serve_forever()
File "C:\Python27\lib\SocketServer.py", line 236, in serve_forever
  poll_interval)
File "C:\Python27\lib\SocketServer.py", line 155, in _eintr_retry
  return func(*args)
File "d:\Documents\Code\python\floware\stacklesslib\replacements\select.py", line 27, in select
  return stacklesslib.threadpool.call_on_thread(real_select.select, args, kwargs)
File "d:\Documents\Code\python\floware\stacklesslib\threadpool.py", line 123, in call_on_thread
  return tasklet_call(wrapped, dispatcher, timeout=timeout, onOrphaned=onOrphaned)
File "d:\Documents\Code\python\floware\stacklesslib\util.py", line 209, in tasklet_call
  return channel_wait(chan, timeout)
File "d:\Documents\Code\python\floware\stacklesslib\util.py", line 53, in channel_wait
  return chan.receive()
File "d:\Documents\Code\python\floware\stacklesslib\util.py", line 192, in helper
  result = function(*args, **kwargs)

TypeError: wrapped() argument after * must be a sequence, not function
--------------------------------------------------------------
is this something i am doing wrong or indeed a buggy in stacklesslib and if so, how can it be fixed?

Cheers, Lars

On Sun, Dec 1, 2013 at 4:05 AM, Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>> wrote:
latest stacklesslib, i mean.

On 12/1/13, Richard Tew <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>> wrote:
> Hi Lars,
>
> stacklessio is internal ccp stuff.
>
> main.py in stacklesslib is pretty much all you need to understand,
> beyond calling patch_all().  It is a scheduler, and makes your Open
> scheduler redundant.
>
> I suggest you get the latest stacklessio from:
>
> https://bitbucket.org/krisvale/stacklesslib
>
> And then you check out test / teststdlibunittests.py, which is a short
> example of monkey patching and running the stacklesslib scheduler.
>
> Cheers,
> Richard.
>
> On 12/1/13, lars van Gemerden <lars at rational-it.com<mailto:lars at rational-it.com>> wrote:
>> sorry,
>>
>> patch_all seemed to simple, question was a bit rethoric ;-) tried it 2
>> mins
>> after pressing send just in case and indeed no cigar.
>>
>> I am trying to get this demo running and all kinds of last minute #### is
>> popping up.
>>
>> I tried reading the source but it doesn't click; i am not familiar with
>> using sockets etc. at all.
>>
>> e.g.
>>
>> - do i need 'stacklessio'?
>> - can i (essentially) just put a stackless.schedule() (or
>> OpenTasklets.schedule() in the code below) in the serve_forever loop?
>> - will this work at all with an adapted scheduler like this:
>>
>> class OpenTasklets(object):
>>
>>     _stop = False
>>
>>     @classmethod
>>     def run(cls, maxcount = None):
>>         cls.schedule_channel = stackless.channel()
>>         cls.schedule_channel.preference = 1
>>         counter = 0
>>         OpenTasklets._stop = False
>>         while stackless.getruncount() != 1:
>>             stackless.run()
>>             cls.reschedule(cls._stop or (maxcount and counter >
>> maxcount))
>>             counter += 1
>>
>>     @classmethod
>>     def schedule(cls):
>>         if cls.schedule_channel.receive():
>>             raise TaskletExit
>>
>>     @classmethod
>>     def reschedule(cls, stop = False):
>>         while cls.schedule_channel.balance < 0:
>>             cls.schedule_channel.send(stop)
>>
>> Any help is appreciated ..
>>
>> Lars
>>
>>
>>
>>
>> On Sat, Nov 30, 2013 at 8:53 PM, Richard Tew
>> <richard.m.tew at gmail.com<mailto:richard.m.tew at gmail.com>>wrote:
>>
>>> Lars :-)
>>>
>>> Yes, there is no documentation for stacklesslib yet.  To use it, you
>>> really need to be willing to read the source code.
>>>
>>> You should already know whether calling patch_all would be enough.  It
>>> would have been something you could have tried immediately, rather
>>> than asking the list and waiting.
>>>
>>> Cheers,
>>> Richard.
>>>
>>> On 11/30/13, lars van Gemerden <lars at rational-it.com<mailto:lars at rational-it.com>> wrote:
>>> > Hi all,
>>> >
>>> > I though i could avoid it to run my demo but it seems i need to let
>>> > the
>>> > webserver (simple server?, single thread, 'bottle' micro framework)
>>> > yield
>>> > to the scheduler i am using. I have downloaded stacklesslib 1.0.3, but
>>> > can't find any documentation to help with monkeypatching.
>>> >
>>> > - Do i just run patch_all at the start of the program?
>>> >
>>> > Any help is very welcome ..
>>> >
>>> > Cheers, Lars
>>> >
>>> > --
>>> > ====================================
>>> > Lars van Gemerden
>>> > lars at rational-it.com<mailto:lars at rational-it.com>
>>> > +31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039>
>>> > ====================================
>>> >
>>>
>>> _______________________________________________
>>> Stackless mailing list
>>> Stackless at stackless.com<mailto:Stackless at stackless.com>
>>> http://www.stackless.com/mailman/listinfo/stackless
>>>
>>
>>
>>
>> --
>> ====================================
>> Lars van Gemerden
>> lars at rational-it.com<mailto:lars at rational-it.com>
>> +31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039>
>> ====================================
>>
>

_______________________________________________
Stackless mailing list
Stackless at stackless.com<mailto:Stackless at stackless.com>
http://www.stackless.com/mailman/listinfo/stackless



--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com>
+31 6 26 88 55 39<tel:%2B31%206%2026%2088%2055%2039>
====================================

_______________________________________________
Stackless mailing list
Stackless at stackless.com
http://www.stackless.com/mailman/listinfo/stackless



--
====================================
Lars van Gemerden
lars at rational-it.com
+31 6 26 88 55 39
====================================

_______________________________________________
Stackless mailing list
Stackless at stackless.com<mailto:Stackless at stackless.com>
http://www.stackless.com/mailman/listinfo/stackless



--
====================================
Lars van Gemerden
lars at rational-it.com<mailto:lars at rational-it.com>
+31 6 26 88 55 39
====================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.stackless.com/pipermail/stackless/attachments/20131204/f3e90ba6/attachment-0001.html>


More information about the Stackless mailing list