[Python-Dev] Re: [Stackless] comments on PEP 219
gmcm at hypernet.com
Tue Mar 13 23:16:24 CET 2001
> >> If a programmer uses a library implement via coroutines, can
> she >> call library methods from an __xxx__ method?
> GMcM> Certain situations won't work, but you knew that.
> I expected that some won't work, but no one seems willing to tell
> me exactly which ones will and which ones won't. Should the
> caveat in the documentation say "avoid using certain __xxx__
> methods" <0.9 wink>.
Within an __xxx__ method, you cannot *use* a coroutine not
created in that method. That is true in current Stackless and
will be true in Stack-lite. The presence of "library" in the
question is a distraction.
I guess if you think of a coroutine as just another kind of
callable object, this looks like a strong limitation. But you
don't find yourself thinking of threads as plain old callable
objects, do you? In a threaded program, no matter how
carefully designed, there is a lot of thread detritus lying
around. If you don't stay concious of the transfers of control
that may happen, you will screw up.
Despite the limitation on using coroutines in magic methods,
coroutines have an advantage in that tranfers of control only
happen when you want them to. So avoiding unwanted
transfers of control is vastly easier.
> >> Can coroutines or microthreads co-exist with callbacks
> invoked by >> C extensions?
> GMcM> Again, in certain situations it won't work. Again, you
> knew GMcM> that.
> Wasn't sure.
It's exactly the same situation.
> >> Can a program do any microthread IO in an __call__ method?
> GMcM> Considering you know the answer to that one too, you
> could've GMcM> phrased it as a parsable question.
> Do I know the answer? I assume the answer is no, but I don't
> feel very certain.
What is "microthreaded IO"? Probably the attempt to yield
control if the IO operation would block. Would doing that
inside __call__ work with microthreads? No.
It's not my decision over whether this particular situation
needs to be documented. Somtime between the 2nd and 5th
times the programmer encounters this exception, they'll say
"Oh phooey, I can't do this in __call__, I need an explicit
method instead." Python has never claimed that __xxx__
methods are safe as milk. Quite the contrary.
> >> If any of these are the sort "in theory" problems that the
> PEP >> alludes to, then we need a full spec for what is and is
> not >> allowed. It doesn't make sense to tell programmers to
> follow >> unspecified "reasonable" programming practices.
> GMcM> That's easy. In a nested invocation of the Python
> interpreter, GMcM> you can't use a coroutine created in an
> outer interpreter.
> Can we define these situations in a way that doesn't appeal to
> the interpreter implementation?
No, because it's implementation dependent.
> If not, can we at least come up
> with a list of what will and will not work at the python level?
Does Python attempt to catalogue all the ways you can screw
up using magic methods? Using threads? How 'bout the
metaclass hook? Even stronger, do we catalogue all the ways
that an end-user-programmer can get bit by using a library
written by someone else that makes use of these facilities?
> GMcM> In the Python 2 documentation, there are 6 caveats listed
> in GMcM> the thread module. That's a couple order of magnitudes
> GMcM> different from the actual number of ways you can screw up
> GMcM> using the thread module.
> The caveats for the thread module seem like pretty minor stuff to
> me. If you are writing a threaded application, don't expect code
> to continue running after the main thread has exited.
Well, the thread caveats don't mention the consequences of
starting and running a thread within an __init__ method.
> The caveats for microthreads seems to cover a vast swath of
> territory: The use of libraries or extension modules that involve
> callbacks or instances with __xxx__ methods may lead to
> application failure.
While your statement is true on the face of it, it is very
misleading. Things will only fall apart when you code an
__xxx__ method or callback that uses a pre-existing coroutine
(or does a uthread swap). You can very easily get in trouble
right now with threads and callbacks. But the real point is that
it is *you* the programmer trying to do something that won't
work (and, BTW, getting notified right away), not some library
pulling a fast one on you. (Yes, the library could make things
very hard for you, but that's nothing new.)
Application programmers do not need magic methods. Ever.
They are very handy for people creating libraries for application
programmers to use, but we already presume (naively) that
these people know what they're doing.
> I worry about it becomes it doesn't sound
> very modular. The use of coroutines in one library means I can't
> use that library in certain special cases in my own code.
With a little familiarity, you'll find that coroutines are a good
deal more modular than threads.
In order for that library to violate your expectations, that library
must be concious of multiple coroutines (otherwise, it's just a
plain stackfull call / return). It must have kept a coroutine from
some other call, or had you pass one in. So you (if at all
cluefull <wink>) will be concious that something is going on
The issue is the same as if you used a framework which used
real threads, but never documented anything about the
threads. You code callbacks that naively and independently
mutate a global collection. Do you blame Python?
> I'm sorry if I sound grumpy, but I feel like I can't get a
> straight answer despite several attempts. At some level, it's
> fine to say that there are some corner cases that won't work well
> with microthreads or coroutines implemented on top of stackless
> python. But I think the PEP should discuss the details. I've
> never written in an application that uses stackless-based
> microthreads or coroutines so I don't feel confident in my
> judgement of the situation.
And where on the fearful to confident scale was the Jeremy
just getting introduced to threads?
> Which gets back to Bernd's original question:
> GMcM> > BR> """ Don't use classes and libraries that use
> classes when GMcM> > BR> IO in microthreaded programs! """
> GMcM> > GMcM> > BR> which might indeed be a problem. Am I
> overlooking something GMcM> > BR> fundamental here?
> and the synopsis of your answer:
> GMcM> Synopsis of my reply: this is more a problem with
> uthreads GMcM> than coroutines. In any (real) thread, you're
> limited to dealing GMcM> with one non-blocking IO technique
> (eg, select) without going GMcM> into a busy loop. If you're
> dedicating a (real) thread to select, it GMcM> makes more sense
> to use coroutines than uthreads.
> I don't understand how this addresses the question, but perhaps I
> haven't seen your reply yet. Mail gets through to python-dev and
> stackless at different rates.
Coroutines only swap voluntarily. It's very obvious where these
transfers of control take place hence simple to control when
they take place. My suspicion is that most people use
uthreads because they use a familiar model. Not many people
are used to coroutines, but many situations would be more
profitably approached with coroutines than uthreads.
Stackless mailing list
Stackless at starship.python.net
More information about the Stackless