[Stackless] Pickling limitations?

[Christian Tismer]

Hi Jørgen,

...

> I was thinking the tasklet had to call schedule() for other tasklets to run.

Yes (without the new auto-scheduler, it is so).

> With a tasklet, then if one enters some C code, and this C code never
> calls schedule, then the erliest moment pickling may occur is when the C
> code returns and schedule is called?

Yes.

> That is it is impossible to pickle a tasklet that runs C code not calling
> PyStackless_Schedule?

Yes.

...

> So any C code that uses PyStackless_Schedule_nr correctly can be pickled and
> restored?

Any C code that used PyStackless_Schedule_nr always, so there are no
nested interpreters on the stack, can be restored.
But not that calling PyStackless_Schedule_nr correctly is unfortunately
very difficult, since you have to do somewhat continuation
passing style.

-------------------------

Normal style:

PyObject * somefunc(...)
{
     some C code...

     call schedule

     more C code...
}
-------------------------

_nr style:

static forward PyObject * post_call(...);

PyObject * somefunc(...)
{
   some C code...

   prepare special base_frame with post_call

   call schedule_nr

   return Py_UnwindToken
}

static PyObject * post_call(...)
{
   more C code...
}


Note that the return Py_UnwindToken *must* return through the
interpreter up to the dispatcher, which then will run the
post_call as a new frame, playing the overall frame game.
And, in order to pickle that, you need to add pickle support
as well.
Will put that in the now really really soon to come WiKi.

> I don't understand how a tasklet can be frozen unless it first calls
> PyStackless_Schedule and then some other task pickles it?

Right, this cannot.

> So how to achieve a pickled tasklet that can't be started again?

Write a PyCFunction that executes some Python code from C code.
Let that python code call schedule(). Then that piece of C code
sitting on the stack above your interpreter will be moved
into a structure. It is re-runnable as long as you don't touch
it. But pickling doesn't work.

Simple example: You can write an object with a special
__getattr__ method in Python, that does a schedule).
Then, if you use an attribute of that object, this will
cause a recursive interpreter call, since you have overwritten
the built-in getattr C method with a Python function.
I have no provision about that, it happens all over the
place and is impossible to change, unless you want to wait
for PyPy.

>>>And since I wonder about that how do you handle calls to code external to
>>>the Python compiler in combination with stackless?

I am hijacking the pice of C stack.

...

> So the tasklets keep everything, including a possible C stack for this
> tasklet, in a Python frame, leaving the C stack free to be modified,
> allowing other Python frames wih C stack to be restored? 

Sure.

> Using stackless to implement some kind of multithreaded server then seems a
> good idea if and only if none of the threads call external blocking code?

Exactly. If you must call an external blocking call, use an extra
real thread for that. Or better, use an extra task, because that
solves the GIL problem from below.

> Unless there was some clever scheme starting those in a new and different
> kernel thread, allowing the other tasklets to use the existing one. And this
> would require a Python that doesn't have GIL as of today, in addition to
> lots of work?

This is correct.

> Have you considered adding asynchronous one-way channels?

What would that be, how would they be different form
my current channels?

ciao - chris

-- 
Christian Tismer             :^)   <mailto:tismer at tismer.com>
Mission Impossible 5oftware  :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34  home +49 30 802 86 56  mobile +49 173 24 18 776
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
      whom do you want to sponsor today?   http://www.stackless.com/


_______________________________________________
Stackless mailing list
Stackless at www.tismer.com
http://www.tismer.com/mailman/listinfo/stackless