Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python . One of the issues I have is live replacing of a method to an existing instance.
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python .
Now I am confused, I thought you already did this? In the other thread you wrote:
"From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
One of the issues I have is live replacing of a method to an existing instance.
And in the other thread you wrote:
"Well live coding is a simple process of reloading code. In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes.
Python you basically reload the module. Because Python does not compile per method but per module which means executing code lively. This happens because when you import a module in Python, Python does nothing special than executing the source code. Not before compilation but during execution. Hence live coding, the Python VM replaces objects lively. Python can also compile any size of code including individual methods.
That happens with one line of code importlib.reload(mymodule) "
So, if python can already reload code and replace the instances, what is missing ?
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
I remember a project that developed some tools for back-in-time debugging and testing https://www.hpi.uni-potsdam.de/hirschfeld/trac/SqueakCommunityProjects/wiki/...
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
on the subject of " In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes" I was wrong, the rest it correct in the live execution of the code and other objects.
Pharo VM will update the class objects (together with other type of objects) but it wont update instance objects.Probably this is done on purpose for not overriding and losing on reload the existing live state and live code (in this case instance methods), because module reloading is not made specific for live coding as I explained in the thread. So a class object will be replaced but not an instance. What led me to this wrong assumption was that the code I am working on is reinitialising my objects for debugging purposes because currently live state is not my concern as I am working on a GUI and some other technical reason. I did not realise that it was my reinisitialisation that was updating my instance objects(method wise, state was lost obviously) and not Python VM. Hence in my code the instance objects were always updated to the latest code as I was changing it. However now I enter the stage that I need to also update the live state without a full reinitialisation.
Its possible to update the instance object manually . So far I have found two ways of doing this a) Reinitialisation and take the state from the old one (instance variables and their values are stored as a dicitionary) and copy them to the new one and discard the old instance or b) take the methods (methods in python are objects and so are functions so they can be referenced) from the new one and copy them to the old one and discard the new instance. I have tested both, both work as expected. Now I am in the process of finding how to detect that a class changed in a module so i dont have to reinitialise all classes (I know the overall solution to this as well) in a module and because reloading works per module at least I have avoided the other instances. But that is not my problem.
My problem is what other challanges I have to face which I am not aware of live coding wise. Because I never tried implementing live coding enviroment in another language (my experiements are focused only on live execution of code because at the time the idea was to keep using Pharo for live state data) before it would be great if experts give me an insight about Pharo's live coding internals. This way I can "steal" cool ideas that I may not be aware they exists and make my live coding enviroment library progressively closer to Pharo.
Hence, I need guidance for the advanced stuff.
On Wed, Oct 11, 2017 at 2:09 PM Nicolai Hess nicolaihess@gmail.com wrote:
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python .
Now I am confused, I thought you already did this? In the other thread you wrote:
"From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
One of the issues I have is live replacing of a method to an existing instance.
And in the other thread you wrote:
"Well live coding is a simple process of reloading code. In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes.
Python you basically reload the module. Because Python does not compile per method but per module which means executing code lively. This happens because when you import a module in Python, Python does nothing special than executing the source code. Not before compilation but during execution. Hence live coding, the Python VM replaces objects lively. Python can also compile any size of code including individual methods.
That happens with one line of code importlib.reload(mymodule) "
So, if python can already reload code and replace the instances, what is missing ?
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
I remember a project that developed some tools for back-in-time debugging and testing
https://www.hpi.uni-potsdam.de/hirschfeld/trac/SqueakCommunityProjects/wiki/...
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
oops forgot to thank you for link, will give a thorough read. It seems the project you talking about is PathFinder.
On Wed, Oct 11, 2017 at 3:11 PM Dimitris Chloupis kilon.alios@gmail.com wrote:
on the subject of " In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes" I was wrong, the rest it correct in the live execution of the code and other objects.
Pharo VM will update the class objects (together with other type of objects) but it wont update instance objects.Probably this is done on purpose for not overriding and losing on reload the existing live state and live code (in this case instance methods), because module reloading is not made specific for live coding as I explained in the thread. So a class object will be replaced but not an instance. What led me to this wrong assumption was that the code I am working on is reinitialising my objects for debugging purposes because currently live state is not my concern as I am working on a GUI and some other technical reason. I did not realise that it was my reinisitialisation that was updating my instance objects(method wise, state was lost obviously) and not Python VM. Hence in my code the instance objects were always updated to the latest code as I was changing it. However now I enter the stage that I need to also update the live state without a full reinitialisation.
Its possible to update the instance object manually . So far I have found two ways of doing this a) Reinitialisation and take the state from the old one (instance variables and their values are stored as a dicitionary) and copy them to the new one and discard the old instance or b) take the methods (methods in python are objects and so are functions so they can be referenced) from the new one and copy them to the old one and discard the new instance. I have tested both, both work as expected. Now I am in the process of finding how to detect that a class changed in a module so i dont have to reinitialise all classes (I know the overall solution to this as well) in a module and because reloading works per module at least I have avoided the other instances. But that is not my problem.
My problem is what other challanges I have to face which I am not aware of live coding wise. Because I never tried implementing live coding enviroment in another language (my experiements are focused only on live execution of code because at the time the idea was to keep using Pharo for live state data) before it would be great if experts give me an insight about Pharo's live coding internals. This way I can "steal" cool ideas that I may not be aware they exists and make my live coding enviroment library progressively closer to Pharo.
Hence, I need guidance for the advanced stuff.
On Wed, Oct 11, 2017 at 2:09 PM Nicolai Hess nicolaihess@gmail.com wrote:
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python .
Now I am confused, I thought you already did this? In the other thread you wrote:
"From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
One of the issues I have is live replacing of a method to an existing instance.
And in the other thread you wrote:
"Well live coding is a simple process of reloading code. In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes.
Python you basically reload the module. Because Python does not compile per method but per module which means executing code lively. This happens because when you import a module in Python, Python does nothing special than executing the source code. Not before compilation but during execution. Hence live coding, the Python VM replaces objects lively. Python can also compile any size of code including individual methods.
That happens with one line of code importlib.reload(mymodule) "
So, if python can already reload code and replace the instances, what is missing ?
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
I remember a project that developed some tools for back-in-time debugging and testing
https://www.hpi.uni-potsdam.de/hirschfeld/trac/SqueakCommunityProjects/wiki/...
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
On Wed, Oct 11, 2017 at 8:11 PM, Dimitris Chloupis kilon.alios@gmail.com wrote:
on the subject of " In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes" I was wrong, the rest it correct in the live execution of the code and other objects.
Pharo VM will update the class objects (together with other type of objects) but it wont update instance objects.Probably this is done on purpose for not overriding and losing on reload the existing live state and live code (in this case instance methods), because module reloading is not made specific for live coding as I explained in the thread. So a class object will be replaced but not an instance. What led me to this wrong assumption was that the code I am working on is reinitialising my objects for debugging purposes because currently live state is not my concern as I am working on a GUI and some other technical reason. I did not realise that it was my reinisitialisation that was updating my instance objects(method wise, state was lost obviously) and not Python VM. Hence in my code the instance objects were always updated to the latest code as I was changing it. However now I enter the stage that I need to also update the live state without a full reinitialisation.
Its possible to update the instance object manually . So far I have found two ways of doing this a) Reinitialisation and take the state from the old one (instance variables and their values are stored as a dicitionary) and copy them to the new one and discard the old instance or b) take the methods (methods in python are objects and so are functions so they can be referenced) from the new one and copy them to the old one and discard the new instance. I have tested both, both work as expected. Now I am in the process of finding how to detect that a class changed in a module so i dont have to reinitialise all classes (I know the overall solution to this as well) in a module and because reloading works per module at least I have avoided the other instances. But that is not my problem.
My problem is what other challanges I have to face which I am not aware of live coding wise. Because I never tried implementing live coding enviroment in another language (my experiements are focused only on live execution of code because at the time the idea was to keep using Pharo for live state data) before it would be great if experts give me an insight about Pharo's live coding internals. This way I can "steal" cool ideas that I may not be aware they exists and make my live coding environment library progressively closer to Pharo.
Hence, I need guidance for the advanced stuff.
I'm not sure if this matches what you want, but I remember finding these articles enlightening about VM internals... * https://clementbera.wordpress.com/2013/08/09/the-cog-vm-lookup/ * http://www.mirandabanda.org/cogblog/2011/03/01/build-me-a-jit-as-fast-as-you...
Alternatively, with your background and focus on Python, maybe you can learn something from RSqueak? ... * http://scg.unibe.ch/archive/papers/Bolz08aSpy.pdf * http://stefan-marr.de/2014/02/how-to-get-a-jit-compiler-for-free-implementin...
cheers -ben
On Wed, Oct 11, 2017 at 2:09 PM Nicolai Hess nicolaihess@gmail.com wrote:
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python .
Now I am confused, I thought you already did this? In the other thread you wrote:
"From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
One of the issues I have is live replacing of a method to an existing instance.
And in the other thread you wrote:
"Well live coding is a simple process of reloading code. In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes.
Python you basically reload the module. Because Python does not compile per method but per module which means executing code lively. This happens because when you import a module in Python, Python does nothing special than executing the source code. Not before compilation but during execution. Hence live coding, the Python VM replaces objects lively. Python can also compile any size of code including individual methods.
That happens with one line of code importlib.reload(mymodule) "
So, if python can already reload code and replace the instances, what is missing ?
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
I remember a project that developed some tools for back-in-time debugging and testing https://www.hpi.uni-potsdam.de/hirschfeld/trac/ SqueakCommunityProjects/wiki/pathToolsFramework
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
unfortunately no, those articles focus on JIT compilation and method lookups and not live coding.
I know how to find method objects , thats not my problem. JIT is not my concern either. But none the less thanks for the links they made a very enlighting read after lunch.
Well lets first start with the obvious questions
1) What are the live coding capabilities/features of the Cog VM ? 2) What are its live coding limitations ? 3) When a live object is updated is it only the new compiled method that is injected ? 4) Does this injection follows a particular schedule or process ? Or is something happening ASAP after the user accepts the method ? 4) Does class and method renaming also happen at VM level ? Is it related to live coding ?
On Wed, Oct 11, 2017 at 3:49 PM Ben Coman btc@openinworld.com wrote:
On Wed, Oct 11, 2017 at 8:11 PM, Dimitris Chloupis kilon.alios@gmail.com wrote:
on the subject of " In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes" I was wrong, the rest it correct in the live execution of the code and other objects.
Pharo VM will update the class objects (together with other type of objects) but it wont update instance objects.Probably this is done on purpose for not overriding and losing on reload the existing live state and live code (in this case instance methods), because module reloading is not made specific for live coding as I explained in the thread. So a class object will be replaced but not an instance. What led me to this wrong assumption was that the code I am working on is reinitialising my objects for debugging purposes because currently live state is not my concern as I am working on a GUI and some other technical reason. I did not realise that it was my reinisitialisation that was updating my instance objects(method wise, state was lost obviously) and not Python VM. Hence in my code the instance objects were always updated to the latest code as I was changing it. However now I enter the stage that I need to also update the live state without a full reinitialisation.
Its possible to update the instance object manually . So far I have found two ways of doing this a) Reinitialisation and take the state from the old one (instance variables and their values are stored as a dicitionary) and copy them to the new one and discard the old instance or b) take the methods (methods in python are objects and so are functions so they can be referenced) from the new one and copy them to the old one and discard the new instance. I have tested both, both work as expected. Now I am in the process of finding how to detect that a class changed in a module so i dont have to reinitialise all classes (I know the overall solution to this as well) in a module and because reloading works per module at least I have avoided the other instances. But that is not my problem.
My problem is what other challanges I have to face which I am not aware of live coding wise. Because I never tried implementing live coding enviroment in another language (my experiements are focused only on live execution of code because at the time the idea was to keep using Pharo for live state data) before it would be great if experts give me an insight about Pharo's live coding internals. This way I can "steal" cool ideas that I may not be aware they exists and make my live coding environment library progressively closer to Pharo.
Hence, I need guidance for the advanced stuff.
I'm not sure if this matches what you want, but I remember finding these articles enlightening about VM internals...
http://www.mirandabanda.org/cogblog/2011/03/01/build-me-a-jit-as-fast-as-you...
Alternatively, with your background and focus on Python, maybe you can learn something from RSqueak? ...
http://stefan-marr.de/2014/02/how-to-get-a-jit-compiler-for-free-implementin...
cheers -ben
On Wed, Oct 11, 2017 at 2:09 PM Nicolai Hess nicolaihess@gmail.com wrote:
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python .
Now I am confused, I thought you already did this? In the other thread you wrote:
"From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
One of the issues I have is live replacing of a method to an existing instance.
And in the other thread you wrote:
"Well live coding is a simple process of reloading code. In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes.
Python you basically reload the module. Because Python does not compile per method but per module which means executing code lively. This happens because when you import a module in Python, Python does nothing special than executing the source code. Not before compilation but during execution. Hence live coding, the Python VM replaces objects lively. Python can also compile any size of code including individual methods.
That happens with one line of code importlib.reload(mymodule) "
So, if python can already reload code and replace the instances, what is missing ?
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
I remember a project that developed some tools for back-in-time debugging and testing
https://www.hpi.uni-potsdam.de/hirschfeld/trac/SqueakCommunityProjects/wiki/...
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
Dimitris,
unfortunately no, those articles focus on JIT compilation and method lookups and not live coding.
Actually, these things are at the very core of live coding. Unless we are using the term in very different ways. So let me explain what I mean by "live coding".
Imagine that I have two code browsers showing methods like:
methA: x ... [...] whileTrue: [ self methB ]. ...
and
methoB ... ...
and I have a third window where I am watching the effect of running #methA. What happens if I change the text in the second window and select "accept"? What happens if I edit the first window and accept?
If the system is a simple interpreter and "accept" generated new bytecodes that get installed in the method dictionary in place of the previous version of the method, we would expect an instant change in behavior when changing #methB but nothing would happen when #methA is modified. That is because #methB is repeatedly called and the current execution finishes executing using the old code and then it gets invoked again and starts executing the new code. #methA, on the other hand, is in the middle of a very long loop and will keep executing the old code (pointed to from the stack) even though the method dictionary now has new code.
An alternative system could use #become: to replace bytecodes when changing a method, but that would be hard to do (suddenly the stack state would no longer make sense with the new bytecodes) and would be harder for the user to understand. So let's not go there.
If we have a JIT and not a simple interpreter, things can be complicated if we want to keep the same semantics. The compiler might have inlined #methB into #methA and in that case changing #methB would no longer have a visible effect in the third window. The technology to make it work as expected is called "dynamic deoptimization" and is pretty sophisticated.
-- Jecel
Jecel , you hit a head on the issues I want to tackle, this exactly the concept I want to understand to implement an efficient live coding enviroment so thank you.
Before starting excuse any misunderstanding I am very fresh into the whole idea of live coding implementation so I may not understand the specific you mention.
Now the way I understand this in Python its possible to keep around both the old and new instance , if a method is replaced what happens is that Python treats methods a bit like variables, again I am not talking VM I know nothing about Python VM internals. This is purely python code.
So if you have a method my_instance.my_method(arg1, arg2) the statement my_instance.my_method wont call the method because it does not contain the parenthesses needed for method and function calls
However it wont throw an error either.
What will happen is that like a variable it will return a reference to the method associated. That method is a separate object. Of type MethodObject which has a function __CALL__ which is doing the work of parentheses and of course it has instance variables for arguments, classes that is bound to etc.
So if I do what you say and change methodB and inject the updated version of methodB which basically will be its new instance containing the modified code in bytecode format to the place of old instance, i will be replacing a reference and as such the 'variable' now will reference another method . so Any call that will now be issues will be sent directly to the updated version of the method.
so basically I would be doing following the previous example
updated_method = updated_instance.method my_instance.my_method=updated_method # now points to updated_instance.method my_instace.my_method(arg1,arg2) #this equals to updated_instance.method(arg1,arg2)
It wont matter if its a long loop , as matter of fact I am working on a loop right now doing these experiment because the Python I use is in Blender (3d app) running concurent with many other stuff sometimes GBs of data etc etc.
Unless we talking here millisecond to microsecond responses ? I have not tested in such speeds
Do I understand you correctly or I miss something here. Why you need a JIT to do this ? I see that pharo has CompiledMethod so methods also in pharo are objects that means they are also referenced not tied to the object instances , right ? Should not this work outside the box ?
So in my scenario of live replacing methods you should see the results immediately , meaning as soon as the method is called again and both editors will have to display the same exact code if they view the same method. All calls are essentialy done via reference.
What am I missing here ?
On Wed, Oct 11, 2017 at 6:02 PM Jecel Assumpcao Jr. jecel@merlintec.com wrote:
Dimitris,
unfortunately no, those articles focus on JIT compilation and method lookups and not live coding.
Actually, these things are at the very core of live coding. Unless we are using the term in very different ways. So let me explain what I mean by "live coding".
Imagine that I have two code browsers showing methods like:
methA: x ... [...] whileTrue: [ self methB ]. ...
and
methoB ... ...
and I have a third window where I am watching the effect of running #methA. What happens if I change the text in the second window and select "accept"? What happens if I edit the first window and accept?
If the system is a simple interpreter and "accept" generated new bytecodes that get installed in the method dictionary in place of the previous version of the method, we would expect an instant change in behavior when changing #methB but nothing would happen when #methA is modified. That is because #methB is repeatedly called and the current execution finishes executing using the old code and then it gets invoked again and starts executing the new code. #methA, on the other hand, is in the middle of a very long loop and will keep executing the old code (pointed to from the stack) even though the method dictionary now has new code.
An alternative system could use #become: to replace bytecodes when changing a method, but that would be hard to do (suddenly the stack state would no longer make sense with the new bytecodes) and would be harder for the user to understand. So let's not go there.
If we have a JIT and not a simple interpreter, things can be complicated if we want to keep the same semantics. The compiler might have inlined #methB into #methA and in that case changing #methB would no longer have a visible effect in the third window. The technology to make it work as expected is called "dynamic deoptimization" and is pretty sophisticated.
-- Jecel
A couple years ago Paul Fernhout did start working on a Morphic environment in Python. He tried the live coding too, but couldn't get it to work. This was prompted by Alan Kay suggesting the need for a live coding environment for OLPC. Guido van Rossum himself stepped in and created a module that lets you replace most things on the fly. I don't recall how it worked exactly, but I'm sure you can find it. But it never found much acceptance in the Python community, AFAIK.
Another approach is the one taken by HPI's RSqueak VM which is built on top of PyPy and lets you develop, run and debug Smalltalk and Python code ( https://conf.researchr.org/profile/MoreVMs-2017/fabioniephaus).
- Bert -
On Wed 11. Oct 2017 at 14:50, Ben Coman btc@openinworld.com wrote:
On Wed, Oct 11, 2017 at 8:11 PM, Dimitris Chloupis kilon.alios@gmail.com wrote:
on the subject of " In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes" I was wrong, the rest it correct in the live execution of the code and other objects.
Pharo VM will update the class objects (together with other type of objects) but it wont update instance objects.Probably this is done on purpose for not overriding and losing on reload the existing live state and live code (in this case instance methods), because module reloading is not made specific for live coding as I explained in the thread. So a class object will be replaced but not an instance. What led me to this wrong assumption was that the code I am working on is reinitialising my objects for debugging purposes because currently live state is not my concern as I am working on a GUI and some other technical reason. I did not realise that it was my reinisitialisation that was updating my instance objects(method wise, state was lost obviously) and not Python VM. Hence in my code the instance objects were always updated to the latest code as I was changing it. However now I enter the stage that I need to also update the live state without a full reinitialisation.
Its possible to update the instance object manually . So far I have found two ways of doing this a) Reinitialisation and take the state from the old one (instance variables and their values are stored as a dicitionary) and copy them to the new one and discard the old instance or b) take the methods (methods in python are objects and so are functions so they can be referenced) from the new one and copy them to the old one and discard the new instance. I have tested both, both work as expected. Now I am in the process of finding how to detect that a class changed in a module so i dont have to reinitialise all classes (I know the overall solution to this as well) in a module and because reloading works per module at least I have avoided the other instances. But that is not my problem.
My problem is what other challanges I have to face which I am not aware of live coding wise. Because I never tried implementing live coding enviroment in another language (my experiements are focused only on live execution of code because at the time the idea was to keep using Pharo for live state data) before it would be great if experts give me an insight about Pharo's live coding internals. This way I can "steal" cool ideas that I may not be aware they exists and make my live coding environment library progressively closer to Pharo.
Hence, I need guidance for the advanced stuff.
I'm not sure if this matches what you want, but I remember finding these articles enlightening about VM internals...
http://www.mirandabanda.org/cogblog/2011/03/01/build-me-a-jit-as-fast-as-you...
Alternatively, with your background and focus on Python, maybe you can learn something from RSqueak? ...
http://stefan-marr.de/2014/02/how-to-get-a-jit-compiler-for-free-implementin...
cheers -ben
On Wed, Oct 11, 2017 at 2:09 PM Nicolai Hess nicolaihess@gmail.com wrote:
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python .
Now I am confused, I thought you already did this? In the other thread you wrote:
"From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
One of the issues I have is live replacing of a method to an existing instance.
And in the other thread you wrote:
"Well live coding is a simple process of reloading code. In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes.
Python you basically reload the module. Because Python does not compile per method but per module which means executing code lively. This happens because when you import a module in Python, Python does nothing special than executing the source code. Not before compilation but during execution. Hence live coding, the Python VM replaces objects lively. Python can also compile any size of code including individual methods.
That happens with one line of code importlib.reload(mymodule) "
So, if python can already reload code and replace the instances, what is missing ?
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
I remember a project that developed some tools for back-in-time debugging and testing
https://www.hpi.uni-potsdam.de/hirschfeld/trac/SqueakCommunityProjects/wiki/...
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
On 11.10.2017, at 15:50, Bert Freudenberg bert@freudenbergs.de wrote:
A couple years ago Paul Fernhout did start working on a Morphic environment in Python. He tried the live coding too, but couldn't get it to work. This was prompted by Alan Kay suggesting the need for a live coding environment for OLPC. Guido van Rossum himself stepped in and created a module that lets you replace most things on the fly. I don't recall how it worked exactly, but I'm sure you can find it. But it never found much acceptance in the Python community, AFAIK.
http://www.diva-portal.org/smash/get/diva2:22296/FULLTEXT01.pdf
Ah PyMorphic, yeah I actually used it for making a Blender Python GUI API. Really clean code but only a tiny fraction of Morphic abilities. I did not know it tackled live coding, I dont remember seeing anything relevant in the source code. Maybe i should take a nother look thanks.
On Wed, Oct 11, 2017 at 4:56 PM Tobias Pape Das.Linux@gmx.de wrote:
On 11.10.2017, at 15:50, Bert Freudenberg bert@freudenbergs.de wrote:
A couple years ago Paul Fernhout did start working on a Morphic
environment in Python. He tried the live coding too, but couldn't get it to work. This was prompted by Alan Kay suggesting the need for a live coding environment for OLPC. Guido van Rossum himself stepped in and created a module that lets you replace most things on the fly. I don't recall how it worked exactly, but I'm sure you can find it. But it never found much acceptance in the Python community, AFAIK.
http://www.diva-portal.org/smash/get/diva2:22296/FULLTEXT01.pdf
Are you talking about live instances updating or module reloading ? Module reloading is very old , certianly not a couple of years
According to the archives of Python documentation it was available at version 1.5 back in Aprill 1998
https://docs.python.org/release/1.5/lib/node38.html#SECTION00411000000000000...
About the other features I cannot vouche for back then , but many do exist nowdays.
live instance updating is not a problem as I said , python can replace methods lively in a existing instance or inject new methods that exist only in a single instance (you can assign even a fuction as a method to an instance runtime because functions are also objects) , manipulate its live state by changing/renaming/creating instance variables again during execution etc. I can do this with simple python commands. Python also like Cog VM has its own dictionarie for methods and classes, seperate dictionaries for instance and class variables. Each method in an instance is a separate object and so is each variable as pretty much anything in Python is an object, etc etc.
No idea about the Guido Van Rossume module you speak of. Those features are not part of a module or library they are an internal part of the language. The one that is a module is the imp library which nowdays is called importlib, which is responsibe for reloading a module and generallly other module functions.
I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ?
On Wed, Oct 11, 2017 at 4:50 PM Bert Freudenberg bert@freudenbergs.de wrote:
A couple years ago Paul Fernhout did start working on a Morphic environment in Python. He tried the live coding too, but couldn't get it to work. This was prompted by Alan Kay suggesting the need for a live coding environment for OLPC. Guido van Rossum himself stepped in and created a module that lets you replace most things on the fly. I don't recall how it worked exactly, but I'm sure you can find it. But it never found much acceptance in the Python community, AFAIK.
Another approach is the one taken by HPI's RSqueak VM which is built on top of PyPy and lets you develop, run and debug Smalltalk and Python code ( https://conf.researchr.org/profile/MoreVMs-2017/fabioniephaus).
- Bert -
On Wed 11. Oct 2017 at 14:50, Ben Coman btc@openinworld.com wrote:
On Wed, Oct 11, 2017 at 8:11 PM, Dimitris Chloupis <kilon.alios@gmail.com
wrote:
on the subject of " In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes" I was wrong, the rest it correct in the live execution of the code and other objects.
Pharo VM will update the class objects (together with other type of objects) but it wont update instance objects.Probably this is done on purpose for not overriding and losing on reload the existing live state and live code (in this case instance methods), because module reloading is not made specific for live coding as I explained in the thread. So a class object will be replaced but not an instance. What led me to this wrong assumption was that the code I am working on is reinitialising my objects for debugging purposes because currently live state is not my concern as I am working on a GUI and some other technical reason. I did not realise that it was my reinisitialisation that was updating my instance objects(method wise, state was lost obviously) and not Python VM. Hence in my code the instance objects were always updated to the latest code as I was changing it. However now I enter the stage that I need to also update the live state without a full reinitialisation.
Its possible to update the instance object manually . So far I have found two ways of doing this a) Reinitialisation and take the state from the old one (instance variables and their values are stored as a dicitionary) and copy them to the new one and discard the old instance or b) take the methods (methods in python are objects and so are functions so they can be referenced) from the new one and copy them to the old one and discard the new instance. I have tested both, both work as expected. Now I am in the process of finding how to detect that a class changed in a module so i dont have to reinitialise all classes (I know the overall solution to this as well) in a module and because reloading works per module at least I have avoided the other instances. But that is not my problem.
My problem is what other challanges I have to face which I am not aware of live coding wise. Because I never tried implementing live coding enviroment in another language (my experiements are focused only on live execution of code because at the time the idea was to keep using Pharo for live state data) before it would be great if experts give me an insight about Pharo's live coding internals. This way I can "steal" cool ideas that I may not be aware they exists and make my live coding environment library progressively closer to Pharo.
Hence, I need guidance for the advanced stuff.
I'm not sure if this matches what you want, but I remember finding these articles enlightening about VM internals...
http://www.mirandabanda.org/cogblog/2011/03/01/build-me-a-jit-as-fast-as-you...
Alternatively, with your background and focus on Python, maybe you can learn something from RSqueak? ...
http://stefan-marr.de/2014/02/how-to-get-a-jit-compiler-for-free-implementin...
cheers -ben
On Wed, Oct 11, 2017 at 2:09 PM Nicolai Hess nicolaihess@gmail.com wrote:
2017-10-11 12:14 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Hey there as it says in the title I am trying to recreat live coding Pharo chracteristics in Python .
Now I am confused, I thought you already did this? In the other thread you wrote:
"From my experience around 100 lines of code are enough to offer most of Pharo’s basic live coding functionality"
One of the issues I have is live replacing of a method to an existing instance.
And in the other thread you wrote:
"Well live coding is a simple process of reloading code. In Pharo the VM recompiles a method and replace it’s old instance with a new one while code executes.
Python you basically reload the module. Because Python does not compile per method but per module which means executing code lively. This happens because when you import a module in Python, Python does nothing special than executing the source code. Not before compilation but during execution. Hence live coding, the Python VM replaces objects lively. Python can also compile any size of code including individual methods.
That happens with one line of code importlib.reload(mymodule) "
So, if python can already reload code and replace the instances, what is missing ?
I know how to do this in Python but that shown me I would benefit greatily from a more deeper understanding of Pharo live coding internals than I have at the moment which is only at the user level. This will help me avoid common pitfalls in a field that I am not experienced with.
Have you guys written any paper , article or blog post in this area, explaining the internals of the VM to that regard ?
I am also interested in time machine livde coding, meaning live coding that does not keeps you in the present but also can send you back in the past restoring the live state of old object instances in a specific time moment.
I remember a project that developed some tools for back-in-time debugging and testing
https://www.hpi.uni-potsdam.de/hirschfeld/trac/SqueakCommunityProjects/wiki/...
Generally any guidance on the art of implementing live coding will be greatly appreciated.
I don't care how this can be done in Python because I have already recreated a very basic live coding enviroment in Python and there is a lot of documentation about Python overall what I am interested only is the Smalltalk way of doing live coding but from the perspective of the implementors not the mere users.
On Wed 11. Oct 2017 at 16:27, Dimitris Chloupis kilon.alios@gmail.com wrote:
I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ?
Instance enumeration, the become operation, and reified stack frames are features that set Smalltalk apart and enable many of its live coding capabilities. Cog supports these features. Cog was specifically designed to allow fast stack operations in the normal case while still allowing frame manipulation. Spur was designed to make "become" fast, it used to be very expensive.
- Bert -
does "become" only exchange references ? Or is it more to it than that ?
where I can find information about what insance enumeration and reified stack frames are ?
Python has methods in gc module (module for its garbage collector) to find objects that point to an object or objects that an object point to . But it has no convenient method to exchange those references . I assume you could do it via an iteration but I never tried it in practice.
Indeed such feature is essential for live coding. I will have to add it to my implementation thanks.
Is this book still relevant for my implementation ? http://stephane.ducasse.free.fr/FreeBooks/BlueBook/Bluebook.pdf
On Wed, Oct 11, 2017 at 6:07 PM Bert Freudenberg bert@freudenbergs.de wrote:
On Wed 11. Oct 2017 at 16:27, Dimitris Chloupis kilon.alios@gmail.com wrote:
I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ?
Instance enumeration, the become operation, and reified stack frames are features that set Smalltalk apart and enable many of its live coding capabilities. Cog supports these features. Cog was specifically designed to allow fast stack operations in the normal case while still allowing frame manipulation. Spur was designed to make "become" fast, it used to be very expensive.
- Bert -
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Indeed such feature is essential for live coding. I will have to add it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for instance does not use #become: yet it still provides live coding (in my opinion).
Do you know how instances are reshaped without become in Smalltalk/X?
But indeed it may not be a problem in Python where instances do not have a fixed number of instance variables anyway. You still need to be able to enumerate them, to fix them up.
- Bert -
On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote:
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Indeed such feature is essential for live coding. I will have to
add
it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for instance does not use #become: yet it still provides live coding (in my opinion).
Do you know how instances are reshaped without become in Smalltalk/X?
Yes, it's very simple: they are not reshaped at all :-)
Jan
2017-10-11 21:56 GMT+02:00 Jan Vrany jan.vrany@fit.cvut.cz:
On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote:
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Indeed such feature is essential for live coding. I will have to
add
it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for instance does not use #become: yet it still provides live coding (in my opinion).
Do you know how instances are reshaped without become in Smalltalk/X?
Yes, it's very simple: they are not reshaped at all :-)
Jan
Then does that mean that - classes are not mutated, but that a new one is created, and old objects are still pointing on old classes, - or that mutation is forbidden as long as instances or sub-instances exist (like in Smalltalk V if I remember) - or that one may expect catastrophic consequences when reshaping? Nicolas
On Wed, 2017-10-11 at 22:01 +0200, Nicolas Cellier wrote:
2017-10-11 21:56 GMT+02:00 Jan Vrany jan.vrany@fit.cvut.cz:
On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote:
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <jan.vrany@fit.cvut.cz
wrote:
Indeed such feature is essential for live coding. I will
have to
add
it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for instance
does
not use #become: yet it still provides live coding (in my opinion).
Do you know how instances are reshaped without become in
Smalltalk/X?
Yes, it's very simple: they are not reshaped at all :-)
Jan
Then does that mean that
- classes are not mutated, but that a new one is created, and old
objects are still pointing on old classes,
Exactly. The old class eventually becomes (not #become:s :-) garbage and gets (eventually) collected.
- or that mutation is forbidden as long as instances or sub-instances
exist (like in Smalltalk V if I remember)
- or that one may expect catastrophic consequences when reshaping?
Depends what you call catastrophic consequences...
Jan
2017-10-11 22:11 GMT+02:00 Jan Vrany jan.vrany@fit.cvut.cz:
On Wed, 2017-10-11 at 22:01 +0200, Nicolas Cellier wrote:
2017-10-11 21:56 GMT+02:00 Jan Vrany jan.vrany@fit.cvut.cz:
On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote:
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <jan.vrany@fit.cvut.cz
wrote:
Indeed such feature is essential for live coding. I will
have to
add
it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for instance
does
not use #become: yet it still provides live coding (in my opinion).
Do you know how instances are reshaped without become in
Smalltalk/X?
Yes, it's very simple: they are not reshaped at all :-)
Jan
Then does that mean that
- classes are not mutated, but that a new one is created, and old
objects are still pointing on old classes,
Exactly. The old class eventually becomes (not #become:s :-) garbage and gets (eventually) collected.
- or that mutation is forbidden as long as instances or sub-instances
exist (like in Smalltalk V if I remember)
- or that one may expect catastrophic consequences when reshaping?
Depends what you call catastrophic consequences...
Jan
Catastrophic could be for example a buffer overflow that would bypass VM safety... Typically when instances and compiled methods idea of inst. var. layout divorces. That does not seem the case in Smalltalk/X from what you describe... For example, in Squeak or VW, when there is a Foo>>bar method up in the stack that triggers removal of an ivar, then writes into an ivar, boom! object memory is corrupted... Indeed the methods cannot easily be become'd while still active... So they are replaced - but the instances are become'd
On Wed, 2017-10-11 at 22:19 +0200, Nicolas Cellier wrote:
2017-10-11 22:11 GMT+02:00 Jan Vrany jan.vrany@fit.cvut.cz:
On Wed, 2017-10-11 at 22:01 +0200, Nicolas Cellier wrote:
2017-10-11 21:56 GMT+02:00 Jan Vrany jan.vrany@fit.cvut.cz:
On Wed, 2017-10-11 at 21:49 +0200, Bert Freudenberg wrote:
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany <jan.vrany@fit.cvu
t.cz
wrote:
> Indeed such feature is essential for live coding. I will
have to
add > it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for
instance
does
not use #become: yet it still provides live coding (in my
opinion).
Do you know how instances are reshaped without become in
Smalltalk/X?
Yes, it's very simple: they are not reshaped at all :-)
Jan
Then does that mean that
- classes are not mutated, but that a new one is created, and old
objects are still pointing on old classes,
Exactly. The old class eventually becomes (not #become:s :-) garbage and gets (eventually) collected.
- or that mutation is forbidden as long as instances or sub-
instances
exist (like in Smalltalk V if I remember)
- or that one may expect catastrophic consequences when
reshaping?
Depends what you call catastrophic consequences...
Jan
Catastrophic could be for example a buffer overflow that would bypass VM safety... Typically when instances and compiled methods idea of inst. var. layout divorces. That does not seem the case in Smalltalk/X from what you describe... For example, in Squeak or VW, when there is a Foo>>bar method up in the stack that triggers removal of an ivar, then writes into an ivar, boom! object memory is corrupted...
I see. No, this should not happen in Smalltalk/X. Indeed all that logic implemented on Smalltalk level so one can construct code that results in this kind of a corruption (simply change offsets in method's bytecode array).
To make the system robust against this sort of problems one might need to take the approach not unlike the one of Java (JVM) which is more robust w.r.t. this. This is one of the reasons (not the only one) why "live coding" in Java is lot more complicated (but not entirely impossible).
Indeed the methods cannot easily be become'd while still active... So they are replaced - but the instances are become'd
Jan
Python allows you to inject new variables to an instance through normal assignement as you can do also with methods. So you replace existing ones as I mentioned before but also extend the instance with new variables and methods after its created.
The one thing I thought Python would not allow me to do was to delete variables and especially methods from an instance but it appears it does that too. Always during run time of course.
Its safe to assume that the same applies to object classes as well. So technically this means you have complete freedom on how much you can affect a live instance. At least thats how it looks to me so far.
On Wed, Oct 11, 2017 at 10:50 PM Bert Freudenberg bert@freudenbergs.de wrote:
On Wed, Oct 11, 2017 at 9:32 PM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Indeed such feature is essential for live coding. I will have to add it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for instance does not use #become: yet it still provides live coding (in my opinion).
Do you know how instances are reshaped without become in Smalltalk/X?
But indeed it may not be a problem in Python where instances do not have a fixed number of instance variables anyway. You still need to be able to enumerate them, to fix them up.
- Bert -
I found actually a way to have become without become :D
what if instead of actually changing the references of one object to the other we changed the object themselves and instead of exchanging references we exchange instance variables and instance methods ?
This way objects that pointed to object a will still point to object a instead of object b but a will now have really become object b , and object b will have become object a.
is there anything more to an instance besides its instance variables and methods ? I am including of course inherited instance variables and methods.
I tried this in Python and it worked but I am not sure if become is doing anything more than that.
On Wed, Oct 11, 2017 at 10:32 PM Jan Vrany jan.vrany@fit.cvut.cz wrote:
Indeed such feature is essential for live coding. I will have to add it to my implementation thanks.
I don't think become: is essential. Smalltalk/X for instance does not use #become: yet it still provides live coding (in my opinion).
HTH, Jan
On Thu, Oct 12, 2017 at 12:38 AM, Dimitris Chloupis kilon.alios@gmail.com wrote:
does "become" only exchange references ? Or is it more to it than that ?
https://gbracha.blogspot.com.au/2009/07/miracle-of-become.html
Also interesting reading... https://clementbera.wordpress.com/2013/08/09/the-cog-vm-lookup/
where I can find information about what insance enumeration and reified stack frames are ?
Lhe last entry here... http://www.mirandabanda.org/cogblog/on-line-papers-and-presentations/
Also, does Python do Polymorphic Inline Caching?... https://en.wikipedia.org/wiki/Inline_caching
or bytecode level hotspot recompilation... https://clementbera.wordpress.com/2014/01/09/the-sista-chronicles-i-an-intro...
cheers -ben
Python has methods in gc module (module for its garbage collector) to find objects that point to an object or objects that an object point to . But it has no convenient method to exchange those references . I assume you could do it via an iteration but I never tried it in practice.
Indeed such feature is essential for live coding. I will have to add it to my implementation thanks.
Is this book still relevant for my implementation ? http://stephane.ducasse.free.fr/FreeBooks/BlueBook/Bluebook.pdf
On Wed, Oct 11, 2017 at 6:07 PM Bert Freudenberg bert@freudenbergs.de wrote:
On Wed 11. Oct 2017 at 16:27, Dimitris Chloupis kilon.alios@gmail.com wrote:
I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ?
Instance enumeration, the become operation, and reified stack frames are features that set Smalltalk apart and enable many of its live coding capabilities. Cog supports these features. Cog was specifically designed to allow fast stack operations in the normal case while still allowing frame manipulation. Spur was designed to make "become" fast, it used to be very expensive.
- Bert -
Thanks Ben actually your first link was the first thing I read when I googled become. I dont think VM lookup benefits me in my case as this is already handled by the Python VM.
You other tlinks are JIT VM related, Python is not friendly to JIT because its creator believe that performance enhancement should come from wrapping C libraries and wants to avoid the cost of more complex VM . In the case of Python this is easy to follow because Python has a ton of high performance libraries which by the way almost never are used when comparing Python with other languages speed wise (kinda fare if you just compare language implementations). Pharo does benefit a lot more from JIT VM as it does not have as many libraries targeting high performance.
On the last link, I do not know if I understand this correctly at sista side but at python side yes you can take any string and compile to bytecode real time and inject that code back to your live execution. If that is what you mean. Live bytecode compilation can happen at module level too of course , and a string can be anything from one line of code to a collection of modules. The cool thing is that you can sandbox the compiling too so you do not cause any security issues, giving the code access only to a fraction of your state and code. So yes it gives you a lot of room to move around in this department but again there is no JIT VM , at least for CPython.
PyPy is a JIT VM for python with Slang like language called rpython that can be used to port any language and I spoken with some of it authoer and seem to follow Cog VM vclosely and of coruse expressed admiration of Eliot's work. Unfortunately PyPy had a long track of issues with python libraries that wrapped C code so it never get a lot of traction, but its non the less used by some python coders
If anyone does mind I would like some pointers on the subject of class enumaration and reified stack frames that were mentioned earlier on, I googled them but did not find enough to clarify what they are in case of Cog VM.
On Thu, Oct 12, 2017 at 12:42 AM Ben Coman btc@openinworld.com wrote:
On Thu, Oct 12, 2017 at 12:38 AM, Dimitris Chloupis <kilon.alios@gmail.com
wrote:
does "become" only exchange references ? Or is it more to it than that ?
https://gbracha.blogspot.com.au/2009/07/miracle-of-become.html
Also interesting reading... https://clementbera.wordpress.com/2013/08/09/the-cog-vm-lookup/
where I can find information about what insance enumeration and reified stack frames are ?
Lhe last entry here... http://www.mirandabanda.org/cogblog/on-line-papers-and-presentations/
Also, does Python do Polymorphic Inline Caching?... https://en.wikipedia.org/wiki/Inline_caching
or bytecode level hotspot recompilation...
https://clementbera.wordpress.com/2014/01/09/the-sista-chronicles-i-an-intro...
cheers -ben
Python has methods in gc module (module for its garbage collector) to find objects that point to an object or objects that an object point to . But it has no convenient method to exchange those references . I assume you could do it via an iteration but I never tried it in practice.
Indeed such feature is essential for live coding. I will have to add it to my implementation thanks.
Is this book still relevant for my implementation ? http://stephane.ducasse.free.fr/FreeBooks/BlueBook/Bluebook.pdf
On Wed, Oct 11, 2017 at 6:07 PM Bert Freudenberg bert@freudenbergs.de wrote:
On Wed 11. Oct 2017 at 16:27, Dimitris Chloupis kilon.alios@gmail.com wrote:
I am talking about beyond those obvious features that Python has them covered is there anything special the Cog VM is doing live coding wise that I am not aware of ?
Instance enumeration, the become operation, and reified stack frames are features that set Smalltalk apart and enable many of its live coding capabilities. Cog supports these features. Cog was specifically designed to allow fast stack operations in the normal case while still allowing frame manipulation. Spur was designed to make "become" fast, it used to be very expensive.
- Bert -
2017-10-12 10:50 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Thanks Ben actually your first link was the first thing I read when I googled become. I dont think VM lookup benefits me in my case as this is already handled by the Python VM.
You other tlinks are JIT VM related, Python is not friendly to JIT because its creator believe that performance enhancement should come from wrapping C libraries and wants to avoid the cost of more complex VM . In the case of Python this is easy to follow because Python has a ton of high performance libraries which by the way almost never are used when comparing Python with other languages speed wise (kinda fare if you just compare language implementations). Pharo does benefit a lot more from JIT VM as it does not have as many libraries targeting high performance.
The interesting parts in this "JIT VM related" links is, if you happen to write a JIT VM , you have to take care of the language features smalltalk provides. For example, you can not just optimize away smalltalks context chain by replacing it with native stack frames, without losing the features a smallalltak method context provides on the language level. This language features are already explained very well in the blue book, but the links about the cog vm and other vm and jit vm papers for dynamic langueges are helpful too.
I don't know which language features (late bindng, method lookup , thisContext / execution context, become operation, etc) are *necessary* to provide the "smalltalk art of live coding". Or to what extent this features exists in other language. But it is one reason why it is "easy" to build within smalltalk itself this smalltalk environment (edit / compile/ run / debug smallalk code from within a running smalltalk environment).
The problem in my case is that I do not build a VM from scratch neither I am modifying an existing VM
For me it boils down to
1) What is this in Smalltalk ? 2) Can Python do this ? 3) Can I make it easier without spending too much time and compromising ?
Python has its own stack frames, they are object too, not native stack frames (they may reference native stack frames though) and if I am not mistaken because unlike my previous statements I have not messed at all with them, they also come with their own context which includes local variables etc
https://stackoverflow.com/questions/38937721/global-frame-vs-stack-frame https://docs.python.org/3/library/inspect.html https://docs.python.org/3/library/traceback.html
I can make to a degree such claims because anything I test live coding wise in python I do it from debugger. The Python debugger is called PDB as you would expect and it works from the terminal , beyond coming with its own commands and can do repl suff it also comes with an instance of Python REPL which of course gives you complete freedom in doing what you would do in source code file. This is how I hack objects removing and adding methods to them , mess with local variables etc. talking about live instance objects etc.
Of course I have to copy the code manually back to the source file , maybe there is a way around this as well or some feature I am not aware of, but the code fragments are usually small so its not a major problem for now. LiveCoding wise, besides the fact that indeed some things do not happen automagically as you would expect in Pharo still the debugger has not stopped me from doing anything live coding wise.
My only big frustration so far has been the fact that classes do not keep a list of their instances, but that is also easy to amend for my own classes adding their instances when it is created in class variable list or dictionary. Technically you can still get all references to an object (via gc module coming included with python) and ask each one if it is an instance of a class , still I think thats a big flaw and introduces extra works that should not happen when Python already tackles far more complex tasks of live coding.
But that's your usual language I guess, there is always something to annoy you.
In any case, I am not dismissing anything and of course I will read and delve deeper live coding related or not, so thanks for any help you guys give even if I do not need it. I may not need it now , but I may need it later on, so learning is always worth it. But I am also careful not to recreate Python functionality because a) I will be wasting my time b) I will be ignoring Python's native design that can create incompatibilities down the road.
So gain a good understanding from both sides Cog VM and Python library (so far I had no need to seek information about Python VM because the functionality I want may not be completely automated but is non the less available as a library part of the standard distribution)
So the deeper I get , of course I expect to get more complex, obviously for now I want to keep it simple and then move step by step to something more sophisticated. But yes I have read the material, not extensively but I did gave it a fast read and I will re read again and I will be also reading the Bluebook.
On Thu, Oct 12, 2017 at 12:58 PM Nicolai Hess nicolaihess@gmail.com wrote:
2017-10-12 10:50 GMT+02:00 Dimitris Chloupis kilon.alios@gmail.com:
Thanks Ben actually your first link was the first thing I read when I googled become. I dont think VM lookup benefits me in my case as this is already handled by the Python VM.
You other tlinks are JIT VM related, Python is not friendly to JIT because its creator believe that performance enhancement should come from wrapping C libraries and wants to avoid the cost of more complex VM . In the case of Python this is easy to follow because Python has a ton of high performance libraries which by the way almost never are used when comparing Python with other languages speed wise (kinda fare if you just compare language implementations). Pharo does benefit a lot more from JIT VM as it does not have as many libraries targeting high performance.
The interesting parts in this "JIT VM related" links is, if you happen to write a JIT VM , you have to take care of the language features smalltalk provides. For example, you can not just optimize away smalltalks context chain by replacing it with native stack frames, without losing the features a smallalltak method context provides on the language level. This language features are already explained very well in the blue book, but the links about the cog vm and other vm and jit vm papers for dynamic langueges are helpful too.
I don't know which language features (late bindng, method lookup , thisContext / execution context, become operation, etc) are *necessary* to provide the "smalltalk art of live coding". Or to what extent this features exists in other language. But it is one reason why it is "easy" to build within smalltalk itself this smalltalk environment (edit / compile/ run / debug smallalk code from within a running smalltalk environment).
Hi Dimitris,
This conference article might be relevant: https://2017.programming-conference.org/event/morevms-2017-papers-when-a-mou...
Cheers, Bernhard
Thanks Berhard
I managed to accomplish a full live coding environment in Python and I am now in the process turning this to a library, to automate all the steps required.
It was with your help I realized my mistakes , especially that post that pointed me to the python implementation of become, guided me towards the solution
My conclusion at least from a user perspective that Python is already a live coding environment, that someone has turned off.
So in order for old instances to be updated there is no need to mess with their objects and hence no need to implement a become functionality. Python allows to change the methods of an old instance with methods of a new updated instances. However this is no solution because other old instances of same class are not updated.
Not to bore you with technical details but the actual solution is to access the references to old classes and update their methods to the methods of the new classes. If that happens automagically all old instances get the new methods and there is no reason for extra steps.
This workflow work before compilation, after compilation and of course the most important during execution. Essentially this accomplishes live coding going the opposite direction of become. It applies for any kind of Repl and any kind of debugger. I have tried python repl and python debugger as well as iPython repl and iPython debugger. No problem so far.
This works also for superclasses and subclasses because Python uses Singleton references wherever it can unless of course instructed otherwise by the coder. So I don’t have to worry about inheritance issues.
I am a bit worried about creating dangling references but this would require some stress testing to inspect how well the garbage collector works but I don’t expect major issues because I am generating new objects with empty state while old state is retained in its place. Of course state can be manipulated lively through live coding as one would expect.
I don’t see any other problem and what is left now is to gather all my tests under a single library. The library will detect when a source file got modified and follow the above workflow.
I would like to thank you all for your assistance I am very happy that accomplished this goal. On Sat, 14 Oct 2017 at 15:29, Bernhard Pieber bernhard@pieber.com wrote:
Hi Dimitris,
This conference article might be relevant:
https://2017.programming-conference.org/event/morevms-2017-papers-when-a-mou...
Cheers, Bernhard
vm-dev@lists.squeakfoundation.org
-
Ben Coman -
Bernhard Pieber -
Bert Freudenberg -
Dimitris Chloupis -
Jan Vrany -
Jecel Assumpcao Jr. -
Nicolai Hess -
Nicolas Cellier -
Tobias Pape