How to include aioplugin in pharovm?
See messages below for the reason why.
Phil ---------- Message transféré ---------- De : "David T. Lewis" lewis@mail.msen.com Date : 16 oct. 2014 01:30 Objet : Re: [Pharo-dev] OSProcess in 3.0 À : "Pharo Development List" pharo-dev@lists.pharo.org Cc :
On Tue, Oct 14, 2014 at 09:35:44PM +0200, Thierry Goubier wrote:
Le 14/10/2014 14:15, David T. Lewis a ?crit :
On Tue, Oct 14, 2014 at 07:19:10AM +0200, Thierry Goubier wrote:
Hi Dave,
took me a while to do some testing. It seems there is an issue, but I couldn't find out where. First, the data:
Loading a package from a local gitfiletree repository (get from the command a zipped git archive containing the package) :
4.5.11, 4.6.8: 2132 msec. 4.5.12, 4.6.9: 50847 msec.
I tried to rewrite the main command, remove the closePipes, etc... No change in the runtime, so your right.
Profiling doesn't help much: execution time is mostly idle time (but with a significant increase on PipeableOSProcess commands: goes from invisible to 2700 ms. A user break in the middle shows it delay waiting on output of the external process).
Only hint I could see is that it seems to be linked with the size of the output of the command.
Thierry,
Aha! I am quite sure that you have located the problem. There must be something preventing the OS process from running to completion with a larger amount of output data. It definitely sounds like a bug in my latest update to Pipeable OSProcess.
The strange thing is that it runs to completion (i.e. I got no errors; if any data was missing or corrupted on output, my load would fail); but a lot slower than it should be.
Thank you very much for finding this. I will get it fixed as quickly as I can.
Thanks. I'm unable to help you much, I'm afraid :(
Thierry,
I do not yet have a solution to this, but I have been able to confirm that the performance difference that you see is primarily associated with the AioPlugin not being present in your VM.
When I changed PipeableOSProcess to handle pipe handle closing properly, I used a buffered output stream that is driven by aio events. When data is available to be read from the external OS process, an aio notification pulls the available data into the buffered stream, and when no more data is available, the pipe (file) handles are closed.
If the plugin is not present in the VM, I provided a polling mechanism to read data from the external OS process. Apparently this is very inefficient, and I am sure it could be greatly improved. That is the cause of the extremely slow performance that you are seeing.
So in summary: The enhancement to PipeableOSProcess relies on an aio event driven reader, and if the AIO plugin is missing, the fallback code relies on polling, and is horribly inefficient.
More to follow, and thank you again for identifying the problem.
If you can convice your VM vendor to include AioPlugin in your VM, that would be helpful also ;-)
Dave
Hi Phil,
On Wed, Oct 15, 2014 at 10:11 PM, phil@highoctane.be phil@highoctane.be wrote:
How to include aioplugin in pharovm?
Ask Esteban and other members of the Pharo team. The AioPlugin is already included in the Squeak Cog and Interpreter VMs. Since the build process is different for Pharo (CMake and VMMakerConfigurqation (sp?) based) telling you how my builds work won't help. But if you want you could try and use one of the latest VMs from my website (see the Downloads link) and you should find that'll work as a Pharo VM.
HTH Eliot
See messages below for the reason why.
Phil ---------- Message transféré ---------- De : "David T. Lewis" lewis@mail.msen.com Date : 16 oct. 2014 01:30 Objet : Re: [Pharo-dev] OSProcess in 3.0 À : "Pharo Development List" pharo-dev@lists.pharo.org Cc :
On Tue, Oct 14, 2014 at 09:35:44PM +0200, Thierry Goubier wrote:
Le 14/10/2014 14:15, David T. Lewis a ?crit :
On Tue, Oct 14, 2014 at 07:19:10AM +0200, Thierry Goubier wrote:
Hi Dave,
took me a while to do some testing. It seems there is an issue, but I couldn't find out where. First, the data:
Loading a package from a local gitfiletree repository (get from the command a zipped git archive containing the package) :
4.5.11, 4.6.8: 2132 msec. 4.5.12, 4.6.9: 50847 msec.
I tried to rewrite the main command, remove the closePipes, etc... No change in the runtime, so your right.
Profiling doesn't help much: execution time is mostly idle time (but with a significant increase on PipeableOSProcess commands: goes from invisible to 2700 ms. A user break in the middle shows it delay waiting on output of the external process).
Only hint I could see is that it seems to be linked with the size of
the
output of the command.
Thierry,
Aha! I am quite sure that you have located the problem. There must be something preventing the OS process from running to completion with a larger amount of output data. It definitely sounds like a bug in my latest update to Pipeable OSProcess.
The strange thing is that it runs to completion (i.e. I got no errors; if any data was missing or corrupted on output, my load would fail); but a lot slower than it should be.
Thank you very much for finding this. I will get it fixed as quickly as I can.
Thanks. I'm unable to help you much, I'm afraid :(
Thierry,
I do not yet have a solution to this, but I have been able to confirm that the performance difference that you see is primarily associated with the AioPlugin not being present in your VM.
When I changed PipeableOSProcess to handle pipe handle closing properly, I used a buffered output stream that is driven by aio events. When data is available to be read from the external OS process, an aio notification pulls the available data into the buffered stream, and when no more data is available, the pipe (file) handles are closed.
If the plugin is not present in the VM, I provided a polling mechanism to read data from the external OS process. Apparently this is very inefficient, and I am sure it could be greatly improved. That is the cause of the extremely slow performance that you are seeing.
So in summary: The enhancement to PipeableOSProcess relies on an aio event driven reader, and if the AIO plugin is missing, the fallback code relies on polling, and is horribly inefficient.
More to follow, and thank you again for identifying the problem.
If you can convice your VM vendor to include AioPlugin in your VM, that would be helpful also ;-)
Dave
On 16 Oct 2014, at 21:01, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Phil,
On Wed, Oct 15, 2014 at 10:11 PM, phil@highoctane.be mailto:phil@highoctane.be <phil@highoctane.be mailto:phil@highoctane.be> wrote:
How to include aioplugin in pharovm?
Ask Esteban and other members of the Pharo team. The AioPlugin is already included in the Squeak Cog and Interpreter VMs. Since the build process is different for Pharo (CMake and VMMakerConfigurqation (sp?) based) telling you how my builds work won't help. But if you want you could try and use one of the latest VMs from my website (see the Downloads link) and you should find that'll work as a Pharo VM.
AioPlugin is since always in the mac version of the Pharo VM. I don’t know why is not in the others, but my best guess is because I was not maintaining them at the time and when I took over I didn’t revised the process :P I will see what I can do :)
Esteban
HTH Eliot See messages below for the reason why.
Phil
---------- Message transféré ---------- De : "David T. Lewis" <lewis@mail.msen.com mailto:lewis@mail.msen.com> Date : 16 oct. 2014 01:30 Objet : Re: [Pharo-dev] OSProcess in 3.0 À : "Pharo Development List" <pharo-dev@lists.pharo.org mailto:pharo-dev@lists.pharo.org> Cc :
On Tue, Oct 14, 2014 at 09:35:44PM +0200, Thierry Goubier wrote:
Le 14/10/2014 14:15, David T. Lewis a ?crit :
On Tue, Oct 14, 2014 at 07:19:10AM +0200, Thierry Goubier wrote:
Hi Dave,
took me a while to do some testing. It seems there is an issue, but I couldn't find out where. First, the data:
Loading a package from a local gitfiletree repository (get from the command a zipped git archive containing the package) :
4.5.11, 4.6.8: 2132 msec. 4.5.12, 4.6.9: 50847 msec.
I tried to rewrite the main command, remove the closePipes, etc... No change in the runtime, so your right.
Profiling doesn't help much: execution time is mostly idle time (but with a significant increase on PipeableOSProcess commands: goes from invisible to 2700 ms. A user break in the middle shows it delay waiting on output of the external process).
Only hint I could see is that it seems to be linked with the size of the output of the command.
Thierry,
Aha! I am quite sure that you have located the problem. There must be something preventing the OS process from running to completion with a larger amount of output data. It definitely sounds like a bug in my latest update to Pipeable OSProcess.
The strange thing is that it runs to completion (i.e. I got no errors; if any data was missing or corrupted on output, my load would fail); but a lot slower than it should be.
Thank you very much for finding this. I will get it fixed as quickly as I can.
Thanks. I'm unable to help you much, I'm afraid :(
Thierry,
I do not yet have a solution to this, but I have been able to confirm that the performance difference that you see is primarily associated with the AioPlugin not being present in your VM.
When I changed PipeableOSProcess to handle pipe handle closing properly, I used a buffered output stream that is driven by aio events. When data is available to be read from the external OS process, an aio notification pulls the available data into the buffered stream, and when no more data is available, the pipe (file) handles are closed.
If the plugin is not present in the VM, I provided a polling mechanism to read data from the external OS process. Apparently this is very inefficient, and I am sure it could be greatly improved. That is the cause of the extremely slow performance that you are seeing.
So in summary: The enhancement to PipeableOSProcess relies on an aio event driven reader, and if the AIO plugin is missing, the fallback code relies on polling, and is horribly inefficient.
More to follow, and thank you again for identifying the problem.
If you can convice your VM vendor to include AioPlugin in your VM, that would be helpful also ;-)
Dave
-- best, Eliot
Hi Esteban,
On Fri, Oct 17, 2014 at 12:13 AM, Esteban Lorenzano estebanlm@gmail.com wrote:
On 16 Oct 2014, at 21:01, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Phil,
On Wed, Oct 15, 2014 at 10:11 PM, phil@highoctane.be phil@highoctane.be wrote:
How to include aioplugin in pharovm?
Ask Esteban and other members of the Pharo team. The AioPlugin is already included in the Squeak Cog and Interpreter VMs. Since the build process is different for Pharo (CMake and VMMakerConfigurqation (sp?) based) telling you how my builds work won't help. But if you want you could try and use one of the latest VMs from my website (see the Downloads link) and you should find that'll work as a Pharo VM.
AioPlugin is since always in the mac version of the Pharo VM. I don’t know why is not in the others, but my best guess is because I was not maintaining them at the time and when I took over I didn’t revised the process :P I will see what I can do :)
It isn't relevant for Windows so you only need to add it to the linux builds.
Esteban
HTH Eliot
See messages below for the reason why.
Phil ---------- Message transféré ---------- De : "David T. Lewis" lewis@mail.msen.com Date : 16 oct. 2014 01:30 Objet : Re: [Pharo-dev] OSProcess in 3.0 À : "Pharo Development List" pharo-dev@lists.pharo.org Cc :
On Tue, Oct 14, 2014 at 09:35:44PM +0200, Thierry Goubier wrote:
Le 14/10/2014 14:15, David T. Lewis a ?crit :
On Tue, Oct 14, 2014 at 07:19:10AM +0200, Thierry Goubier wrote:
Hi Dave,
took me a while to do some testing. It seems there is an issue, but I couldn't find out where. First, the data:
Loading a package from a local gitfiletree repository (get from the command a zipped git archive containing the package) :
4.5.11, 4.6.8: 2132 msec. 4.5.12, 4.6.9: 50847 msec.
I tried to rewrite the main command, remove the closePipes, etc... No change in the runtime, so your right.
Profiling doesn't help much: execution time is mostly idle time (but with a significant increase on PipeableOSProcess commands: goes from invisible to 2700 ms. A user break in the middle shows it delay
waiting
on output of the external process).
Only hint I could see is that it seems to be linked with the size of
the
output of the command.
Thierry,
Aha! I am quite sure that you have located the problem. There must be something preventing the OS process from running to completion with a larger amount of output data. It definitely sounds like a bug in my latest update to Pipeable OSProcess.
The strange thing is that it runs to completion (i.e. I got no errors; if any data was missing or corrupted on output, my load would fail); but a lot slower than it should be.
Thank you very much for finding this. I will get it fixed as quickly as I can.
Thanks. I'm unable to help you much, I'm afraid :(
Thierry,
I do not yet have a solution to this, but I have been able to confirm that the performance difference that you see is primarily associated with the AioPlugin not being present in your VM.
When I changed PipeableOSProcess to handle pipe handle closing properly, I used a buffered output stream that is driven by aio events. When data is available to be read from the external OS process, an aio notification pulls the available data into the buffered stream, and when no more data is available, the pipe (file) handles are closed.
If the plugin is not present in the VM, I provided a polling mechanism to read data from the external OS process. Apparently this is very inefficient, and I am sure it could be greatly improved. That is the cause of the extremely slow performance that you are seeing.
So in summary: The enhancement to PipeableOSProcess relies on an aio event driven reader, and if the AIO plugin is missing, the fallback code relies on polling, and is horribly inefficient.
More to follow, and thank you again for identifying the problem.
If you can convice your VM vendor to include AioPlugin in your VM, that would be helpful also ;-)
Dave
-- best, Eliot
vm-dev@lists.squeakfoundation.org
-
Eliot Miranda -
Esteban Lorenzano -
phil＠highoctane.be