Hi,
Has anyone tried to build a plugin outside the vm tree?
In a vm build, how do I generate the necessary header files and libraries to be able to build plugins later outside the source tree?
TIA .. Subbu
Hi,
On Tue, Mar 13, 2018 at 8:02 PM, K K Subbu kksubbu.ml@gmail.com wrote:
Hi,
Has anyone tried to build a plugin outside the vm tree?
I've done this with external plugins. The idea was to compile it on the side to a dynamic library (Pharo VM) or a bundle (Squeak VM) and to copy that inside the VM plugin repository afterwards.
Although in general I would recommend to use FFI over VM plugins, unless you need crazy call-backs that do not work with FFI.
In a VM build, how do I generate the necessary header files and libraries to be able to build plugins later outside the source tree?
Not sure about that, some one else may answer...
The plugin I've dealt with were written in Slang and I could generate the C code (.h and .c file) using the same Smalltalk to C compilation process.
TIA .. Subbu
Hi Subbu,
On Tue, Mar 13, 2018 at 12:02 PM, K K Subbu kksubbu.ml@gmail.com wrote:
Hi,
Has anyone tried to build a plugin outside the vm tree?
Not easy, even if possible. But it is straight forward to build an external plugin separate from the VM. e.g. look at how the external plugins are built for each platform (a process driven from plugins.ext) and invoke make in the same way, but specifying your plugin name.
In a vm build, how do I generate the necessary header files and libraries to be able to build plugins later outside the source tree?
Not sure I understand the question. All the include files are in the platforms (platforms/Cross/vm & platforms/Cross/plugins/*/*.h) and relevant source tree. So if you look at the include paths for a build that'll give you the relevant includes. If you're building a plugin you *have* to build different versions for V3 (the original pre-Spur VM), 32-bit Spur and 64-bit Spur. So for example you'll see that src/vm/interp.h spursrc/vm/interp.h and spur64src/vm/interp.h all differ slightly. So, other than writing a plugin-specific source file, e.g. platforms/Cross/plugins/FilePlugin/FilePlugin.h) and platform-and/or-library-specific include files, everything else should be taken from the opensmalltalk-vm source tree.
TIA .. Subbu
HTH _,,,^..^,,,_ best, Eliot
On Thursday 15 March 2018 09:42 AM, Eliot Miranda wrote:
In a vm build, how do I generate the necessary header files and libraries to be able to build plugins later outside the source tree?Not sure I understand the question. All the include files are in the platforms (platforms/Cross/vm & platforms/Cross/plugins/*/*.h) and relevant source tree. So if you look at the include paths for a build that'll give you the relevant includes.
The src tree combines headers needed to build vm from those needed for plugins. Most of the build time is spent on plugins. If the build process is split into four+ stages:
first-stage(buildconfig, $VMDK) -> $VMDK/include/, $VMDK/include/$PLAT, $VMDK/lib/, $VMDK/lib/$PLAT, $VMDK/lib64/, ....
second-stage($VMDK) - update the above with vm-specific files (say sqst.h, sqcog.h sqspur.h, libstvm.a, libcog.a libspur.a etc). $VMDK contents are packaged for developers (say libostvm-dev).
third-stage($VMDK) - for each plugin do build plugin using headers and libraries in $VMDK/. install header and libraries under $VMDK/. This would consume most of the build time, so it could be done in parallel or even outside the source directory at a later date. The build would depend only on files in $VMDK. Each plugin could choose its code structure, platforms and makefiles. Internal plugins would be generated in both obj and DLL while external plugins would be DLLs only.
fourth-stage($VMDK with internal plugins) -> squeakvm binary. Install libostvm-dev and the dependent external plugin packages. Generate multiple headless or GUI launchers and platform-specific packages and main package (say ostvm).
f you're building a plugin you *have* to build different versions for V3 (the original pre-Spur VM), 32-bit Spur and 64-bit Spur. So for example you'll see that src/vm/interp.h spursrc/vm/interp.h and spur64src/vm/interp.h all differ slightly. So, other than writing a plugin-specific source file, e.g. platforms/Cross/plugins/FilePlugin/FilePluginh) and platform-and/or-library-specific include files, everything else should be taken from the opensmalltalk-vm source tree.
I see plugins falling into two types - algorithms (e.g. BitBlt) and device reifiers (e.g. SoundPlugin). Source level coupling crimps device plugin developers. Device plugins tend to be small and the overhead of doing a full build is very high. Also, in embedded or IoT deployments, the life cycle for plugins can be very different from that of the main VM. So it is imperative that they be decoupled from VM build process through an explicit SDK.
Regards .. Subbu
The way it should go for developing plugins is roughly like this (and it's definitely possible details will get missed in this missive)
Download full sources tree. Download and compose VMMaker system. Do a full clean vm build, just to make sure all the config stuff is done locally.
Work on your plugin code in the VMMaker image and/or C code tools. Generate new plugin code from VMMaker when required. This does not involve rebuilding the entire vm tree from VMMaker unless perhaps you are making a plugin that requires total changes to the entire vm. Which would kind of spoil the point of a plugin.
`make` the plugin install the plugin in the appropriate vm related directory, which is a far more complex story than it should be. run your system intended to test the new plugin. Remember that the intent is that you can unload a plugin within a running image and then make use of something in the plugin to reload it.
Rinse and repeat until it works. Test on other platforms as appropriate. Redesign to handle the other platform stuff you didn't know about. Rinse and repeat until you have to release because IPO is upon you and The Product Must Roll!
No need at all for a special set of sources. The entire source tree is tiny by comparison to many projects and you don't need to look at much of it to develop a plugin.
Are there odd corners and gotchas? You bet. Modern platforms have become hives of blathering insanity requiring special this, proprietary that and secret handshakes galore. Does there need to be better documentation? Duh - is there any software project that has even a half-hearted set of doc these days?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful Latin Phrases:- Oblitus sum perpolire clepsydras! = I forgot to polish the clocks!
On Thursday 15 March 2018 11:36 PM, tim Rowledge wrote:
`make` the plugin install the plugin in the appropriate vm related directory, which is a far more complex story than it should be.
Precisely! Making a plugin shouldn't be this hard. When I looked at the build log, a large chunk of build time was taken up in making plugins - one at a time :-(.
Once I install a developer version of Squeak VM, I should be able to make a plugin without having to regenerate the entire multi-platform VM source tree. As a bonus, the developer image can be used as a fallback or bootstrap when no image is given.
Currently, plugin making process is like expecting a linux module developer to install the entire linux kernel source tree just to build a kernel module.
Regards .. Subbu
Hi Subbu,
On Mar 18, 2018, at 11:15 AM, K K Subbu kksubbu.ml@gmail.com wrote:
On Thursday 15 March 2018 11:36 PM, tim Rowledge wrote: `make` the plugin install the plugin in the appropriate vm related directory, which is a far more complex story than it should be.
Precisely! Making a plugin shouldn't be this hard. When I looked at the build log, a large chunk of build time was taken up in making plugins - one at a time :-(.
Once I install a developer version of Squeak VM, I should be able to make a plugin without having to regenerate the entire multi-platform VM source tree. As a bonus, the developer image can be used as a fallback or bootstrap when no image is given.
But you don't have to. You can generate the source of a single plugin from a VMMaker image using the VMMakerTool. This last week I got the tool to open with its fields filled in correctly gif default production.
Currently, plugin making process is like expecting a linux module developer to install the entire linux kernel source tree just to build a kernel module.
We can do better, but it's work. At least the new build makefiles to allow one to make a plugin using a single make invocation. You're welcome to make improvements!
Regards .. Subbu
On Monday 19 March 2018 04:14 AM, Eliot Miranda wrote:
Once I install a developer version of Squeak VM, I should be able to make a plugin without having to regenerate the entire multi-platform VM source tree. As a bonus, the developer image can be used as a fallback or bootstrap when no image is given.
But you don't have to. You can generate the source of a single plugin from a VMMaker image using the VMMakerTool. This last week I got the tool to open with its fields filled in correctly gif default production.
Generating C code is the least of the problem. The complexity is in compiling it to a shared library. Hardware plugins face a much bigger issue than algorithmic plugins because they have to integrate slang generated code with hand-coded and local headers/libs.
Currently, plugin making process is like expecting a linux module developer to install the entire linux kernel source tree just to build a kernel module.
We can do better, but it's work. At least the new build makefiles to allow one to make a plugin using a single make invocation. You're welcome to make improvements!
Understood. I really appreciate all the work that you have been putting it into the VM and responding to queries. I really didn't mean to add to your todo list. I was just trying to get a feel for possible areas of improvements.
Regards .. Subbu
On 19-03-2018, at 4:46 AM, K K Subbu kksubbu.ml@gmail.com wrote: [snip] Generating C code is the least of the problem. The complexity is in compiling it to a shared library. Hardware plugins face a much bigger issue than algorithmic plugins because they have to integrate slang generated code with hand-coded and local headers/libs.
You'll have the complexity of the build system whatever you do within a C/make world. You need the relevant headers and hand-written code and make related files. Nothing is likely to change that much. You edit the hand-written code, generate any Slang code from your VMMaker image, go to the right directory and type make.
What could we do to make that easier?
I guess *if* everything worked properly on all platforms (which it never has and likely never will) we could imagine each plugin directory containing links to every relevant file so that the insane complexity of paths that look like ../../../../../platforms/unix/sysV4/plugins/WibbleWriter could be avoided. That was actually a part of my original plan when I started the VMMaker stuff back in ... 2000. Eek. But linking simply didn't work across platforms back then and I don't honestly imagine it would work neatly now. Andreas insisted on a strange arrangement of files for Window because he didn't want to sort out nmake (which was the windows make equivalent back then and since I had had some very unpleasant experience making it work well enough to build VisualWorks a few years before that I couldn't really argue). RISC OS could do links but only really of directories. Mac couldn't do links at all, just sometihng I can't even remember the name of now, that had to be manually interpreted. Perhaps some git magic might make it work so that all the file appear in the plugin build directcory?
We could conceivably build a tool in Squeak to drive it all a bit more pleasantly, using stuff from OSProcess and CommandTool etc. Platform spread, as so often, is a major cause of work here.
We could certainly improve documentation so it is easier to find out what is needed.
I'd even argue that we would be better off *not* having copies of the generated code publicly available and looking like you can edit them. Generating the files from Slang is not really all that time consuming (even on a Pi) and you don't have to do all of it every time, and we could probably improve detection of when you do.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful Latin Phrases:- Fac me cocleario vomere! = Gag me with a spoon!
On Monday 19 March 2018 11:03 PM, tim Rowledge wrote:
I guess *if* everything worked properly on all platforms (which it never has and likely never will) we could imagine each plugin directory containing links to every relevant file so that the insane complexity of paths that look like ../../../../../platforms/unix/sysV4/plugins/WibbleWriter could be avoided.
You're right. Plugins, particularly those tied to hardware like GPIO on Raspberry Pi, need not be multi-platform. This is all the more reason to decouple their builds from that of the main vm.
I wasn't really advocating for links. I don't think links are essential to decouple *.h and lib files required by plugin build. The early build process just needs to "install" these files under the product directory instead of leaving them scattered all the vm source tree. These extra files can be put into a separate squeakvm-dev package and used only for plugin-dev.
We could conceivably build a tool in Squeak to drive it all a bit more pleasantly, using stuff from OSProcess and CommandTool etc. Platform spread, as so often, is a major cause of work here.
Great idea! PluginMaker! A new vmmaker package could ship with a vmmaker image with all platform-fu and Slang transcoding required for plugin builds. If someone wants to build FooPlugin on a specific platform, they can use it to generate customized *.h files along with the initial fooPlugin.c file.
We could certainly improve documentation so it is easier to find out what is needed.
I would be interested in contributing here. Any suggestions on existing plugins (hardware-related) that can serve as a model or a good starting point in squeak wiki?
I'd even argue that we would be better off *not* having copies of the generated code publicly available and looking like you can edit them. Generating the files from Slang is not really all that time consuming (even on a Pi) and you don't have to do all of it every time, and we could probably improve detection of when you do.
Slang transcoding is easy and convenient, but it requires a working VM+image creating a cyclic dependency between making a VM and making a plugin. Currently, the dependency is resolved by squishing everything into one monolithic, sequential build. I had pursued the idea of modular plugins way back in 2009 but got whacked by Cog transition :-(
http://forum.world.st/Modular-Plugins-td975999.html
Regards .. Subbu
On 20 March 2018 at 01:33, tim Rowledge tim@rowledge.org wrote:
I'd even argue that we would be better off *not* having copies of the generated code publicly available and looking like you can edit them. Generating the files from Slang is not really all that time consuming (even on a Pi) and you don't have to do all of it every time, and we could probably improve detection of when you do.
IIUC, the requirement for versioning generated sources is that some places in the code generation rely on Sets, and so is not completely deterministic. i.e. the same Slang code is not *guaranteed* to produce the same C code. So the C code used to build the VM needs to be versioned to ensure you can work backwards while debugging.
cheers -ben
Hi Tim,
On Mon, Mar 19, 2018 at 10:33 AM, tim Rowledge tim@rowledge.org wrote:
On 19-03-2018, at 4:46 AM, K K Subbu kksubbu.ml@gmail.com wrote: [snip] Generating C code is the least of the problem. The complexity is in
compiling it to a shared library. Hardware plugins face a much bigger issue than algorithmic plugins because they have to integrate slang generated code with hand-coded and local headers/libs.
You'll have the complexity of the build system whatever you do within a C/make world. You need the relevant headers and hand-written code and make related files. Nothing is likely to change that much. You edit the hand-written code, generate any Slang code from your VMMaker image, go to the right directory and type make.
What could we do to make that easier?
I guess *if* everything worked properly on all platforms (which it never has and likely never will) we could imagine each plugin directory containing links to every relevant file so that the insane complexity of paths that look like ../../../../../platforms/unix/sysV4/plugins/WibbleWriter could be avoided. That was actually a part of my original plan when I started the VMMaker stuff back in ... 2000. Eek. But linking simply didn't work across platforms back then and I don't honestly imagine it would work neatly now. Andreas insisted on a strange arrangement of files for Window because he didn't want to sort out nmake (which was the windows make equivalent back then and since I had had some very unpleasant experience making it work well enough to build VisualWorks a few years before that I couldn't really argue). RISC OS could do links but only really of directories. Mac couldn't do links at all, just sometihng I can't even remember the name of now, that had to be manually interpreted. Perhaps some git magic might make it work so that all the file appear in the plugin build directcory?
We could conceivably build a tool in Squeak to drive it all a bit more pleasantly, using stuff from OSProcess and CommandTool etc. Platform spread, as so often, is a major cause of work here.
We could certainly improve documentation so it is easier to find out what is needed.
I'd even argue that we would be better off *not* having copies of the generated code publicly available and looking like you can edit them. Generating the files from Slang is not really all that time consuming (even on a Pi) and you don't have to do all of it every time, and we could probably improve detection of when you do.
Over my dead body. Look, the generated sources all start with comments that say
/* Automatically generated by
they could say
/* Automatically generated by ..... DO NOT EDIT */
But going back to generating the VM sources every time one wants to build a VM, having, as Ben points out the possibility of changes in Slang, variability due to sets changing generated sources, date stamps changing, etc, etc. NO!!!!!!! I can't believe you're serious.
_,,,^..^,,,_ best, Eliot
On Tuesday 20 March 2018 11:21 PM, Eliot Miranda wrote:
Over my dead body. Look, the generated sources all start with comments that say
/* Automatically generated by
they could say
/* Automatically generated by ..... DO NOT EDIT */
Is it necessary to generate human-readable C code? If the code can be filled out on a line completely, it should dissuade people from editing it while keeping the compiler happy. E.g. ------interp.c------------------------------------------- typedef struct _StackPage { char *stackLimit; char *headSP; char *headFP; char *baseFP; char *baseAddress; char *realStackLimit; char *lastAddress; int trace; struct _StackPage *nextPage; struct _StackPage *prevPage; } StackPage; ---------------------------------------------------------
The code can be restored if necessary with indent(1).
Regards .. Subbu
On Tue, Mar 20, 2018 at 11:10 AM, K K Subbu kksubbu.ml@gmail.com wrote:
On Tuesday 20 March 2018 11:21 PM, Eliot Miranda wrote:
Over my dead body. Look, the generated sources all start with comments that say
/* Automatically generated by
they could say
/* Automatically generated by ..... DO NOT EDIT */
Is it necessary to generate human-readable C code?
Very very much. If one wants to debug the C, which is indeed a strong requirement, then having the generated C be readable is essential.
If the code can be filled out on a line completely, it should dissuade people from editing it while keeping the compiler happy. E.g. ------interp.c------------------------------------------- typedef struct _StackPage { char *stackLimit; char *headSP; char *headFP; char *baseFP; char *baseAddress; char *realStackLimit; char *lastAddress; int trace; struct _StackPage *nextPage; struct _StackPage *prevPage; } StackPage;
The code can be restored if necessary with indent(1).
Regards .. Subbu
On 20-03-2018, at 10:51 AM, Eliot Miranda eliot.miranda@gmail.com wrote:
Over my dead body. Look, the generated sources all start with comments that say
/* Automatically generated by
they could say
/* Automatically generated by ..... DO NOT EDIT */
OR we could run the generated code through an obfuscator... wait, what am I saying, this is C code already. An important problem here is that I can pretty much guarantee nobody ever 'see's that comment, nor would take any notice of it if they did. Because code mokeys have all taken off their hoodies, handed over their beer/red bull/mountain dew/whatever and told their buddies "ok, watch *this* dude!"
But going back to generating the VM sources every time one wants to build a VM, having, as Ben points out the possibility of changes in Slang, variability due to sets changing generated sources, date stamps changing, etc, etc. NO!!!!!!! I can't believe you're serious.
Who needs to be serious? Fair point about the variability of details important to other tools though. I can't imagine we couldn't work around all that but as you say, who has the time?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Concurrent: an object that looks like a raisin but isn't
On Tue, Mar 20, 2018 at 11:10 AM, tim Rowledge tim@rowledge.org wrote:
On 20-03-2018, at 10:51 AM, Eliot Miranda eliot.miranda@gmail.com
wrote:
Over my dead body. Look, the generated sources all start with comments
that say
/* Automatically generated by
they could say
/* Automatically generated by ..... DO NOT EDIT */
OR we could run the generated code through an obfuscator... wait, what am I saying, this is C code already. An important problem here is that I can pretty much guarantee nobody ever 'see's that comment, nor would take any notice of it if they did. Because code mokeys have all taken off their hoodies, handed over their beer/red bull/mountain dew/whatever and told their buddies "ok, watch *this* dude!"
But going back to generating the VM sources every time one wants to
build a VM, having, as Ben points out the possibility of changes in Slang, variability due to sets changing generated sources, date stamps changing, etc, etc. NO!!!!!!! I can't believe you're serious.
Who needs to be serious? Fair point about the variability of details important to other tools though. I can't imagine we couldn't work around all that but as you say, who has the time?
That's not the point. The point is that *unless* one versions the generated source code that produced a specific VM, you could *never know for certain* that the source you did generate later on was the same. So for reproducibility and general sanity the generated C is versioned.
_,,,^..^,,,_ best, Eliot
Hi Subbu,
On Thu, Mar 15, 2018 at 12:11 AM, K K Subbu kksubbu.ml@gmail.com wrote:
On Thursday 15 March 2018 09:42 AM, Eliot Miranda wrote:
In a vm build, how do I generate the necessary header files and libraries to be able to build plugins later outside the source tree?Not sure I understand the question. All the include files are in the platforms (platforms/Cross/vm & platforms/Cross/plugins/*/*.h) and relevant source tree. So if you look at the include paths for a build that'll give you the relevant includes.
Are you proposing we support this VMDK idea? Ain't going to happen. No cycles. If I'm misunderstanding then what is VMDK? And what are sqst,h sqcog.h sqspur.h? Look if you're proposing to evolve the build system then you're going to have to drive that process and you'll get push back because - what we have works - most of us are busy developing within that framework and busy developing useful things - rearranging things to make things conceptually clearer for newbies is a laudable goal but expensive and has a very high opportunity cost
The src tree combines headers needed to build vm from those needed for plugins. Most of the build time is spent on plugins. If the build process is split into four+ stages:
first-stage(buildconfig, $VMDK) -> $VMDK/include/, $VMDK/include/$PLAT, $VMDK/lib/, $VMDK/lib/$PLAT, $VMDK/lib64/, ....
second-stage($VMDK) - update the above with vm-specific files (say sqst.h, sqcog.h sqspur.h, libstvm.a, libcog.a libspur.a etc). $VMDK contents are packaged for developers (say libostvm-dev).
third-stage($VMDK) - for each plugin do build plugin using headers and libraries in $VMDK/. install header and libraries under $VMDK/. This would consume most of the build time, so it could be done in parallel or even outside the source directory at a later date. The build would depend only on files in $VMDK. Each plugin could choose its code structure, platforms and makefiles. Internal plugins would be generated in both obj and DLL while external plugins would be DLLs only.
fourth-stage($VMDK with internal plugins) -> squeakvm binary. Install libostvm-dev and the dependent external plugin packages. Generate multiple headless or GUI launchers and platform-specific packages and main package (say ostvm).
f you're building a plugin you *have* to build different versions for V3
(the original pre-Spur VM), 32-bit Spur and 64-bit Spur. So for example you'll see that src/vm/interp.h spursrc/vm/interp.h and spur64src/vm/interp.h all differ slightly. So, other than writing a plugin-specific source file, e.g. platforms/Cross/plugins/FilePlugin/FilePluginh) and platform-and/or-library-specific include files, everything else should be taken from the opensmalltalk-vm source tree.
I see plugins falling into two types - algorithms (e.g. BitBlt) and device reifiers (e.g. SoundPlugin). Source level coupling crimps device plugin developers. Device plugins tend to be small and the overhead of doing a full build is very high. Also, in embedded or IoT deployments, the life cycle for plugins can be very different from that of the main VM. So it is imperative that they be decoupled from VM build process through an explicit SDK.
Regards .. Subbu
On Friday 16 March 2018 12:48 AM, Eliot Miranda wrote:
Are you proposing we support this VMDK idea? Ain't going to happen. No cycles. If I'm misunderstanding then what is VMDK?
$VMDK is a staging directory for final targets. For cog+spur on linux:
VMDK=opensmalltalk-vm/image/cogspurlinuxht
The current build places only the squeak executable in $VMDK/bin and the external plugins in $VMDK/lib/$VERSION etc.
When a plugin is translated from .st to .c, a set of header files are automatically included in it: #include "sqVirtualMachine.h" #include "sqConfig.h" #include "sqPlatformSpecific.h" #include "sqMemoryAccess.h" ...
These files are spread throughout the vm source tree. Instead, if we gather such header files and libraries under appropriate subdirs in $VMDK early in the build, then plugins development can be decoupled from the vm source tree.
The motivation for coding plugins in ST instead of C was "coding shouldn't be this hard". My proposal has the same motivation - "building plugins shouldn't be this hard" ;-).
Regards .. Subbu
vm-dev@lists.squeakfoundation.org
-
Ben Coman -
Clément Bera -
Eliot Miranda -
K K Subbu -
tim Rowledge