[commit][2807] look for plugins.{int,ext,exc} in build, unix/cmake and src; parse plugin lists more robustly
Revision: 2807 Author: piumarta Date: 2013-11-13 19:44:42 -0800 (Wed, 13 Nov 2013) Log Message: ----------- look for plugins.{int,ext,exc} in build, unix/cmake and src; parse plugin lists more robustly
Modified Paths: -------------- trunk/platforms/unix/cmake/Plugins.cmake
Modified: trunk/platforms/unix/cmake/Plugins.cmake =================================================================== --- trunk/platforms/unix/cmake/Plugins.cmake 2013-11-14 02:20:08 UTC (rev 2806) +++ trunk/platforms/unix/cmake/Plugins.cmake 2013-11-14 03:44:42 UTC (rev 2807) @@ -1,38 +1,44 @@ # Figure out which plugins to build and create a configuration for each. # -# Last edited: 2013-11-11 19:12:21 by piumarta on emilia +# Last edited: 2013-11-13 19:43:20 by piumarta on emilia
-IF (EXISTS ${config}/plugins.int) +IF (EXISTS ${bld}/plugins.int) + FILE (STRINGS ${bld}/plugins.int plugins_int) +ELSEIF (EXISTS ${config}/plugins.int) FILE (STRINGS ${config}/plugins.int plugins_int) ELSEIF (EXISTS ${src}/plugins.int) FILE (STRINGS ${src}/plugins.int plugins_int) ELSE (EXISTS ${src}/plugins.int) - MESSAGE (FATAL_ERROR "Cannot find plugins.int in ${src} or ${config}") -ENDIF (EXISTS ${config}/plugins.int) + MESSAGE (FATAL_ERROR "Cannot find plugins.int in ${bld}, ${config} or ${src}") +ENDIF (EXISTS ${bld}/plugins.int)
-STRING (REGEX REPLACE ".*= (.*)" "\1" plugins_int ${plugins_int}) -STRING (REPLACE " " ";" plugins_int ${plugins_int}) +STRING (REGEX REPLACE ".*= *(.*)" "\1" plugins_int ${plugins_int}) +STRING (REPLACE " " ";" plugins_int "${plugins_int}")
-IF (EXISTS ${config}/plugins.ext) +IF (EXISTS ${bld}/plugins.ext) + FILE (STRINGS ${bld}/plugins.ext plugins_ext) +ELSEIF (EXISTS ${config}/plugins.ext) FILE (STRINGS ${config}/plugins.ext plugins_ext) ELSEIF (EXISTS ${src}/plugins.ext) FILE (STRINGS ${src}/plugins.ext plugins_ext) ELSE (EXISTS ${src}/plugins.ext) - MESSAGE (FATAL_ERROR "Cannot find plugins.ext in ${src} or ${config}") -ENDIF (EXISTS ${config}/plugins.ext) + MESSAGE (FATAL_ERROR "Cannot find plugins.ext in ${bld}, ${config} or ${src}") +ENDIF (EXISTS ${bld}/plugins.ext)
-STRING (REGEX REPLACE ".*= (.*)" "\1" plugins_ext ${plugins_ext}) -STRING (REPLACE " " ";" plugins_ext ${plugins_ext}) +STRING (REGEX REPLACE ".*= *(.*)" "\1" plugins_ext ${plugins_ext}) +STRING (REPLACE " " ";" plugins_ext "${plugins_ext}")
+IF (EXISTS ${bld}/plugins.exc) + FILE (STRINGS ${bld}/plugins.exc plugins_exc) IF (EXISTS ${config}/plugins.exc) FILE (STRINGS ${config}/plugins.exc plugins_exc) ELSEIF (EXISTS ${src}/plugins.exc) FILE (STRINGS ${src}/plugins.exc plugins_exc) -ENDIF (EXISTS ${config}/plugins.exc) +ENDIF (EXISTS ${bld}/plugins.exc)
IF (DEFINED plugins_exc) - STRING (REGEX REPLACE ".*= (.*)" "\1" plugins_exc ${plugins_exc}) - STRING (REPLACE " " ";" plugins_exc ${plugins_exc}) + STRING (REGEX REPLACE ".*= *(.*)" "\1" plugins_exc ${plugins_exc}) + STRING (REPLACE " " ";" plugins_exc "${plugins_exc}") FOREACH (plugin ${plugins_exc}) MESSAGE ("!! excluding plugin ${plugin}") LIST (REMOVE_ITEM plugins_int ${plugin})
Hi Tim,
Can you please check if the following makes sense for compiling on Risc OS?
As previously discussed, we are changing the directory layout of generated sources so that all plugins (internal and external) are generated into a single subdirectory. This matches the organization that Eliot is using for Cog builds. VMMaker is already updated to do this by default unless you explicitly set "VMMaker useSinglePluginsDirectory: false".
You may have noticed from the recent string of commit notices that Ian has reworked the Unix build to use the new flattened directory structure. Along with this, we are adopting the conventions of using the plugins.int, plugins.ext, and plugins.exc configuration files to control the build:
On Wed, Nov 13, 2013 at 07:44:43PM -0800, commits@squeakvm.org wrote:
Revision: 2807 Author: piumarta Date: 2013-11-13 19:44:42 -0800 (Wed, 13 Nov 2013) Log Message:
look for plugins.{int,ext,exc} in build, unix/cmake and src; parse plugin lists more robustly
This is intended to be compatible with the Cog build conventions, in which plugins.int/ext/exc are located in the build directory, and are used at compile time to control which plugins are built. This permits a full set of generated sources to be present in the src directory, even if only a subset is used for a particular platform build.
Along with this, there is a search path convention for the plugins.int/ext/exc files:
- If plugins.int/ext/exc exist in the build directory, these are used. This matches the approach for Cog build, and is generally appropriate for a person or organization that wants to build VMs for multiple platforms from a single controlled set of C source files.
- Otherwise, if plugins.int/ext/exc exist in the unix/cmake directory (or presumably in the case of Risc OS, in some location that you might choose in platforms/RiscOC), then these are used. This is the use case of a platform maintainer (e.g. Ian) building official VMs for distribution, where the trunk/src files may have been generated by someone else (e.g. you or me), and he wants to explicitly specify the configuration for the platform build. It may also be appropriate for Linux distribution maintainers.
- Otherwise, use the plugins.int/ext/exc in the src directory. This is the default case, in which VMMaker generates plugins.int and plugins.ext as part of the source generation. This is also the normal case for an individual person (such as myself) building a VM, or using VMMaker to develop a new plugin, debug a VM problem, etc. Very likely it is also the normal case when you build a VM for Risc OS.
I think that most of the above is pretty much self-evident, but I wanted to run it by you in case any of this conflicts with the way in which you build a VM for Risc OS (or more generally, any non-unixish platform).
Thanks, Dave
Thanks for the explanation. It certainly sounds good to me and I'll see what RISC OS vmakes of it when i get home. Getting all systems on the same track will be nice.
When I originally wrote vmmaker I wanted to use the same trick Eliot used to use for brouhaha sources - directories of sub component files with target platform directories consisting of links to the appropriate files. But sadly squeak didn't support links and so I couldn't think of a way to get that back in 2000. Of course most non unix OS didn't do Iinks anyway then, either; I'm not sure they do them properly even now.
Maybe it's a dumb idea but how about having some Svn branches for each platform, so that the right subset of the entire tree can be quickly downloaded without the 'waste' of unneeded files? And we really ought to develop the habit of tagging releases so one can get the right set of files for an older vm - I had great fun early this year trying to work out which revision was a match for the 4.0 vm I needed to build.
/tim {insert witticism here}
On Sat, Nov 16, 2013 at 09:52:39PM -0800, tim Rowledge wrote:
Thanks for the explanation. It certainly sounds good to me and I'll see what RISC OS vmakes of it when i get home. Getting all systems on the same track will be nice.
There is one other issue that is not yet resolved in trunk. We should discuss after you get back and have had a chance to look at Risc OS builds, but just FYI the issue is that in trunk, sqNamedPrims.h is being generated by VMMaker at source generation time, and it needs to match the contents of plugins.int. In Cog this is done with a shell script at build time and the sqNamedPrims.h generation is removed from VMMaker. We will need to do this or something similar in trunk, but I'm not sure how the issue should be handled for non-Unix platforms.
When I originally wrote vmmaker I wanted to use the same trick Eliot used to use for brouhaha sources - directories of sub component files with target platform directories consisting of links to the appropriate files. But sadly squeak didn't support links and so I couldn't think of a way to get that back in 2000. Of course most non unix OS didn't do Iinks anyway then, either; I'm not sure they do them properly even now.
I think this remains a limitation today today because of the wide use of USB sticks formatted with simple file systems that do not understand links.
Maybe it's a dumb idea but how about having some Svn branches for each platform, so that the right subset of the entire tree can be quickly downloaded without the 'waste' of unneeded files? And we really ought to develop the habit of tagging releases so one can get the right set of files for an older vm - I had great fun early this year trying to work out which revision was a match for the 4.0 vm I needed to build.
We can't fix the past, but moving forward this may be less of a concern. Suppose that you have a VM identified as follows:
lewis@linux-jh8m:/tmp/squeak/VMM-20131027/platforms> squeak -version 4.12.8-2812 #1 XShm Fri Nov 15 13:32:00 EST 2013 gcc 4.5.0 Linux linux-jh8m 2.6.34.10-0.6-desktop #1 SMP PREEMPT 2011-12-13 18:27:38 +0100 x86_64 x86_64 x86_64 GNU/Linux plugin path: /usr/local/lib/squeak/4.12.8-2812 [default: /usr/local/lib/squeak/4.12.8-2812/]
And suppose also that you know that the VM was compiled from trunk/platforms and trunk/src (where trunk/src is whatever you and I remembered to check in to SVN after making our VMMaker updates). Then you can check out of full set of source code from Subversion by specifying revision level 2812. This is not as reliable as building from a known source tarball, but it should get you pretty close.
/tim {insert witticism here}
Wow, no witty signature? You must be finding yoursome in some very remote and primitive part of the world. I'm picturing you doing email on a stone tablet with a stick.
Dave
Home At Last! Travel may broaden the mind but it sure wears out the body.
So I want to check on the state of CMakeificiation; I saw all the emails about assorted build related files go by whilst travelling but an up to date summary would be nice. Are we, for example, at a stage where adding the faster bitblt plugin to the Pi stack vm can be done? That after all is my main immediate aim. I’d do a checkout and try to work it out for myself but given that I pretty much threw the towel in for understanding autoCmakeconfigMagic (2.15j) it’ll probably be quicker to ask for an explanation. If I actually understand it I might even be able to provide some doc.
On 17-11-2013, at 7:31 AM, David T. Lewis lewis@mail.msen.com wrote:
/tim {insert witticism here}
Wow, no witty signature? You must be finding yoursome in some very remote and primitive part of the world. I'm picturing you doing email on a stone tablet with a stick.
Almost. Except I was in California and using a new iPad Air (really nice machine). Apple haven’t got around to adding the random choosing of .sig yet for iOS, sadly. Bandwidth seems to be lower in silicon valley than here in rural VI too, which was a surprise. Lots of expensive cars though; Teslas and Porsches everywhere.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- Wise as the world is flat.
On Tue, Nov 19, 2013 at 04:42:56PM -0800, tim Rowledge wrote:
Home At Last! Travel may broaden the mind but it sure wears out the body.
So I want to check on the state of CMakeificiation; I saw all the emails about assorted build related files go by whilst travelling but an up to date summary would be nice. Are we, for example, at a stage where adding the faster bitblt plugin to the Pi stack vm can be done? That after all is my main immediate aim. I?d do a checkout and try to work it out for myself but given that I pretty much threw the towel in for understanding autoCmakeconfigMagic (2.15j) it?ll probably be quicker to ask for an explanation. If I actually understand it I might even be able to provide some doc.
Hi Tim,
I'm not sure I can answer the question regarding the bitblt plugin, but with respect to changes to the build conventions, here is an attempt at a summary of the changes:
- Generated sources now have the plugins in all in a single directory.
- plugins.int, plugins.ext work the same as before, except that now we have a search path convention for finding them in one of possibly three locations.
- plugins.ext may optionally be provided as a means of explicitly excluding plugins from the build. Probably not of interest for a RiscOS build, and somewhat redundant with the "--without-foo" option for Unix CMake, but possibly useful and provided for compatibility with the Cog build.
- Search path for the plugins.int/ext/exc files is bld, config, src.
- bld means your own build directory (in which you may have added a local plugins.int and/or plugins.ext file).
- config means a location in the platforms tree, for Unix this is the platforms/unix/cmake directory. You can ignore this if you do not need it.
- src means the directory containing generated sources from VMMaker. This may be the generated sources that we are now saving in SVN in trunk/src, or it may be the local directory in which you generated your own sources from VMMaker. This is the default location for plugins.int and plugins.ext if not overridden by a file of the same name in bld or config.
- The src directory may contain a full set of generated sources (as would generated from a CrossPlatformVMMaker). The build procedure is expected to select the modules of interest based on plugins.int/ext/exc.
- Alternatively, the src directly may contain just the sources that you generated locally using a VMMakerTool for your own system platform. In either case, the build system should be able to use plugins.int/ext to identify the internal and external plugins.
- The sqNamedPrims.h header was (and currently still is) generated by VMMaker. However, it needs to match the plugins listed in plugins.int, which may now be e.g. a hand-edited file in your build directory. Therefore it must now be generated by the build process. In the case of the Unix CMake build, this is now done by some script code added to platforms/unix/cmake/Platforms.cmake (because CMake is already figuring out which plugins.int file to use). Something similar may be needed for RiscOS if you build from trunk/src.
- VMMaker still generates plugins.int, plugins.ext, and sqNamedPrims.h as before. If you are building from locally generated sources from a VMMakerTool, the only thing that has actually changed is the directory structure in which all plugins are in a single directory.
Dave
Hi Tim,
On Tue, Nov 19, 2013 at 8:09 PM, David T. Lewis lewis@mail.msen.com wrote:
On Tue, Nov 19, 2013 at 04:42:56PM -0800, tim Rowledge wrote:
Home At Last! Travel may broaden the mind but it sure wears out the body.
So I want to check on the state of CMakeificiation; I saw all the emails
about assorted build related files go by whilst travelling but an up to date summary would be nice. Are we, for example, at a stage where adding the faster bitblt plugin to the Pi stack vm can be done? That after all is my main immediate aim. I?d do a checkout and try to work it out for myself but given that I pretty much threw the towel in for understanding autoCmakeconfigMagic (2.15j) it?ll probably be quicker to ask for an explanation. If I actually understand it I might even be able to provide some doc.
Hi Tim,
I'm not sure I can answer the question regarding the bitblt plugin, but with respect to changes to the build conventions, here is an attempt at a summary of the changes:
Generated sources now have the plugins in all in a single directory.
plugins.int, plugins.ext work the same as before, except that now we
have a search path convention for finding them in one of possibly three locations.
- plugins.ext may optionally be provided as a means of explicitly excluding
plugins from the build. Probably not of interest for a RiscOS build, and somewhat redundant with the "--without-foo" option for Unix CMake, but possibly useful and provided for compatibility with the Cog build.
Search path for the plugins.int/ext/exc files is bld, config, src.
bld means your own build directory (in which you may have added
a local plugins.int and/or plugins.ext file).
- config means a location in the platforms tree, for Unix this is
the platforms/unix/cmake directory. You can ignore this if you do not need it.
- src means the directory containing generated sources from VMMaker.
This may be the generated sources that we are now saving in SVN in trunk/src, or it may be the local directory in which you generated your own sources from VMMaker. This is the default location for plugins.int and plugins.ext if not overridden by a file of the same name in bld or config.
- The src directory may contain a full set of generated sources (as
would generated from a CrossPlatformVMMaker). The build procedure is expected to select the modules of interest based on plugins.int/ext/exc.
- Alternatively, the src directly may contain just the sources that
you generated locally using a VMMakerTool for your own system platform. In either case, the build system should be able to use plugins.int/ext to identify the internal and external plugins.
- The sqNamedPrims.h header was (and currently still is) generated by
VMMaker. However, it needs to match the plugins listed in plugins.int, which may now be e.g. a hand-edited file in your build directory. Therefore it must now be generated by the build process. In the case of the Unix CMake build, this is now done by some script code added to platforms/unix/cmake/Platforms.cmake (because CMake is already figuring out which plugins.int file to use). Something similar may be needed for RiscOS if you build from trunk/src.
- VMMaker still generates plugins.int, plugins.ext, and sqNamedPrims.h
as before. If you are building from locally generated sources from a VMMakerTool, the only thing that has actually changed is the directory structure in which all plugins are in a single directory.
Let me ‘splain…No, there is too much. Let me sum up.
Its inflexible to determine which plugins are included at VM generation time. Means different generations for different platforms, etc. Mar simpler to determine it at build time. That way we can use one set of sources (possibly under source code control) and build any VM. Instead, determine which plugins are included, and whether internal or external, at build time. So...
1. generate any and all plugins in one place, e.g. src/plugins. 2. in the build directory have plugins.int and plugins.ext decide which plugins are included and whether they're internal or external. 3. have the build generate sqNamedPrims.h from plugins.int
Hence the base build directory for a unix build on cog contains:
mvm - a script that invokes configure and make install plugins.ext - which plugins to build external plugins.int - which plugins to build internal
HTH
vm-dev@lists.squeakfoundation.org
-
commits＠squeakvm.org -
David T. Lewis -
Eliot Miranda -
tim Rowledge