adds the minheadless sources from @ronsaldo + my own changes. this is still WIP (is working for Mac and Linux but not for Win), but to be honest, I want this integrated before the structure reorganisation so I do not need to do it myself *after* ;)
You can view, comment on, or merge this pull request online at:
https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/298
-- Commit Summary --
* Starting to make a build without the windowing stuff. * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into MinimalisticHeadless * I am starting to load a basic image with the minimalistic vm (crashing). * Oops. * I keep doing this refactoring. * The completely headless VM is starting to work in Linux. * Now I am building more plugins. * I am starting to implement a SDL2 based Window backend for compatibility. * The SDL2 based window backend is now usable. * I am starting to compile the VM using Visual Studio in Windows (via projects generated by CMake). * The minimalistic headless VM is starting to work in Windows. * I keep working on making this work on Windows. * Minor change. * I am cleaning the Win32 security plugin unicode mess. * I added the missing files from the previous commit. * Merge branch 'MinimalisticHeadless' of github.com:ronsaldo/opensmalltalk-vm into MinimalisticHeadless * Adding some const to char*. * I am starting to generate the config.h using CMake. * Oops, missing file from the last commit. * I am starting to generate the config.h. Building and working in Linux. * Only use the -pthread option in Unixes. * I managed to get the FFI working back with MinGW. I was missing some flags in the CMake scripts. I still need to get this working with MSVC. * Minor change to the cmake files. * Minor changes for OSX. * Merge branch 'MinimalisticHeadless' of github.com:ronsaldo/opensmalltalk-vm into MinimalisticHeadless * The minheadless VM is now working on OS X. * Minheadless VM is now working on Windows with SDL2. * Oops. * I refactored the windowing system for choosing between the null window system and the SDL2 based system depending wheter the VM was executed in headless mode or not. * I keep working on the SDL2 backend. * Use a SDL2 hint instead of restoring the signal handlers for sigint and sigterm. * I added missing flags to the cmake building scripts. * I implemented legacy file dropping support with the SDL2 backend. * I started to create the initial version of the embedding interface. * I added back some floating point flags to the minheadless vm. * I added back the processor relinquishing in the Windows Minheadless vm. * I fixed a minor bug when using the SDL2 based window and Alt-Gr+Key combination. * Merge branch 'Cog' into MinimalisticHeadless * Merge branch 'MinimalisticHeadless' of github.com:ronsaldo/opensmalltalk-vm into MinimalisticHeadless * I refactored the locations of the files composing the minheadless favlour of the VM. * I removed the old FFI workaround in Windows. * I fixed the VirtualProtect failure in Windows. The size was not being computed in the correct way. * I fixed the errors when calling mprotect with the spur memory manager. * Oops. * I added back the stack trace printing to the Unix version of the Minheadless favlour. * I fixed the stack trace printing in OS X when in the Minheadless favlour. * I added back the stack trace to the Windows version of the Minheadless VM Favlour. * Swapping some lines before mprotect call. * I fixed some compilation errors with the original Windows compilation scripts. I added some convenience scripts for building the different variants. * Merge branch 'MinimalisticHeadless' of github.com:ronsaldo/opensmalltalk-vm into MinimalisticHeadless * I added some missing execution permissions. * Fixing the building of some VM variants when using CMake for the minheadless VM. * Set more lf eol attributes. * I am fixing the build scripts for Cygwin. * Restore the OpenGL context whenever we use a SDL2 renderer in legacy display mode. * Removing the hack for getting the Git version string by using CMake. * Merge branch 'Cog' into MinimalisticHeadless * Adding missing toolchain files. * Cleaning a bit the coding style. Fixing a bug after merging the upstream Cog branch. * Use the osvm prefix instead of squeak for the public interface. * Merge branch 'MinimalisticHeadless' of github.com:ronsaldo/opensmalltalk-vm into MinimalisticHeadless * Fixing some coding style issues. * Merge pull request #1 from ronsaldo/MinimalisticHeadless * makefiles for build minheadless on macOS * add EXTRAVMLIBS * pharo minheadless for macOS (i386 and x86) working * Adding the rest of the Pharo plugins to the cmake building scripts. * Merge pull request #2 from ronsaldo/add-minheadless-vm * pharo minheadless for linux (i386 and x86) threaded hearbeat working * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * fixes * Merge branch 'Cog' into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * update freetype to 2.9.1 * update libSDL to 2.0.7 * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * update paths * Merge branch 'update-libsdl-and-freetype' into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm * Merge branch 'Cog' of github.com:OpenSmalltalk/opensmalltalk-vm into add-minheadless-vm
-- File Changes --
A .clang_complete (9) M .gitattributes (3) A CMakeLists.txt (559) A build.linux32x86/pharo.cog.spur.minheadless/build.assert.itimerheartbeat/mvm (37) A build.linux32x86/pharo.cog.spur.minheadless/build.assert/mvm (50) A build.linux32x86/pharo.cog.spur.minheadless/build.debug.itimerheartbeat/mvm (37) A build.linux32x86/pharo.cog.spur.minheadless/build.itimerheartbeat/mvm (60) A build.linux32x86/pharo.cog.spur.minheadless/build/mvm (51) A build.linux32x86/pharo.cog.spur.minheadless/makeallclean (15) A build.linux32x86/pharo.cog.spur.minheadless/makealldirty (15) A build.linux64x64/pharo.cog.spur.minheadless/build.assert.itimerheartbeat/mvm (37) A build.linux64x64/pharo.cog.spur.minheadless/build.assert/mvm (41) A build.linux64x64/pharo.cog.spur.minheadless/build.debug.itimerheartbeat/mvm (37) A build.linux64x64/pharo.cog.spur.minheadless/build.debug/mvm (41) A build.linux64x64/pharo.cog.spur.minheadless/build.itimerheartbeat/mvm (60) A build.linux64x64/pharo.cog.spur.minheadless/build/mvm (41) A build.linux64x64/pharo.cog.spur.minheadless/makeallclean (15) A build.linux64x64/pharo.cog.spur.minheadless/makealldirty (15) A build.macos32x86/common.minheadless/Makefile.app (190) A build.macos32x86/common.minheadless/Makefile.app.newspeak (29) A build.macos32x86/common.minheadless/Makefile.app.squeak (22) A build.macos32x86/common.minheadless/Makefile.clangversion (26) A build.macos32x86/common.minheadless/Makefile.flags (56) A build.macos32x86/common.minheadless/Makefile.lib.extra (33) A build.macos32x86/common.minheadless/Makefile.plugin (243) A build.macos32x86/common.minheadless/Makefile.rules (34) A build.macos32x86/common.minheadless/Makefile.sources (24) A build.macos32x86/common.minheadless/Makefile.vm (299) A build.macos32x86/common.minheadless/mkInternalPluginsList.sh (11) A build.macos32x86/common.minheadless/mkNamedPrims.sh (24) A build.macos32x86/pharo.cog.spur.minheadless/Makefile (35) A build.macos32x86/pharo.cog.spur.minheadless/mvm (32) A build.macos32x86/pharo.cog.spur.minheadless/plugins.ext (27) A build.macos32x86/pharo.cog.spur.minheadless/plugins.int (28) A build.minheadless.cmake/x64/common/Toolchain-mingw32-cygwin.cmake (21) A build.minheadless.cmake/x64/common/configure_variant.sh (18) A build.minheadless.cmake/x64/pharo.cog.spur/Makefile (18) A build.minheadless.cmake/x64/pharo.cog.spur/mvm (4) A build.minheadless.cmake/x64/pharo.cog.spur/mvm_configure (6) A build.minheadless.cmake/x64/pharo.cog.spur/mvm_configure_variant (4) A build.minheadless.cmake/x64/pharo.stack.spur/Makefile (18) A build.minheadless.cmake/x64/pharo.stack.spur/mvm (4) A build.minheadless.cmake/x64/pharo.stack.spur/mvm_configure (6) A build.minheadless.cmake/x64/pharo.stack.spur/mvm_configure_variant (4) A build.minheadless.cmake/x64/squeak.cog.spur/Makefile (18) A build.minheadless.cmake/x64/squeak.cog.spur/mvm (4) A build.minheadless.cmake/x64/squeak.cog.spur/mvm_configure (6) A build.minheadless.cmake/x64/squeak.cog.spur/mvm_configure_variant (4) A build.minheadless.cmake/x64/squeak.stack.spur/Makefile (18) A build.minheadless.cmake/x64/squeak.stack.spur/mvm (4) A build.minheadless.cmake/x64/squeak.stack.spur/mvm_configure (6) A build.minheadless.cmake/x64/squeak.stack.spur/mvm_configure_variant (4) A build.minheadless.cmake/x86/common/Toolchain-mingw32-cygwin.cmake (21) A build.minheadless.cmake/x86/common/configure_variant.sh (24) A build.minheadless.cmake/x86/pharo.cog.spur/Makefile (18) A build.minheadless.cmake/x86/pharo.cog.spur/mvm (4) A build.minheadless.cmake/x86/pharo.cog.spur/mvm_configure (6) A build.minheadless.cmake/x86/pharo.cog.spur/mvm_configure_variant (4) A build.minheadless.cmake/x86/pharo.stack.spur/Makefile (18) A build.minheadless.cmake/x86/pharo.stack.spur/mvm (4) A build.minheadless.cmake/x86/pharo.stack.spur/mvm_configure (6) A build.minheadless.cmake/x86/pharo.stack.spur/mvm_configure_variant (4) A build.minheadless.cmake/x86/squeak.cog.spur/Makefile (18) A build.minheadless.cmake/x86/squeak.cog.spur/mvm (4) A build.minheadless.cmake/x86/squeak.cog.spur/mvm_configure (6) A build.minheadless.cmake/x86/squeak.cog.spur/mvm_configure_variant (4) A build.minheadless.cmake/x86/squeak.stack.spur/Makefile (18) A build.minheadless.cmake/x86/squeak.stack.spur/mvm (4) A build.minheadless.cmake/x86/squeak.stack.spur/mvm_configure (6) A build.minheadless.cmake/x86/squeak.stack.spur/mvm_configure_variant (4) A cmake/Mpeg3Plugin.cmake (44) A cmake/Plugins.cmake (119) A cmake/PluginsPharo.cmake (59) A include/OpenSmalltalkVM.h (148) M platforms/Cross/vm/sq.h (12) M platforms/Cross/vm/sqAssert.h (43) A platforms/Cross/vm/sqCircularQueue.h (62) A platforms/Cross/vm/sqPath.c (135) A platforms/Cross/vm/sqPath.h (42) A platforms/Cross/vm/sqTextEncoding.c (191) A platforms/Cross/vm/sqTextEncoding.h (39) A platforms/iOS/plugins/FilePlugin/Makefile (3) A platforms/iOS/plugins/FilePlugin/sqUnixFile.c (361) D platforms/iOS/plugins/SecurityPlugin/sqMacSecurity.c (272) A platforms/iOS/plugins/SecurityPlugin/sqUnixSecurity.c (208) A platforms/minheadless/common/English.lproj/Newspeak-Localizable.strings (3) A platforms/minheadless/common/English.lproj/Pharo-Localizable.strings (3) A platforms/minheadless/common/English.lproj/Squeak-Localizable.strings (9) A platforms/minheadless/common/debug.h (6) A platforms/minheadless/common/glibc.h (37) A platforms/minheadless/common/mac-alias.inc (42) A platforms/minheadless/common/sqConfig.h (21) A platforms/minheadless/common/sqEventCommon.c (50) A platforms/minheadless/common/sqEventCommon.h (64) A platforms/minheadless/common/sqExternalPrimitives.c (252) A platforms/minheadless/common/sqExternalPrimitives.c.orig (241) A platforms/minheadless/common/sqInternalPrimitives.c (74) A platforms/minheadless/common/sqMain.c (34) A platforms/minheadless/common/sqNamedPrims.h (1) A platforms/minheadless/common/sqPlatformSpecific.h (43) A platforms/minheadless/common/sqPlatformSpecificCommon.h (71) A platforms/minheadless/common/sqPrinting.c (313) A platforms/minheadless/common/sqVirtualMachineInterface.c (654) A platforms/minheadless/common/sqWindow-Dispatch.c (286) A platforms/minheadless/common/sqWindow-Null.c (271) A platforms/minheadless/common/sqWindow.h (81) A platforms/minheadless/common/sqaio.h (125) A platforms/minheadless/common/version.c (25) A platforms/minheadless/config.h.in (192) A platforms/minheadless/generic/sqPlatformSpecific-Generic.c (270) A platforms/minheadless/generic/sqPlatformSpecific-Generic.h (1) A platforms/minheadless/sdl2-window/sqWindow-SDL2.c (1186) A platforms/minheadless/unix/BlueSistaSqueak.icns (0) A platforms/minheadless/unix/GreenCogSqueak.icns (0) A platforms/minheadless/unix/NewspeakDocuments.icns (0) A platforms/minheadless/unix/NewspeakVirtualMachine.icns (0) A platforms/minheadless/unix/Pharo-Info.plist (538) A platforms/minheadless/unix/Pharo.icns (0) A platforms/minheadless/unix/PharoChanges.icns (0) A platforms/minheadless/unix/PharoImage.icns (0) A platforms/minheadless/unix/PharoSources.icns (0) A platforms/minheadless/unix/Squeak.icns (0) A platforms/minheadless/unix/SqueakChanges.icns (0) A platforms/minheadless/unix/SqueakGeneric.icns (0) A platforms/minheadless/unix/SqueakImage.icns (0) A platforms/minheadless/unix/SqueakPlugin.icns (0) A platforms/minheadless/unix/SqueakProject.icns (0) A platforms/minheadless/unix/SqueakScript.icns (0) A platforms/minheadless/unix/SqueakSources.icns (0) A platforms/minheadless/unix/aioUnix.c (438) A platforms/minheadless/unix/sqPlatformSpecific-Unix.c (698) A platforms/minheadless/unix/sqPlatformSpecific-Unix.h (128) A platforms/minheadless/unix/sqUnixCharConv.c (474) A platforms/minheadless/unix/sqUnixCharConv.h (60) A platforms/minheadless/unix/sqUnixHeartbeat.c (410) A platforms/minheadless/unix/sqUnixMemory.c (369) A platforms/minheadless/unix/sqUnixSpurMemory.c (221) A platforms/minheadless/unix/sqUnixThreads.c (244) A platforms/minheadless/windows/sqGnu.h (189) A platforms/minheadless/windows/sqPlatformSpecific-Win32.c (680) A platforms/minheadless/windows/sqPlatformSpecific-Win32.h (188) A platforms/minheadless/windows/sqWin32.h (19) A platforms/minheadless/windows/sqWin32Alloc.c (207) A platforms/minheadless/windows/sqWin32Alloc.h (32) A platforms/minheadless/windows/sqWin32Backtrace.c (631) A platforms/minheadless/windows/sqWin32Backtrace.h (8) A platforms/minheadless/windows/sqWin32Common.c (42) A platforms/minheadless/windows/sqWin32Directory.c (419) A platforms/minheadless/windows/sqWin32HandleTable.h (127) A platforms/minheadless/windows/sqWin32Heartbeat.c (479) A platforms/minheadless/windows/sqWin32Main.c (43) A platforms/minheadless/windows/sqWin32SpurAlloc.c (292) A platforms/minheadless/windows/sqWin32Threads.c (272) A platforms/minheadless/windows/sqWin32Time.c (149) M platforms/win32/plugins/FilePlugin/sqWin32File.h (1) M platforms/win32/plugins/FilePlugin/sqWin32FilePrims.c (23) M platforms/win32/plugins/SecurityPlugin/sqWin32Security.c (191) M platforms/win32/plugins/SocketPlugin/sqWin32NewNet.c (1) M platforms/win32/vm/sqPlatformSpecific.h (6) M platforms/win32/vm/sqWin32Window.c (10) M spursrc/vm/cogit.h (2) M spursrc/vm/cogitIA32.c (2) M spursrc/vm/cointerp.c (2) M third-party/freetype2.spec (15) M third-party/libsdl2.spec (16)
-- Patch Links --
https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/298.patch https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/298.diff
eliotmiranda commented on this pull request.
Hi Esteban,
this looks good, but there is at least one remaining issue. For example, the changes to platforms/win32/plugins/FilePlugin/sqWin32FilePrims.c eliminate necessary "if (f->isStdioStream) Read/WriteConsole(..." variants. Can you take a pass over platforms/win32/vm and roll back any incidental changes?
Hi @eliotmiranda,
I didn't have time to work on this (and I don't have much time now to be honest, I need to release Pharo7 and there are still too many things open :( ) Specially, because those changes are not mine but from @ronsaldo and I first need to see what he did and what you want to to be able to fix it :P
I think is to merge latest changes to File plugin there? (no idea how hard would it be :P)
I can have a look at platforms/win32/plugins/FilePlugin/sqWin32FilePrims.c and platforms/win32/vm if that helps. And try building the headless version on linux.
@estebanlm pushed 1 commit.
52f258a Merge branch 'Cog' into add-minheadless-vm
When I started making the minheadless vm, I rewrote most of the file primitives to force Unicode usage by manually converting from UTF-8 into UTF-16, and removing the usage of tchar for explicit wchar. I remember that I was having Unicode related issues with spanish accents in directories and folders, and also some differences when compiling with MSVC. In parallel, in the main vm branch I think that there was a parallel refactoring on these primitives, so another pass on it should be needed.
Hi Esteban, Eliot & Ronie,
Ronie, thanks for the history and explanation re Unicode.
Esteban & Ronnie, can I ask what the long term goals are for this code base?
The reason is that it appears to be adding significant complexity to the code base:
It's breaking the existing directory structure, e.g. the two instances of sqUnixCharConv.c are in:
``` ./platforms/minheadless/unix/sqUnixCharConv.c ./platforms/unix/vm/sqUnixCharConv.c ```
It adds 69 file names. Some of these obviously need to be duplicated, e.g. make files are specific to the build type, but other examples include:
- sqUnixCharConv.h is copied unchanged. - Changes the return types of some functions with the same name, e.g. 'int' to 'sqInt'. - sqWin32Heartbeat.c where the entire file is copied so that 1 function can be extended (ioHighResClock()).
And with an admission of being too lazy to figure it out, I'm not sure how to build the Windows minheadless variant of, e.g., pharo.cog.spur. There's:
``` build.linux64x64/pharo.cog.spur.minheadless/build/mvm ```
for linux 64 bit, and the equivalent for linux 32 bit and MacOS 32 bit, but not MacOS 64 bit or Windows. Should:
``` build.minheadless.cmake/x86/pharo.cog.spur/mvm ```
be used?
I can understand the problems of chasing a moving target, but without a commitment to continue working on this it looks like it will increase technical debt.
Or am I missing something?
Thanks, Alistair
I can address the high level concern. I will let the others address the specifics in the code. The high level goal of this code is to enable the VM to not assume a UI, or even a specific UI. This is important for enabling other user interface technologies.
The Windows part does not work yet. However, the goal of this branch is to integrate it now to not lose the work. The Windows support will be added later.
Medium term, it is intended to use this codebase as the foundation for the official PharoVMs (because we want a real headless option, with the responsibility of starting a world from the image, not the VM).
Of course, that does not means it does not needs work on it.
Most of the duplicates and that is because when started, @ronsaldo wanted to keep it as separated as possible from the main sources, as a way to preserve them and being able to modify what he needed.
not sure how to build the Windows minheadless variant of, e.g., pharo.cog.spur.
I documented my experience getting the Windows build working with MSVC here... http://forum.world.st/Minheadless-trial-td5082569.html
After taking a short break from it I was planning to catch it up to Esteban's branch but I hadn't got an opportunity yet. Seeing Esteban integrating it prompted me to plan some time for this at the end of next week, but thats not assured so if you are so inclined to have a go at it, please do.
From memory this gets at least the StackInterpreter doing simple arithmetic
from the command line, which should allow you to test file primitives from the command-line. Esteban mentioned he had a custom Image for a minheadless+GUI. I don't remember the state I got Cog to.
@girba, @estebanlm & @ronsaldo thanks for the explanation. That makes sense, although I think we need to be careful about the duplication.
@bencoman thanks for the write-up, that explains quite a bit that I didn't really understand.
Progress report:
I've successfully built a linux 64 bit cog spur VM and started a standard image with it. Changes include:
- platforms/win32/plugins/FilePlugin/sqWin32FilePrims.c -- Reinstate stdio processing as highlighted by @eliotmiranda -- I also looked at platforms/win32/vm, but the changes there all look fine. - build.linux64x64/pharo.cog.spur.minheadless/build/mvm -- Fix the install directory processing and change it so it doesn't write in the middle of the 32 bit VM - cmake/Plugins.cmake -- Add FileAttributesPlugin - third-party/libsdl2.spec -- fix libSDL2 linux64 so file name
See: https://github.com/akgrant43/opensmalltalk-vm/tree/akg-minheadless-vm
The only issue I've found so far is that right mouse click isn't working.
The MacOS build will still be broken.
Cheers, Alistair
Current status:
- The CI build completes for: -- Windows -- Main Squeak and Pharo linux and Mac builds -- Other linux builds -- Other Mac builds fails in the newspeak tests. I don't think this is related.
I've only tested linux64 at this stage, but given:
- minheadless are (mostly) separate builds. - both Esteban and I have done some ad-hoc testing. - the CI succeeds. - the fact that this code is more susceptible than most to diverging from the main repository (due to code duplication). - Esteban and Doru both seem committed to continuing with this work. - integrating the code now will facilitate others looking at the code, e.g. Ben planned to have a look next week.
it may be worthwhile integrating this now.
What do you think?
Just a reminder, the PR is not this one, but:
https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/310
My Win32 build fails trying to build and install freetype:
-- Installing: /home/alistair/pharo7/pharo-snap/pharo-vm/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/tttables.h -- Installing: /home/alistair/pharo7/pharo-snap/pharo-vm/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/freetype/tttags.h -- Installing: /home/alistair/pharo7/pharo-snap/pharo-vm/opensmalltalk-vm/.thirdparty-cache/windows/i386/include/freetype2/ft2build.h CMake Error at cmake_install.cmake:35 (file): file INSTALL cannot find "/home/alistair/pharo7/pharo-snap/pharo-vm/opensmalltalk-vm/build.win32x86/pharo.cog.spur/build/third-party/freetype-2.9.1/build/include/freetype/config/ftconfig.h".
I've never used cmake before, so am starting to learn about it, but does anyone recognise the above?
Cheers, Alistair
Closed #298.
This has been added through PR https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/310 which builds on this one.
Closing.
vm-dev@lists.squeakfoundation.org
-
akgrant43 -
Ben Coman
-
Eliot Miranda
-
Esteban Lorenzano
-
Ronie Salgado
-
Tudor Girba