in my spare time today I was going to run the Sunits for OSProcess under mac carbon 3.8.10b15. However I'm unable to find a version of the plugin anywhere and I'm a bit lazy and don't want to face the hassle of trying to build a plugin from scratch. Therefore if someone has a pre-build plugin for os-x somewhere please email a link or how to obtain it.
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Mon, Feb 27, 2006 at 10:57:45AM -0800, John M McIntosh wrote:
in my spare time today I was going to run the Sunits for OSProcess under mac carbon 3.8.10b15. However I'm unable to find a version of the plugin anywhere and I'm a bit lazy and don't want to face the hassle of trying to build a plugin from scratch. Therefore if someone has a pre-build plugin for os-x somewhere please email a link or how to obtain it.
Hi John,
In case you don't get someone to send a pre-built OSPP:
If you are building a VM with VMMaker, I would expect that adding OSPP to the build should be pretty easy. I'm not familiar with the Mac, but here is more or less what I think you'll need to know:
- There are no external support files or additional headers required.
- Get OSProcessPlugin from SqueakMap. If the carbon VM supports Ian's asynchronous IO routines, then also grab AioPlugin. If X window is of any interest, then also grab XDisplayControlPlugin.
- You will need to put FilePlugin.h and SocketPlugin.h in the build path for OSPP (and for AioPlugin if you build that).
- You will need to include the sqaio.h file from the unix platform sources for both OSPP and AioPlugin, and also define SQAIO_H to point to this file (#define SQAIO_H "sqaio.h"). This is in OSPP only for backward compatibility, but you still need to have it for the build. The #define is also for backward compatibility, as the file used to be named "aio.h" rather than "sqaio.h".
- If you build XDisplayControlPlugin, you'll need to get the X headers and libraries into the build and link paths. I don't know how this would be done on the Mac, but the acinclude.m4 Makefile.inc files in platforms/unix/plugins/XDisplayControlPlugin might provide a hint.
As far as I know, that should be all you need to build the plugins.
When you run the tests, I'd suggest that you load both OSProcess and CommandShell, because CommandShell will add a lot of additional unit tests when you run "OSProcess allTestResults". Get the latest version of both packages, as I recently added a few tweaks to make the tests fail more gracefully on Mac.
The unit tests will write a results file when they are completed, but there is a pretty good chance that they will not get that far if one of the tests hangs up due to pthread signal handling problems (and I think this is quite likely). However, you will also get useful output written to stdout as the tests run, so if you can run the VM from a console window on the Mac, you should be able to see the output there (and capture it to a file). Or course you can also run the unit tests in the usual manner with the test runner.
I do not expect that you will find endianness problems. I do expect that there will be problems related to signal handling and errno handling with pthreads, as I have not done anything in OSPP to accomodate this. Most notably, the Squeak process that handles child process exit status relies on catching the SIGCHLD signal and signaling a Squeak semaphore. This may not work properly in a pthread environment, as the SIGCHLD may not be delivered to the right pthread.
Dave
David thank you the instructions, I was hoping to avoid having to follow all of them. I'll do other things this week, and perhaps revisit the issue in a couple of weeks..
On 28-Feb-06, at 4:01 AM, David T. Lewis wrote:
On Mon, Feb 27, 2006 at 10:57:45AM -0800, John M McIntosh wrote:
in my spare time today I was going to run the Sunits for OSProcess under mac carbon 3.8.10b15. However I'm unable to find a version of the plugin anywhere and I'm a bit lazy and don't want to face the hassle of trying to build a plugin from scratch. Therefore if someone has a pre-build plugin for os-x somewhere please email a link or how to obtain it.
Hi John,
In case you don't get someone to send a pre-built OSPP:
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
OK, let me know if I can be of help.
Dave
On Wed, Mar 01, 2006 at 02:38:55AM -0800, John M McIntosh wrote:
David thank you the instructions, I was hoping to avoid having to follow all of them. I'll do other things this week, and perhaps revisit the issue in a couple of weeks..
On 28-Feb-06, at 4:01 AM, David T. Lewis wrote:
On Mon, Feb 27, 2006 at 10:57:45AM -0800, John M McIntosh wrote:
in my spare time today I was going to run the Sunits for OSProcess under mac carbon 3.8.10b15. However I'm unable to find a version of the plugin anywhere and I'm a bit lazy and don't want to face the hassle of trying to build a plugin from scratch. Therefore if someone has a pre-build plugin for os-x somewhere please email a link or how to obtain it.
Hi John,
In case you don't get someone to send a pre-built OSPP:
ok, after some clues from David I built a mostly broken unix vm, however it was enough to build a OSProcessPlugin as a side effect.
In testing I see that calling primitiveForkSqueakWithoutSigHandler crashes things, I'll suspect forking the carbon vm is a no no.
clues are...
[
squeak: could not load plugin `XDisplayControlPlugin' ioFindExternalFunctionIn(primitiveForkSqueakWithoutSigHandler, 4251344) Switching to process 4134 thread 0x6977] Program received signal: "EXC_BAD_ACCESS".
(gdb) call (int) printAllStacks() SmallInteger 149288092 ProcessorScheduler class>idleProcess 149211076 [] in ProcessorScheduler class>startUp 149211168 [] in BlockContext>newProcess $1 = 259985308 (gdb) bt #0 0x0002bd24 in resume (aProcess=131462172) at /Users/johnmci/ Documents/Squeak3.8.0/src/vm/interp.c:21212 #1 0x0002eb98 in synchronousSignal (aSemaphore=131461696) at /Users/ johnmci/Documents/Squeak3.8.0/src/vm/interp.c:22390 #2 0x00005b38 in checkForInterrupts () at /Users/johnmci/Documents/ Squeak3.8.0/src/vm/interp.c:3134 #3 0x000103fc in interpret () at /Users/johnmci/Documents/ Squeak3.8.0/src/vm/interp.c:8247 #4 0x9002b1e0 in _pthread_body ()
Anyway other things work, geting file permissions, working directory etc etc, and I guess it's enough for folks who want limited support under os-x using the carbon vm at 3.8.10 level. IF anyone wants to fix or alter the sunits so that they won't fork the vm if platform is Mac OS & not unix then feel free...
On 1-Mar-06, at 3:57 AM, David T. Lewis wrote:
OK, let me know if I can be of help.
Dave
On Wed, Mar 01, 2006 at 02:38:55AM -0800, John M McIntosh wrote:
David thank you the instructions, I was hoping to avoid having to follow all of them. I'll do other things this week, and perhaps revisit the issue in a couple of weeks..
On 28-Feb-06, at 4:01 AM, David T. Lewis wrote:
On Mon, Feb 27, 2006 at 10:57:45AM -0800, John M McIntosh wrote:
in my spare time today I was going to run the Sunits for OSProcess under mac carbon 3.8.10b15. However I'm unable to find a version of the plugin anywhere and I'm a bit lazy and don't want to face the hassle of trying to build a plugin from scratch. Therefore if someone has a pre-build plugin for os-x somewhere please email a link or how to obtain it.
Hi John,
In case you don't get someone to send a pre-built OSPP:
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Thu, Mar 02, 2006 at 12:33:02AM -0800, John M McIntosh wrote:
Anyway other things work, geting file permissions, working directory etc etc, and I guess it's enough for folks who want limited support under os-x using the carbon vm at 3.8.10 level. IF anyone wants to fix or alter the sunits so that they won't fork the vm if platform is Mac OS & not unix then feel free...
Probably safer would be to just force a #primitiveFail in UnixOSProcessPlugin>>primitiveForkSqueakWithoutSigHandler, and let the tests fail.
Does an ordinary fork-and-exec (not forkSqueak) work? For example does UnixProcessTestCase>>testRunCommand pass?
Dave
On Thu, Mar 02, 2006 at 12:33:02AM -0800, John M McIntosh wrote:
ok, after some clues from David I built a mostly broken unix vm, however it was enough to build a OSProcessPlugin as a side effect.
In testing I see that calling primitiveForkSqueakWithoutSigHandler crashes things, I'll suspect forking the carbon vm is a no no.
Hi John,
If you have a chance, could you please try something for me? Change the method UnixOSProcessAccessor>>canForwardExternalSignals to answer false, then save your image and restart.
This will replace the event-driven SIGCHLD handler with a polling loop, and it should work around the problem of SIGCHLD being handled in some pthread other than the interpreter thread.
I think this should prevent the VM crashes you were getting, and it may allow you to run external programs with OSProcess. (But I'll be quite surprised if #forkSqueak works, because I have no clue how or if all those pthreads would be restarted in a child Squeak.)
If the #canForwardExternalSignals workaround helps, we could change it in OSProcess to be:
canForwardExternalSignals "Answer true if it is possible to forward OS signals to a Smalltalk Semaphore." ^ OSProcess isUnixMac not
Dave
p.s. I started reading up on pthreads, and I'm recoiling in horror at the thought. There appears to be no way for the interpreter thread to find the identity of other running threads, hence no straightforward way to set the signal masks such that unix signals can be reliably delivered to the interpreter thread (which needs to do a #signalSemaphoreWithIndex:). I can imagine a workaround, but it in't gonna be pretty.
Ok, I'm running a few tests this am, mind perhaps there are no OS-X OSProcessPlugin users out there? however thank you for taking the time to look into this.
First I went in and mangled MacOSProcessAccessor and MacProcess to remove the overrides for access to unix superclass logic then I prim failed the forksqueak prim call, and made the suggested change to UnixOSProcessAccessor>>canForwardExternalSignals
Testing
UnixProcessAccessorTestCase>testIsExecutableForUserInGroup accessor isExecutable: '/tmp/delete.me' forUser: 99999 inGroup: accessor primGetGid fails, but I'll note: -r-xr-x--- 1 johnmci wheel 0 Mar 4 13:47 /tmp/delete.me
testIsReadableForUserInGroup didn't look likely related to above testIsWriteableForUserInGroup didn't look likely related to above
testChDir for this accessor primGetCurrentWorkingDirectory returns /private/tmp on the mac
UnixProcessFileLockingtestCase wants to fork a headless squeak.... They all fail then...
UnixProcessFileLockTestcase They all RUN (GREEN)
UnixProcessTestCase 5 run, 13 error because they all attempt to fork a squeak vm.
Likely I should look at the ancient code for headless squeak, if there is any visible demand?
On 4-Mar-06, at 7:28 AM, David T. Lewis wrote:
On Thu, Mar 02, 2006 at 12:33:02AM -0800, John M McIntosh wrote:
ok, after some clues from David I built a mostly broken unix vm, however it was enough to build a OSProcessPlugin as a side effect.
In testing I see that calling primitiveForkSqueakWithoutSigHandler crashes things, I'll suspect forking the carbon vm is a no no.
Hi John,
If you have a chance, could you please try something for me? Change the method UnixOSProcessAccessor>>canForwardExternalSignals to answer false, then save your image and restart.
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
Thanks John,
A key question is whether forking a simple process works:
"(UnixProcessTestCase selector: #testRunCommand) run"
Forking a running Squeak image is a rather tricky business. By contrast, a fork-and-exec to run an external program is simple, and has a good chance of working as long as we steer clear of the pthreads signal handling for the time being.
Let me know if #testRunCommand passes.
Dave
On Sat, Mar 04, 2006 at 02:22:40PM -0800, John M McIntosh wrote:
Ok, I'm running a few tests this am, mind perhaps there are no OS-X OSProcessPlugin users out there? however thank you for taking the time to look into this.
First I went in and mangled MacOSProcessAccessor and MacProcess to remove the overrides for access to unix superclass logic then I prim failed the forksqueak prim call, and made the suggested change to UnixOSProcessAccessor>>canForwardExternalSignals
Testing
UnixProcessAccessorTestCase>testIsExecutableForUserInGroup accessor isExecutable: '/tmp/delete.me' forUser: 99999 inGroup: accessor primGetGid fails, but I'll note: -r-xr-x--- 1 johnmci wheel 0 Mar 4 13:47 /tmp/delete.me
testIsReadableForUserInGroup didn't look likely related to above testIsWriteableForUserInGroup didn't look likely related to above
testChDir for this accessor primGetCurrentWorkingDirectory returns /private/tmp on the mac
UnixProcessFileLockingtestCase wants to fork a headless squeak.... They all fail then...
UnixProcessFileLockTestcase They all RUN (GREEN)
UnixProcessTestCase 5 run, 13 error because they all attempt to fork a squeak vm.
Likely I should look at the ancient code for headless squeak, if there is any visible demand?
On 4-Mar-06, at 7:28 AM, David T. Lewis wrote:
On Thu, Mar 02, 2006 at 12:33:02AM -0800, John M McIntosh wrote:
ok, after some clues from David I built a mostly broken unix vm, however it was enough to build a OSProcessPlugin as a side effect.
In testing I see that calling primitiveForkSqueakWithoutSigHandler crashes things, I'll suspect forking the carbon vm is a no no.
Hi John,
If you have a chance, could you please try something for me? Change the method UnixOSProcessAccessor>>canForwardExternalSignals to answer false, then save your image and restart.
--
=== John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
Hi John,
I managed to overcome my squeamishness about pthreads long enough to write pthread-aware signal forwarding for OSPP. This is at:
http://www.squeaksource.com/OSProcessPlugin/VMConstruction-Plugins-OSProcessPlugin-dtl.3.mcz
The approach I took is: - OSPP plugin initialization remembers pthread ID of the interpreter. - After calling signal forwarding primitive, OSPP sets signal handler to forward SIGFOO to Squeak semaphore. - On receipt of signal, the handler in OSPP checks to see if it is executing in the context of the interpreter pthread. if in interpreter thread: signal the Squeak semaphore. otherwise: set sigmask to prevent future delivery of SIGFOO to this pthread. resend the signal to the interpreter pthread.
This seems to all work fine on my Linux system (meaning that it does not break anything), and I did some tests to make sure the signal re-sending mechanism works. So the idea would be that on OS X, Squeak would be able to handle signals such as SIGCHLD by having any non-interpreter threads that receive the signal just resend it to the correct interpreter pthread, and also mask the signal so it does not end up going to the wrong pthread next time.
If you try this, you would want to change #canForwardExternalSignals back to true.
I would still like to know if #testRunCommand passes on your current configuration, that's an important data point.
Thanks,
Dave
On Sat, Mar 04, 2006 at 06:18:10PM -0500, David T. Lewis wrote:
Thanks John,
A key question is whether forking a simple process works:
"(UnixProcessTestCase selector: #testRunCommand) run"Forking a running Squeak image is a rather tricky business. By contrast, a fork-and-exec to run an external program is simple, and has a good chance of working as long as we steer clear of the pthreads signal handling for the time being.
Let me know if #testRunCommand passes.
Dave
On Sat, Mar 04, 2006 at 02:22:40PM -0800, John M McIntosh wrote:
Ok, I'm running a few tests this am, mind perhaps there are no OS-X OSProcessPlugin users out there? however thank you for taking the time to look into this.
First I went in and mangled MacOSProcessAccessor and MacProcess to remove the overrides for access to unix superclass logic then I prim failed the forksqueak prim call, and made the suggested change to UnixOSProcessAccessor>>canForwardExternalSignals
Testing
UnixProcessAccessorTestCase>testIsExecutableForUserInGroup accessor isExecutable: '/tmp/delete.me' forUser: 99999 inGroup: accessor primGetGid fails, but I'll note: -r-xr-x--- 1 johnmci wheel 0 Mar 4 13:47 /tmp/delete.me
testIsReadableForUserInGroup didn't look likely related to above testIsWriteableForUserInGroup didn't look likely related to above
testChDir for this accessor primGetCurrentWorkingDirectory returns /private/tmp on the mac
UnixProcessFileLockingtestCase wants to fork a headless squeak.... They all fail then...
UnixProcessFileLockTestcase They all RUN (GREEN)
UnixProcessTestCase 5 run, 13 error because they all attempt to fork a squeak vm.
Likely I should look at the ancient code for headless squeak, if there is any visible demand?
On 4-Mar-06, at 7:28 AM, David T. Lewis wrote:
On Thu, Mar 02, 2006 at 12:33:02AM -0800, John M McIntosh wrote:
ok, after some clues from David I built a mostly broken unix vm, however it was enough to build a OSProcessPlugin as a side effect.
In testing I see that calling primitiveForkSqueakWithoutSigHandler crashes things, I'll suspect forking the carbon vm is a no no.
Hi John,
If you have a chance, could you please try something for me? Change the method UnixOSProcessAccessor>>canForwardExternalSignals to answer false, then save your image and restart.
--
=== John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
Ah, so back to checking this, low interesting (non-existent as far as I can tell) means slow turn around, Mind getting all the plugins to work with MacIntel has been a chore.
So (UnixProcessTestCase selector: #testRunCommand) run does run and does not throw any errors...
However lets see, build debug version of VM with GCC 4.0 oops...
dlopen(/Users/johnmci/Documents/Squeak3.8.0/build/Development/Plugins/ UnixOSProcessPlugin, 10): Symbol not found: _argCnt. Referenced from: /Users/johnmci/Documents/ Squeak3.8.0/build/Development/Plugins/UnixOSProcessPlugin. Expected in: flat namespace
(gdb) call (int) printAllStacks() Process 129010824 UnixOSProcessAccessor>forwardSigChld 129010732 UnixOSProcessAccessor>sigChldSemaphore 129010640 UnixOSProcessAccessor>finalize 129007008 OSProcessAccessor class>startUp: 129006916 ThisOSProcess class>startUp: 126364916 [] in SystemDictionary>send:toClassesNamedIn:with: 126364692 OrderedCollection>do: 126364600 SystemDictionary>send:toClassesNamedIn:with: 126364508 SystemDictionary>processStartUpList: 126227260 SmalltalkImage>snapshot:andQuit:embedded: 126227168 SmalltalkImage>snapshot:andQuit:
Mmm, much reading, turns out in gcc 4.0 xcode decides that - fvisibility=hidden a *new* feature to hide or not export symbols in the linker. Nice. Of course this worked before because I was compiling with GCC3.3 for powerpc during development and GCC 3.3 for Deployment powerpc. However macIntel uses GCC 4.0 so loading the UnixOSProcessPlugin would work on ppc, but not on macintel. Guess it's good I discovered this problem before someone else did.
On 4-Mar-06, at 3:18 PM, David T. Lewis wrote:
(UnixProcessTestCase selector: #testRunCommand) run
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Sun, Mar 12, 2006 at 04:15:06PM -0800, John M McIntosh wrote:
Ah, so back to checking this, low interesting (non-existent as far as I can tell) means slow turn around, Mind getting all the plugins to work with MacIntel has been a chore.
So (UnixProcessTestCase selector: #testRunCommand) run does run and does not throw any errors...
Excellent, that mean that running an external program and getting its exit status works with the basic mechanism of polling for exit status. The event-driven signal forwarding probably is not working yet, but that is the subject of the pthread-aware issue, which may or may not be fixed by the update to OSPP for pthreads.
However lets see, build debug version of VM with GCC 4.0 oops...
dlopen(/Users/johnmci/Documents/Squeak3.8.0/build/Development/Plugins/ UnixOSProcessPlugin, 10): Symbol not found: _argCnt. Referenced from: /Users/johnmci/Documents/ Squeak3.8.0/build/Development/Plugins/UnixOSProcessPlugin. Expected in: flat namespace
This would be from platforms/unix/sqUnixMain.c, which make these available to plugins as follows:
int main(int argc, char **argv, char **envp) { /* Make parameters global for access from plugins */ argCnt= argc; argVec= argv; envVec= envp;
These would not be in the Mac OS sources, hence symbol not found.
Dave
Actually I did add these to the mac carbon source so I can run the unix command line processing, mind I actually added them years ago for OSProcess plugin btw. The fact I got a symbol not found is due to choices made by Apple in the gcc 3.3 to 4.0.1 migration. Usage of a compiler flag solved the problem by un-hiding global symbols.
On 12-Mar-06, at 6:57 PM, David T. Lewis wrote:
int main(int argc, char **argv, char **envp) { /* Make parameters global for access from plugins */ argCnt= argc; argVec= argv; envVec= envp;
These would not be in the Mac OS sources, hence symbol not found.
Dave
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
squeak-dev@lists.squeakfoundation.org
-
David T. Lewis -
John M McIntosh