Hey folks,
We are working on a plugin using the unix VM for squeak under OSX Tiger. I'd like to have the resulting squeak VM link in some foundation stuff from OSX (CoreFoundation CoreServices...). Is there a way to trigger this using the Makefile.in stuff (tried altering XLDFLAGS, PLIBS, LIBS with no success).
It would be nice to have this linkage only occur if adding in our plugin.
Thanks in advance, Alan
Am 21.12.2005 um 12:35 schrieb Alan Sibley:
Hey folks,
We are working on a plugin using the unix VM for squeak under OSX Tiger. I'd like to have the resulting squeak VM link in some foundation stuff from OSX (CoreFoundation CoreServices...). Is there a way to trigger this using the Makefile.in stuff (tried altering XLDFLAGS, PLIBS, LIBS with no success).
It would be nice to have this linkage only occur if adding in our plugin.
Use the source, Luke:
bert.vektor platforms/unix/vm-display-Quartz % cat Makefile.inc XCPPFLAGS= -I$(topdir)/platforms/Cross/plugins/ B3DAcceleratorPlugin \ -I$(topdir)/platforms/unix/plugins/ B3DAcceleratorPlugin \ -framework Cocoa -framework OpenGL
PLIBS= -Wl,-framework -Wl,Cocoa \ -Wl,-framework -Wl,OpenGL
- Bert -
Hmmm... when I try that... "ar" complains mightily
ar: -Wl,-framework: No such file or directory ar: -Wl,Cocoa:.....
when vm-display-Quartz compiles and links... it doesn't look to be using ar. I'm not the "makefile jockey" to quite determine why that is :(
Thanks for you patience...
On Dec 21, 2005, at 7:26 AM, Bert Freudenberg wrote:
Am 21.12.2005 um 12:35 schrieb Alan Sibley:
Hey folks,
We are working on a plugin using the unix VM for squeak under OSX Tiger. I'd like to have the resulting squeak VM link in some foundation stuff from OSX (CoreFoundation CoreServices...). Is there a way to trigger this using the Makefile.in stuff (tried altering XLDFLAGS, PLIBS, LIBS with no success).
It would be nice to have this linkage only occur if adding in our plugin.
Use the source, Luke:
bert.vektor platforms/unix/vm-display-Quartz % cat Makefile.inc XCPPFLAGS= -I$(topdir)/platforms/Cross/plugins/ B3DAcceleratorPlugin \ -I$(topdir)/platforms/unix/plugins/ B3DAcceleratorPlugin \ -framework Cocoa -framework OpenGL
PLIBS= -Wl,-framework -Wl,Cocoa \ -Wl,-framework -Wl,OpenGL
- Bert -
I am not sure why ar would be involved here ... can you post the commands issued by make?
This is an external module, right?
- Bert -
Am 21.12.2005 um 16:18 schrieb Alan Sibley:
Hmmm... when I try that... "ar" complains mightily
ar: -Wl,-framework: No such file or directory ar: -Wl,Cocoa:.....
when vm-display-Quartz compiles and links... it doesn't look to be using ar. I'm not the "makefile jockey" to quite determine why that is :(
Thanks for you patience...
On Dec 21, 2005, at 7:26 AM, Bert Freudenberg wrote:
Am 21.12.2005 um 12:35 schrieb Alan Sibley:
Hey folks,
We are working on a plugin using the unix VM for squeak under OSX Tiger. I'd like to have the resulting squeak VM link in some foundation stuff from OSX (CoreFoundation CoreServices...). Is there a way to trigger this using the Makefile.in stuff (tried altering XLDFLAGS, PLIBS, LIBS with no success).
It would be nice to have this linkage only occur if adding in our plugin.
Use the source, Luke:
bert.vektor platforms/unix/vm-display-Quartz % cat Makefile.inc XCPPFLAGS= -I$(topdir)/platforms/Cross/plugins/ B3DAcceleratorPlugin \ -I$(topdir)/platforms/unix/plugins/ B3DAcceleratorPlugin \ -framework Cocoa -framework OpenGL
PLIBS= -Wl,-framework -Wl,Cocoa \ -Wl,-framework -Wl,OpenGL
- Bert -
I am building an internal plugin... I can switch to external, it would be nice for it to work either way. I will a) try external and let you know and b) paste in the link command from the internal makefile.
Thanks Alan
On Wednesday, December 21, 2005, at 07:46AM, Bert Freudenberg bert@impara.de wrote:
I am not sure why ar would be involved here ... can you post the commands issued by make?
This is an external module, right?
- Bert -
Am 21.12.2005 um 16:18 schrieb Alan Sibley:
Hmmm... when I try that... "ar" complains mightily
ar: -Wl,-framework: No such file or directory ar: -Wl,Cocoa:.....
when vm-display-Quartz compiles and links... it doesn't look to be using ar. I'm not the "makefile jockey" to quite determine why that is :(
Thanks for you patience...
On Dec 21, 2005, at 7:26 AM, Bert Freudenberg wrote:
Am 21.12.2005 um 12:35 schrieb Alan Sibley:
Hey folks,
We are working on a plugin using the unix VM for squeak under OSX Tiger. I'd like to have the resulting squeak VM link in some foundation stuff from OSX (CoreFoundation CoreServices...). Is there a way to trigger this using the Makefile.in stuff (tried altering XLDFLAGS, PLIBS, LIBS with no success).
It would be nice to have this linkage only occur if adding in our plugin.
Use the source, Luke:
bert.vektor platforms/unix/vm-display-Quartz % cat Makefile.inc XCPPFLAGS= -I$(topdir)/platforms/Cross/plugins/ B3DAcceleratorPlugin \ -I$(topdir)/platforms/unix/plugins/ B3DAcceleratorPlugin \ -framework Cocoa -framework OpenGL
PLIBS= -Wl,-framework -Wl,Cocoa \ -Wl,-framework -Wl,OpenGL
- Bert -
External should be a lot easier to deal with. When building internally, the VM binary itself would have to be linked with the new frameworks. Not sure if the current build-infrastructure supports this.
- Bert -
Am 21.12.2005 um 18:36 schrieb Alan Sibley:
I am building an internal plugin... I can switch to external, it would be nice for it to work either way. I will a) try external and let you know and b) paste in the link command from the internal makefile.
Thanks Alan
On Wednesday, December 21, 2005, at 07:46AM, Bert Freudenberg bert@impara.de wrote:
I am not sure why ar would be involved here ... can you post the commands issued by make?
This is an external module, right?
- Bert -
Am 21.12.2005 um 16:18 schrieb Alan Sibley:
Hmmm... when I try that... "ar" complains mightily
ar: -Wl,-framework: No such file or directory ar: -Wl,Cocoa:.....
when vm-display-Quartz compiles and links... it doesn't look to be using ar. I'm not the "makefile jockey" to quite determine why that is :(
Thanks for you patience...
On Dec 21, 2005, at 7:26 AM, Bert Freudenberg wrote:
Am 21.12.2005 um 12:35 schrieb Alan Sibley:
Hey folks,
We are working on a plugin using the unix VM for squeak under OSX Tiger. I'd like to have the resulting squeak VM link in some foundation stuff from OSX (CoreFoundation CoreServices...). Is there a way to trigger this using the Makefile.in stuff (tried altering XLDFLAGS, PLIBS, LIBS with no success).
It would be nice to have this linkage only occur if adding in our plugin.
Use the source, Luke:
bert.vektor platforms/unix/vm-display-Quartz % cat Makefile.inc XCPPFLAGS= -I$(topdir)/platforms/Cross/plugins/ B3DAcceleratorPlugin \ -I$(topdir)/platforms/unix/plugins/ B3DAcceleratorPlugin \ -framework Cocoa -framework OpenGL
PLIBS= -Wl,-framework -Wl,Cocoa \ -Wl,-framework -Wl,OpenGL
- Bert -
On Dec 21, 2005, at 9:46 AM, Bert Freudenberg wrote:
External should be a lot easier to deal with. When building internally, the VM binary itself would have to be linked with the new frameworks. Not sure if the current build-infrastructure supports this.
No. (Sorry. It's a 'to-do' item.)
Cheers, Ian
I'm feeling less stupid now ;) We will be external 5 minutes after I get home tonight!
I have a suggested modification for mkmf... it doesn't autodetect .cpp files or .M (objc++)... care for my patch?
On Wednesday, December 21, 2005, at 10:49AM, Ian Piumarta ian.piumarta@inria.fr wrote:
On Dec 21, 2005, at 9:46 AM, Bert Freudenberg wrote:
External should be a lot easier to deal with. When building internally, the VM binary itself would have to be linked with the new frameworks. Not sure if the current build-infrastructure supports this.
No. (Sorry. It's a 'to-do' item.)
Cheers, Ian
On 21-Dec-05, at 9:36 AM, Alan Sibley wrote:
I am building an internal plugin... I can switch to external, it would be nice for it to work either way. I will a) try external and let you know and b) paste in the link command from the internal makefile.
Unless you have some particular need for the plugin to be internal (and the only one I can imagine is a need to distribute a single file executable because your users ares are inexperienced) then an external is much easier to live with *especially* during development. a) you can compile it individually b) you can load it to test c) you can unload it d) you can regenerate and recompile it to fix the bug you found e) you can reload it f) see b)
Note that barring truly nasty gotchas -that I can't recall ever hearing about- there is nothing to stop you developing the plugin as an external and then finally generating it as internal for the last build before shipping. All the code in the 'normal' plugins works either way except in a couple of cases where someone has decided to force a fixed dependency between the core vm and a plugin - the security plugin on windows comes to mind. I don't know how unices do things like linking large libraries to dynamically loaded libraries (which is what external plugins are) but I'd be astonished if you couldn't do it. On RISC OS for example the socket libs and so on are linked only to the socket plugin, not to the core vm.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: CSD: Charge Surreptitiously to DOE
Almost in time for Nigelmass (my brother's dirthday is on the 25th) I've just completed another step in the releasing of the 64bit clean VM code with the latest SVN level (1282 or thereabouts) and the SM package VMMaker3.8b5-64 'WillYouStillNeedMe'.
Read the comments attached to the SM package details. You must do certain things before it will work. Certain things will probably not work even then. So far it builds, compiles and runs in 32 bit form on RISC OS and OSX.
You will need to decide upon a way to appropriately define 'VMENDIANNESS' (0 for little, 1 for big) as part of the compile. You might need to pay some attention to the definition of the sqFilenameFromString macro.
When we can feel confident that it is ok for a beta release I can make an SVN branch to tie to the package.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Computer Science: solving today's problems tomorrow.
Hi Tim -
This looks actually pretty good - I was finally able to compile a working VM again (yay!). One problem I had was with the definition and use of the sqFilenamefromblabla macro - ioFilenamefromblabla (the only user of that macro) is defined as taking sqInt but the macro makes happy use of [] which makes the compiler complain that the "subscripted value is neither array nor pointer". This should either be cast properly or declared to take pointer arguments.
Cheers, - Andreas
tim Rowledge wrote:
Almost in time for Nigelmass (my brother's dirthday is on the 25th) I've just completed another step in the releasing of the 64bit clean VM code with the latest SVN level (1282 or thereabouts) and the SM package VMMaker3.8b5-64 'WillYouStillNeedMe'.
Read the comments attached to the SM package details. You must do certain things before it will work. Certain things will probably not work even then. So far it builds, compiles and runs in 32 bit form on RISC OS and OSX.
You will need to decide upon a way to appropriately define 'VMENDIANNESS' (0 for little, 1 for big) as part of the compile. You might need to pay some attention to the definition of the sqFilenameFromString macro.
When we can feel confident that it is ok for a beta release I can make an SVN branch to tie to the package.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Computer Science: solving today's problems tomorrow.
On 28-Dec-05, at 6:19 PM, Andreas Raab wrote:
problem I had was with the definition and use of the sqFilenamefromblabla macro - ioFilenamefromblabla (the only user of that macro) is defined as taking sqInt but the macro makes happy use of [] which makes the compiler complain that the "subscripted value is neither array nor pointer". This should either be cast properly or declared to take pointer arguments.
Yah, the macro is a bit crufty. We should probably junk it and just agree to have a suitable function for all platforms. I think we actually *do* in fact, so it would merely involve agreeing to hook it up neatly everywhere. The original use for the macro was swapping dots with slashes for RISC OS but the requirements got a bit more complex and I had to implement a fairly complicated function to canonicalise filenames. It turns out I wasn't even using the macro anymore, which is just as well since I had defined it with the arguments in the wrong order...
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: HBT: Harvest Binary Tree
I say, let's kick the macro then and give that function a proper prototype. The way it is right now is really not all that useful.
Cheers, - Andreas
tim Rowledge wrote:
On 28-Dec-05, at 6:19 PM, Andreas Raab wrote:
problem I had was with the definition and use of the sqFilenamefromblabla macro - ioFilenamefromblabla (the only user of that macro) is defined as taking sqInt but the macro makes happy use of [] which makes the compiler complain that the "subscripted value is neither array nor pointer". This should either be cast properly or declared to take pointer arguments.
Yah, the macro is a bit crufty. We should probably junk it and just agree to have a suitable function for all platforms. I think we actually *do* in fact, so it would merely involve agreeing to hook it up neatly everywhere. The original use for the macro was swapping dots with slashes for RISC OS but the requirements got a bit more complex and I had to implement a fairly complicated function to canonicalise filenames. It turns out I wasn't even using the macro anymore, which is just as well since I had defined it with the arguments in the wrong order...
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: HBT: Harvest Binary Tree
On 28-Dec-05, at 6:53 PM, Andreas Raab wrote:
I say, let's kick the macro then and give that function a proper prototype. The way it is right now is really not all that useful.
Sigh. It seems it isn't as simple as one would like here. It's - again - OSX causing trouble with its demented approach to aliases. Apparently we need to be able to specify whether the aliases potentially in the filepath get resolved or not; when opening a file we do, for renaming or deleting we don't. So far as I can tell only OSX has to do this but it means that any code hoping to work across platforms has to know when to do 'filenameMungerWithOpen' as opposed to plain 'filenameMunger' if it hopes to run properly on OSX.
Right now all I can think of is either a) making a a proxy function with 4 args - say ioFilenamefromStringofLengthresolveAlias(char* cBuffer, char*filename, sqInt fnSize, sqInt resolveFlag) - and two macros b) or two proxy calls - ioFilenamefromStringofLength & ioFilenameOpenfromStringofLength
Ugly! Ugly! Clearer thoughts needed!
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: L: Lie!
I vote for a) - I doubt anybody would ever bother to remember that there are two variants (I sure wouldn't ;-) And there might be reasons to request alias resolution for other purposes too.
Cheers, - Andreas
tim Rowledge wrote:
On 28-Dec-05, at 6:53 PM, Andreas Raab wrote:
I say, let's kick the macro then and give that function a proper prototype. The way it is right now is really not all that useful.
Sigh. It seems it isn't as simple as one would like here. It's - again
- OSX causing trouble with its demented approach to aliases. Apparently
we need to be able to specify whether the aliases potentially in the filepath get resolved or not; when opening a file we do, for renaming or deleting we don't. So far as I can tell only OSX has to do this but it means that any code hoping to work across platforms has to know when to do 'filenameMungerWithOpen' as opposed to plain 'filenameMunger' if it hopes to run properly on OSX.
Right now all I can think of is either a) making a a proxy function with 4 args - say ioFilenamefromStringofLengthresolveAlias(char* cBuffer, char*filename, sqInt fnSize, sqInt resolveFlag) - and two macros b) or two proxy calls - ioFilenamefromStringofLength & ioFilenameOpenfromStringofLength
Ugly! Ugly! Clearer thoughts needed!
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: L: Lie!
On 29-Dec-05, at 1:37 PM, Andreas Raab wrote:
I vote for a) - I doubt anybody would ever bother to remember that there are two variants (I sure wouldn't ;-) And there might be reasons to request alias resolution for other purposes too.
Problem is that you *have* to remember there are two variants in any hopefully-portable code. Stupid, eh? Actually if we had the 4-arg function it would serve as a constant reminder of that so perhaps doing that and no reality-hiding-convenience macros would be best.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim People who deal with bits should expect to get bitten.
On Tue, Dec 27, 2005 at 09:28:40PM -0800, tim Rowledge wrote:
Almost in time for Nigelmass (my brother's dirthday is on the 25th) I've just completed another step in the releasing of the 64bit clean VM code with the latest SVN level (1282 or thereabouts) and the SM package VMMaker3.8b5-64 'WillYouStillNeedMe'.
Read the comments attached to the SM package details. You must do certain things before it will work. Certain things will probably not work even then. So far it builds, compiles and runs in 32 bit form on RISC OS and OSX.
Worked fine on Intel Linux 32 bit. That's all I've tried so far.
You will need to decide upon a way to appropriately define 'VMENDIANNESS' (0 for little, 1 for big) as part of the compile. You might need to pay some attention to the definition of the sqFilenameFromString macro.
The VMENDIANNESS macro should not be necessary, since endianness can be tested at runtime. Change sets attached.
Interpreter-isBigEnder-dtl.1.cs provides the runtime check.
EndianPlugin-dtl.1.cs is a plugin to verify that it works.
This has been tested only on an Intel 32 bit box, so it needs at least a check on a Mac. To test, build the EndianPlugin, then evaluate "EndianPlugin endianness = Smalltalk endianness", which should evaluate to true.
Cross posted to Mantis #2404.
Dave
On 28-Dec-05, at 6:36 PM, David T. Lewis wrote:
Worked fine on Intel Linux 32 bit. That's all I've tried so far.
Excellent, thanks for testing it Dave.
You will need to decide upon a way to appropriately define 'VMENDIANNESS' (0 for little, 1 for big) as part of the compile. You might need to pay some attention to the definition of the sqFilenameFromString macro.
The VMENDIANNESS macro should not be necessary, since endianness can be tested at runtime. Change sets attached.
Cute! It's fractionally slower at runtime in return for slightly simpler make/autoconf setup. Does anyone have a strong opinion either way?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim If it was easy, the hardware people would take care of it.
The VMENDIANNESS macro should not be necessary, since endianness can be tested at runtime. Change sets attached.
Cute! It's fractionally slower at runtime in return for slightly simpler make/autoconf setup. Does anyone have a strong opinion either way?
Nope. And I don't think it matters much.
Cheers, - Andreas
On Wed, Dec 28, 2005 at 06:44:48PM -0800, tim Rowledge wrote:
On 28-Dec-05, at 6:36 PM, David T. Lewis wrote:
The VMENDIANNESS macro should not be necessary, since endianness can be tested at runtime. Change sets attached.
Cute! It's fractionally slower at runtime in return for slightly simpler make/autoconf setup. Does anyone have a strong opinion either way?
Well, it's a bit simpler in BCPL (ref. Martin Richards' bcpl source):
bigender := (!"AAA" & 255) = 'A' // =TRUE if running on a bigender
The slang version handles 64 bit/32 bit word size, and caches the result so it may not be much slower.
Dave
On Wed, Dec 28, 2005 at 06:44:48PM -0800, tim Rowledge wrote:
On 28-Dec-05, at 6:36 PM, David T. Lewis wrote:
Worked fine on Intel Linux 32 bit. That's all I've tried so far.
Excellent, thanks for testing it Dave.
Interim update on building 64 bit VMs with the latest VMM/SVN:
Unfortunately I have not been able to build a working 64 VM yet. I'm using VMMaker-tpr.39 (loaded from Squeak Map), SVN 1282, plus my local patches (attached for reference). The OS is 32 bit Intel Linux.
I was previously able to build a 64 bit VM with VMMaker-tpr.37 and SVN 1259, with the same local patches.
I am building the 64 bit VM with internal plugins BalloonEnginePlugin, BitBltSimulation, FilePlugin, and SocketPlugin, and no external plugins. The resulting VM crashes with a segfault after entering the interpreter and opening the display. This is using a 64 bit image that does open successfully with my earlier 64 bit VM (VMMaker-tpr.37/SVN 1259).
Also note, VMMaker versionString => '3.8b4' should be '3.8b5-64'.
More to follow when I get it figured out.
Dave
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
platforms-Cross-vm-sqMemoryAccess.h.diff - Add missing memory access macros (the inline functions are complete, but some of the corresponding macros are missing).
platforms-unix-vm-sqUnixExternalPrims.c.diff - Provide meaningful console error message if an external module (e.g. vm-display-X11) fails to load.
platforms-unix-config-config.h.in.diff - Add a definition to unix/config/config.in to #define SQAIO_H "sqaio.h" due to renaming aio.h to sqaio.h. Permits OSPP and AIO plugins to be backward compatible with older source trees. (Note to Tim: RiscOS uses a copy of aio.h in its platform tree, should sqaio.h be moved to Cross?)
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
On Wed, Dec 28, 2005 at 06:44:48PM -0800, tim Rowledge wrote:
On 28-Dec-05, at 6:36 PM, David T. Lewis wrote:
Worked fine on Intel Linux 32 bit. That's all I've tried so far.
Excellent, thanks for testing it Dave.
Interim update on building 64 bit VMs with the latest VMM/SVN:
Unfortunately I have not been able to build a working 64 VM yet. I'm using VMMaker-tpr.39 (loaded from Squeak Map), SVN 1282, plus my local patches (attached for reference). The OS is 32 bit Intel Linux.
I was previously able to build a 64 bit VM with VMMaker-tpr.37 and SVN 1259, with the same local patches.
Hmm, sadly not something I can be of much help with; no linux machine etc. All I can say is that *very* little changed between .37 and .39 that I could imagine having any such effect. Um, well diff is your reasonably polite acquaintance I think.
I am building the 64 bit VM with internal plugins BalloonEnginePlugin, BitBltSimulation, FilePlugin, and SocketPlugin, and no external plugins. The resulting VM crashes with a segfault after entering the interpreter and opening the display. This is using a 64 bit image that does open successfully with my earlier 64 bit VM (VMMaker-tpr.37/SVN 1259).
Also note, VMMaker versionString => '3.8b4' should be '3.8b5-64'.
D'oh.
More to follow when I get it figured out.
Dave
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
platforms-Cross-vm-sqMemoryAccess.h.diff - Add missing memory access macros (the inline functions are complete, but some of the corresponding macros are missing).
OK, I knew we had skipped some but hadn't had any problems as such.
platforms-unix-vm-sqUnixExternalPrims.c.diff - Provide meaningful console error message if an external module (e.g. vm-display-X11) fails to load.
platforms-unix-config-config.h.in.diff - Add a definition to unix/ config/config.in to #define SQAIO_H "sqaio.h" due to renaming aio.h to sqaio.h. Permits OSPP and AIO plugins to be backward compatible with older source trees. (Note to Tim: RiscOS uses a copy of aio.h in its platform tree, should sqaio.h be moved to Cross?)
It does? Gosh, I had no idea. Let me see what to do about that.
<DavesPlatformPatches-SVN-1282.zip>
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim When you earnestly believe you can compensate for a lack of skill by doubling your efforts, there's no end to what you can't do
tim Rowledge wrote:
Unfortunately I have not been able to build a working 64 VM yet. I'm using VMMaker-tpr.39 (loaded from Squeak Map), SVN 1282, plus my local patches (attached for reference). The OS is 32 bit Intel Linux.
I was previously able to build a 64 bit VM with VMMaker-tpr.37 and SVN 1259, with the same local patches.
Hmm, sadly not something I can be of much help with; no linux machine etc. All I can say is that *very* little changed between .37 and .39 that I could imagine having any such effect. Um, well diff is your reasonably polite acquaintance I think.
It is possible that this might be caused by changing the compiler. Ian and I had a lot of fun figuring out which versions of gcc are broken in subtle enough ways so that certain "optimizations" completely break the 64bit VM. I'd try using a different version of gcc (preferredly the one that worked before) and see if that helps.
Cheers, - Andreas
On Thu, Dec 29, 2005 at 01:00:05PM -0800, Andreas Raab wrote:
tim Rowledge wrote:
Unfortunately I have not been able to build a working 64 VM yet. I'm using VMMaker-tpr.39 (loaded from Squeak Map), SVN 1282, plus my local patches (attached for reference). The OS is 32 bit Intel Linux.
I was previously able to build a 64 bit VM with VMMaker-tpr.37 and SVN 1259, with the same local patches.
Hmm, sadly not something I can be of much help with; no linux machine etc. All I can say is that *very* little changed between .37 and .39 that I could imagine having any such effect. Um, well diff is your reasonably polite acquaintance I think.
It is possible that this might be caused by changing the compiler. Ian and I had a lot of fun figuring out which versions of gcc are broken in subtle enough ways so that certain "optimizations" completely break the 64bit VM. I'd try using a different version of gcc (preferredly the one that worked before) and see if that helps.
Thanks,
I have not changed compilers, so that's probably not it. But I have seen this kind of symptom before, so I might just be making some dumb mistake. I'll report back when I figure it out.
Dave
On Thu, Dec 29, 2005 at 12:55:03PM -0800, tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
platforms-Cross-vm-sqMemoryAccess.h.diff - Add missing memory access macros (the inline functions are complete, but some of the corresponding macros are missing).
OK, I knew we had skipped some but hadn't had any problems as such.
According to the comment in sqMemoryAccess.h:
#ifdef INLINING_WORKS_FOR_SMALL_ACCESSORS /* use these if static inline produces efficient code for such small accessor * functions. It doesn't on RISC OS and OSX as of may 2005 */
So the macros are there to accomodate the RiscOS and OSX compilers. But for some reason, configure ends up telling my Linux system to use the macros, even though it does not need to do so. The missing macros are apparently used only in the vm-display-X11 module, which of course is not needed for RiscOS but I suppose might be of interest on OSX.
In any case, I get the same results whether using the macros or the inline functions.
Dave
If you compile for unix on any PowerPC based platform or RISC you *will* need the INLINING_WORKS_FOR_SMALL_ACCESSORS logic on or off, which ever. The wrong choice will cause the some versions of the GCC compiler to generate most interesting confused PowerPC and RISC code leading to abysmal VM performance.
The macros are used throughout the interp.c when using the 32bit/ 64bit vmmaker.
On 29-Dec-05, at 2:22 PM, David T. Lewis wrote:
On Thu, Dec 29, 2005 at 12:55:03PM -0800, tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
platforms-Cross-vm-sqMemoryAccess.h.diff - Add missing memory access macros (the inline functions are complete, but some of the corresponding macros are missing).
OK, I knew we had skipped some but hadn't had any problems as such.
According to the comment in sqMemoryAccess.h:
#ifdef INLINING_WORKS_FOR_SMALL_ACCESSORS /* use these if static inline produces efficient code for such small accessor
- functions. It doesn't on RISC OS and OSX as of may 2005 */
So the macros are there to accomodate the RiscOS and OSX compilers. But for some reason, configure ends up telling my Linux system to use the macros, even though it does not need to do so. The missing macros are apparently used only in the vm-display-X11 module, which of course is not needed for RiscOS but I suppose might be of interest on OSX.
In any case, I get the same results whether using the macros or the inline functions.
Dave
-- ======================================================================== === John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===
On Thu, Dec 29, 2005 at 12:55:03PM -0800, tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Unfortunately I have not been able to build a working 64 VM yet. I'm using VMMaker-tpr.39 (loaded from Squeak Map), SVN 1282, plus my local patches (attached for reference). The OS is 32 bit Intel Linux.
I was previously able to build a 64 bit VM with VMMaker-tpr.37 and SVN 1259, with the same local patches.
Hmm, sadly not something I can be of much help with; no linux machine etc. All I can say is that *very* little changed between .37 and .39 that I could imagine having any such effect. Um, well diff is your reasonably polite acquaintance I think.
Well, there was not much you could have done to help me short of offering a brain transplant. I just had my plugin path screwed up and was accidentally loading a 32 bit display module with a 64 bit VM. Sorry about that.
So, back to the matter at hand. I've tested the new VMM with no problems on:
- 32 bit Intel, Linux, 32 bit VM - 32 bit Intel, Linux, 64 bit VM - 64 bit AMD Turion, 64 bit Linux, 64 bit VM - 64 bit AMD Turion, 64 bit Linux, 32 bit VM
The tests were done with VMMaker-tpr.39, SVN 1282, and the patches that I attached to my earlier message.
I did spot one issue that I had not previously noticed. With the combination of 64 bit machine and 32 bit VM, the memory access inline functions in sqMemoryAccess.h work fine, but the macros do not work. This is not related to my patch for sqMemoryAccess.h, since I tried building with the original version and running with no display, and still fail in a longAtput() macro.
Dave
On 30-Dec-05, at 9:59 AM, David T. Lewis wrote:
Well, there was not much you could have done to help me short of offering a brain transplant.
Well, I know a guy who knows a guy...
So, back to the matter at hand. I've tested the new VMM with no problems on:
- 32 bit Intel, Linux, 32 bit VM
- 32 bit Intel, Linux, 64 bit VM
- 64 bit AMD Turion, 64 bit Linux, 64 bit VM
- 64 bit AMD Turion, 64 bit Linux, 32 bit VM
The tests were done with VMMaker-tpr.39, SVN 1282, and the patches that I attached to my earlier message.
Good news. I'll soon release some new code to break all that ;-)
I did spot one issue that I had not previously noticed. With the combination of 64 bit machine and 32 bit VM, the memory access inline functions in sqMemoryAccess.h work fine, but the macros do not work. This is not related to my patch for sqMemoryAccess.h, since I tried building with the original version and running with no display, and still fail in a longAtput() macro.
I don't feel very surprised about that to be honest. With no 64 bit machine to test on and no very strong interest in messing with it it would be very plausible that I didn't write them adequately for a real 64bit machine. The possibility of the 'baseOffset' coming into play leaps to my currently addled brain for example.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Dreams are free, but you get soaked on the connect time.
Hokay, After taking Andreas' suggestion to dump the tacky old sqFilenameFromString macro to heart, SM has a new release 11 - 3.8b5-64B and SVN has level 1284 for you to play with. Read the SM page comments. It may well break things but I think it gets the basic facility cleaner. I also cleaned out some ugly arg-typing floobs while I was at it.
Note that it looks to me as if AsynchFilePlugin and DeflatePlugin could do with some looking at wrt 64bit cleanliness.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Last one out, turn off the computer!
On Fri, Dec 30, 2005 at 05:14:06PM -0800, tim Rowledge wrote:
Hokay, After taking Andreas' suggestion to dump the tacky old sqFilenameFromString macro to heart, SM has a new release 11 - 3.8b5-64B and SVN has level 1284 for you to play with. Read the SM page comments. It may well break things but I think it gets the basic facility cleaner. I also cleaned out some ugly arg-typing floobs while I was at it.
There is a cut'n'paste typo in one of the arg-type defloobifications, fix attached.
I've built a unix VM from the new VMM/svn sources, but I'm getting problems opening the sources and changes files. Something has changed that affects the operation of "FileDirectory default oldFileNamed: 'squeak.changes'". I'm getting a nil from that, while "FileStream oldFileNamed: 'squeak.changes'" still works as expected.
I have not spotted the source of the problem yet...
Dave
On 2-Jan-06, at 9:16 AM, David T. Lewis wrote:
On Fri, Dec 30, 2005 at 05:14:06PM -0800, tim Rowledge wrote:
Hokay, After taking Andreas' suggestion to dump the tacky old sqFilenameFromString macro to heart, SM has a new release 11 - 3.8b5-64B and SVN has level 1284 for you to play with. Read the SM page comments. It may well break things but I think it gets the basic facility cleaner. I also cleaned out some ugly arg-typing floobs while I was at it.
There is a cut'n'paste typo in one of the arg-type defloobifications, fix attached.
Ta. Mpeg3Plugin is one I don't/can't compile anyway so it doesn't get checked in a ny way but visual inspection at my end.
I've built a unix VM from the new VMM/svn sources, but I'm getting problems opening the sources and changes files. Something has changed that affects the operation of "FileDirectory default oldFileNamed: 'squeak.changes'". I'm getting a nil from that, while "FileStream oldFileNamed: 'squeak.changes'" still works as expected.
I have not spotted the source of the problem yet...
RISC OS uses its own equivalent ofthe Cross/plugins/FilePlugin/ sqFilePluginBasicPrims.c and so again it doesn't get used by my system. Most likely another typo since I didn't make any structural changes.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: FA: Failsafe Armed
On Mon, Jan 02, 2006 at 09:41:55AM -0800, tim Rowledge wrote:
On 2-Jan-06, at 9:16 AM, David T. Lewis wrote:
On Fri, Dec 30, 2005 at 05:14:06PM -0800, tim Rowledge wrote:
Hokay, After taking Andreas' suggestion to dump the tacky old sqFilenameFromString macro to heart, SM has a new release 11 - 3.8b5-64B and SVN has level 1284 for you to play with. Read the SM page comments. It may well break things but I think it gets the basic facility cleaner. I also cleaned out some ugly arg-typing floobs while I was at it.
There is a cut'n'paste typo in one of the arg-type defloobifications, fix attached.
Ta. Mpeg3Plugin is one I don't/can't compile anyway so it doesn't get checked in a ny way but visual inspection at my end.
I've built a unix VM from the new VMM/svn sources, but I'm getting problems opening the sources and changes files. Something has changed that affects the operation of "FileDirectory default oldFileNamed: 'squeak.changes'". I'm getting a nil from that, while "FileStream oldFileNamed: 'squeak.changes'" still works as expected.
I have not spotted the source of the problem yet...
RISC OS uses its own equivalent ofthe Cross/plugins/FilePlugin/ sqFilePluginBasicPrims.c and so again it doesn't get used by my system. Most likely another typo since I didn't make any structural changes.
Mea culpa. The sqGetFileNameFromString() function needs to null-terminate the file name string after copying it, and I had not done that.
The new VMM/SVN works fine on Intel 32 bit Linux for both a 32 bit and 64 bit VM. I'll check AMD-64 when I get a chance, but I don't think there will be any problems there.
The sqGetFileNameFromString() function I used is just this:
sqInt sqGetFilenameFromString(char *aCharBuffer, char *aFilenameString, sqInt filenameLength, sqInt aBoolean) { strncpy(aCharBuffer, aFilenameString, filenameLength); aCharBuffer[filenameLength] = 0; }
BTW, the Mpeg3Plugin is working fine on 32 bit Linux, so the rest of the cut'n'pasting must have been OK.
Dave
On 2-Jan-06, at 3:23 PM, David T. Lewis wrote:
Mea culpa. The sqGetFileNameFromString() function needs to null- terminate the file name string after copying it, and I had not done that.
That would cause interesting problems, certainly.
The new VMM/SVN works fine on Intel 32 bit Linux for both a 32 bit and 64 bit VM. I'll check AMD-64 when I get a chance, but I don't think there will be any problems there.
The sqGetFileNameFromString() function I used is just this:
sqInt sqGetFilenameFromString(char *aCharBuffer, char *aFilenameString, sqInt filenameLength, sqInt aBoolean) { strncpy(aCharBuffer, aFilenameString, filenameLength); aCharBuffer[filenameLength] = 0; }
I think Ian had some quite extensive charset converting code attached to the old macro? Ah, yes see sqUnixCharConv.c RISC OS uses the OS call to canonicalize the path and make sure it is fully qualified.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: LB: connect Line-voltage to BITNET
On Mon, Jan 02, 2006 at 04:38:54PM -0800, tim Rowledge wrote:
On 2-Jan-06, at 3:23 PM, David T. Lewis wrote:
Mea culpa. The sqGetFileNameFromString() function needs to null- terminate the file name string after copying it, and I had not done that.
That would cause interesting problems, certainly.
The new VMM/SVN works fine on Intel 32 bit Linux for both a 32 bit and 64 bit VM. I'll check AMD-64 when I get a chance, but I don't think there will be any problems there.
I built VMs on AMD 64, no problems. All looks well for:
Intel 32 bit Linux 32 bit VM Intel 32 bit Linux 64 bit VM AMD 64 bit Linux 32 bit VM AMD 64 bit Linux 64 bit VM
The previously mentioned patches still apply, including requirement for inlined memory access functions rather than macros for 32 bit VM on 64 bit AMD Linux.
The sqGetFileNameFromString() function I used is just this:
sqInt sqGetFilenameFromString(char *aCharBuffer, char *aFilenameString, sqInt filenameLength, sqInt aBoolean) { strncpy(aCharBuffer, aFilenameString, filenameLength); aCharBuffer[filenameLength] = 0; }
I think Ian had some quite extensive charset converting code attached to the old macro? Ah, yes see sqUnixCharConv.c RISC OS uses the OS call to canonicalize the path and make sure it is fully qualified.
I'll have a look. Should have no effect on the VMM release, which looks fine as near as I can tell.
Dave
On Mon, Jan 02, 2006 at 12:16:58PM -0500, David T. Lewis wrote:
I've built a unix VM from the new VMM/svn sources, but I'm getting problems opening the sources and changes files. Something has changed that affects the operation of "FileDirectory default oldFileNamed: 'squeak.changes'". I'm getting a nil from that, while "FileStream oldFileNamed: 'squeak.changes'" still works as expected.
Sorry, FileDirectory>>oldFileNamed: is fine, I must have mistyped something. I'm not sure what is causing the problem opening the sources and changes files. I'll report back when I find it (but maybe not today).
Dave
There was a bit of discussion about removing the caching of the filesize from the file prims last week. I see Andreas has checked in a win32 file package that made the change and John & I were just discussing it as well. The bit that has always puzzled me is why it was there in the first place and I think I know the answer now. The original basic file access code was written to use only the ansi C lib calls and that doesn't inclide a file size function. You *can* do an fseek to EOF and an ftell when you get there, which is pretty much what is done. Once you step outside the ansilib room you can just find the actual file size and drop the caching.
We could of course change the Cross/plugins/FilePlugin file to be posix 'portable' and allow an fstat call to replace the caching. Or we could make a 'getFileSize()' that has to be platform supplied just like all the directory calls. Or we could leave the Cross file and note that just about any non-trivial-embedded/experimental platform will likely benefit from a more specific file access code and see the various examples in the svn tree. Unix and Mac would then need to implement such, presumably as posix calls.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: BA: Branch Approximate
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
OK, that is up to andreas - I couldn't commit it if I wanted to.
platforms-Cross-vm-sqMemoryAccess.h.diff - Add missing memory access macros (the inline functions are complete, but some of the corresponding macros are missing).
OK, made the changes. Am I hallucinating or should we really make the #define oopForPointer(ptr) ((sqInt)(ptr)) read #define oopForPointer(ptr) ((sqInt)(ptr-sqMemoryBase)) instead?
platforms-unix-vm-sqUnixExternalPrims.c.diff - Provide meaningful console error message if an external module (e.g. vm-display-X11) fails to load.
That's one for Ian.
platforms-unix-config-config.h.in.diff - Add a definition to unix/ config/config.in to #define SQAIO_H "sqaio.h" due to renaming aio.h to sqaio.h. Permits OSPP and AIO plugins to be backward compatible with older source trees. (Note to Tim: RiscOS uses a copy of aio.h in its platform tree, should sqaio.h be moved to Cross?)
Well, maybe it should. I suppose if it is used by two or more platforms it makes a reasonable candidate for Cross. It is another posixy rather than ansi-ish file though.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Never write software that anthropomorphizes the machine. They hate that.
tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
OK, that is up to andreas - I couldn't commit it if I wanted to.
Now that's interesting. I can fix that easy enough but I'm curious as to the reason for having a "global" session ID instead of a local one. I always found it advantageous that (in all likelyhood) you couldn't use the session ID for a file interchangeably with the session ID of a socket. And I must say that I find the idea of exposing what essentially amounts to a "VM secret" to other plugins quite peculiar.
Can somebody remind what the intent of that change was? I'm sure there must've been some reason for it but I think I've missed the discussion of that feature.
Cheers, - Andreas
On Tue, Jan 03, 2006 at 02:21:29PM -0800, Andreas Raab wrote:
tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
OK, that is up to andreas - I couldn't commit it if I wanted to.
Now that's interesting. I can fix that easy enough but I'm curious as to the reason for having a "global" session ID instead of a local one. I always found it advantageous that (in all likelyhood) you couldn't use the session ID for a file interchangeably with the session ID of a socket. And I must say that I find the idea of exposing what essentially amounts to a "VM secret" to other plugins quite peculiar.
Can somebody remind what the intent of that change was? I'm sure there must've been some reason for it but I think I've missed the discussion of that feature.
There are are probably lots of reasons why you might want to identify the current "squeak session" for some particular image. My specific reason for wanting to do this was that, in WindowsOSProcessAccessor, I want to associate the squeak session identifier with the Win32 HANDLE of the input/output/error console streams. There is no point in asking the FilePlugin for its current session identifier if the Interpreter can provide it.
The fileID is an opaque value that just happens to correspond to some data structure that we are not supposed to know about, but it is easier to handle the issue in the Win32OSProcessPlugin that to try to design a platform independent solution that can be incorporated into FilePlugin for all supported platforms. The same argument applies to UnixOSProcessPlugin, and of course to RiscOSProcessPlugin, although the former is at a somewhat more advanced stage of implementation.
Thus in WindowsOSProcessAccessor>>primGetStdOutputForSession,
primGetStdOutputForSession: sessionIdentifierByteArray "Answer a two element array containing the sqFile data structure representing standard output stream for my OS process, and a flag (true or false) to indicate whether the sqFile data structure contains a valid HANDLE. If no standard output stream is available for this OS process, the sqFile data structure will contain an invalid HANDLE value, which will result in failures on subsequent accesses."
"OSProcess accessor primGetStdOutput"
<primitive: 'primitiveGetStdOutput' module: 'Win32OSProcessPlugin'> ^ nil
Hi Dave -
The fileID is an opaque value that just happens to correspond to some data structure that we are not supposed to know about, but it is easier to handle the issue in the Win32OSProcessPlugin that to try to design a platform independent solution that can be incorporated into FilePlugin for all supported platforms.
*Gasp* If I've ever seen "evil" code, this got to be it ;-) Not because it's badly written but because it's designed to break the (supposed) encapsulation of the file structure. If the file plugin code ever changes (such as when removing the fileSize) that code will make the VM explode.
The main lesson for me here is that if it's *that* easy to fake a file handle we've got a real problem with security. Basically, all of our I/O operations are prone to this kind of attack and I don't even want to think about all of the different ways in which this could be exploited.
In order to fix this, I just added a tiny little hash table to the windows file plugin which checks whether a file handle is genuine, e.g., has been produced inside the the file plugin or not. Since that hash table is not publicly accessible you can't store into it, thus you can no longer fabricate a random file handle.
BTW, this is not designed "just so your evil code breaks" - as far as I am concerned it's actually a *major* step forward towards a secure VM design because one of the effects this achieves is that the handle (token) that's used in the plugin is actually a capability to invoke a given set of primitives and no others, and it is solely at the plugins discretion which set of primitives to assign to which handle. Nice, simple, and secure (just as it should be).
One of the effects of that change is that it immediately makes the session ID superfluous from a security POV. The worst that could ever happen is that an "old" handle refers to a "new" file (if handles aren't unique across sessions) but an invalid handle can never be used in any of these operations. (note that from a security POV an old handle referring to a new file is indistinguishable from cloning an existing handle which is something that we don't deal with either right now - fixing this would be straightforward if we were able to deal with oops instead of third-party handles but that would require dealing with oop relocation during GC; ouch).
And now that we got this done ;-) we can start talking about how to expose the functionality that you need in a nice and cross-platform way. Like, accessing standard file handles: What's wrong with a primitiveGetStdFileHandle that takes an integer argument for the kind of standard handle to retrieve (0 - stdin, 1 - stdout, 2 - stderr)? Sounds simple enough to me. Any others you'd need?
Cheers, - Andreas
On Tue, Jan 03, 2006 at 10:41:29PM -0800, Andreas Raab wrote:
Hi Dave -
The fileID is an opaque value that just happens to correspond to some data structure that we are not supposed to know about, but it is easier to handle the issue in the Win32OSProcessPlugin that to try to design a platform independent solution that can be incorporated into FilePlugin for all supported platforms.
*Gasp* If I've ever seen "evil" code, this got to be it ;-) Not because it's badly written but because it's designed to break the (supposed) encapsulation of the file structure. If the file plugin code ever changes (such as when removing the fileSize) that code will make the VM explode.
Sure. In fact this has happened quite recently. The SQFile struct was changed to handle some alignment issues for the 64 bit VM. I had to put out an update to OSPP to accommodate the change. If nobody had been maintaining OSPP, this would have caused a problem. Thankfully the VM did not explode, although I did have some failing prims in the unit tests.
Manipulating fileID is a Bad Thing. In the case of OSPP, the tradeoff was between doing that particular bad thing versus attempting to maintain a package of platform-specific extensions as part of the FilePlugin, SocketPlugin, and so forth. I chose to implement the extensions separately, and this has worked well for me.
The main lesson for me here is that if it's *that* easy to fake a file handle we've got a real problem with security. Basically, all of our I/O operations are prone to this kind of attack and I don't even want to think about all of the different ways in which this could be exploited.
Yes. I suspect this would be among the least of our security problems, but you are absolutely right.
In order to fix this, I just added a tiny little hash table to the windows file plugin which checks whether a file handle is genuine, e.g., has been produced inside the the file plugin or not. Since that hash table is not publicly accessible you can't store into it, thus you can no longer fabricate a random file handle.
BTW, this is not designed "just so your evil code breaks" - as far as I am concerned it's actually a *major* step forward towards a secure VM design because one of the effects this achieves is that the handle (token) that's used in the plugin is actually a capability to invoke a given set of primitives and no others, and it is solely at the plugins discretion which set of primitives to assign to which handle. Nice, simple, and secure (just as it should be).
One of the effects of that change is that it immediately makes the session ID superfluous from a security POV. The worst that could ever happen is that an "old" handle refers to a "new" file (if handles aren't unique across sessions) but an invalid handle can never be used in any of these operations. (note that from a security POV an old handle referring to a new file is indistinguishable from cloning an existing handle which is something that we don't deal with either right now - fixing this would be straightforward if we were able to deal with oops instead of third-party handles but that would require dealing with oop relocation during GC; ouch).
This sounds like a pretty reasonable approach. It will have a rather large impact on OSPP file operations (e.g. file locking on Unix), but probably manageable. I would like to look it over and remind myself of all the things that are going to break. If you can hold off until say next week before diving into this I'd appreciate it.
And now that we got this done ;-) we can start talking about how to expose the functionality that you need in a nice and cross-platform way. Like, accessing standard file handles: What's wrong with a primitiveGetStdFileHandle that takes an integer argument for the kind of standard handle to retrieve (0 - stdin, 1 - stdout, 2 - stderr)? Sounds simple enough to me. Any others you'd need?
I don't really like the 0, 1, 2 convention, because it's an ancient Unix convention, and really they are supposed to be treated as opaque handles (seriously). It's probably OK to implement primitiveGetStdOut, primitiveGetStdErr, and primitiveGetStdErr in the FilePlugin as long as they are optional primitives and fail in reasonable ways. that these are really still unix conventions that have been adopted elsewhere, and may be rather platform-dependent. They behave differently on Win32 and unix, and I don't know what RiscOS does about them. Tim?
Pipe handles are definitely needed, and given Squeak's current approach to files and sockets, these should be treated as file handles. Anonymous pipes don't have path names, so the process of opening them is of course different. Pipes are very platform-dependent, and I'm not sure this belongs in the core plugins. I'll see if I can think of a way to do this that is not too "evil" ;)
Dave
On Wed, Jan 04, 2006 at 07:22:43AM -0500, David T. Lewis wrote:
On Tue, Jan 03, 2006 at 10:41:29PM -0800, Andreas Raab wrote:
And now that we got this done ;-) we can start talking about how to expose the functionality that you need in a nice and cross-platform way. Like, accessing standard file handles: What's wrong with a primitiveGetStdFileHandle that takes an integer argument for the kind of standard handle to retrieve (0 - stdin, 1 - stdout, 2 - stderr)? Sounds simple enough to me. Any others you'd need?
I don't really like the 0, 1, 2 convention, because it's an ancient Unix convention, and really they are supposed to be treated as opaque handles (seriously). It's probably OK to implement primitiveGetStdOut, primitiveGetStdErr, and primitiveGetStdErr in the FilePlugin as long as they are optional primitives and fail in reasonable ways. that these are really still unix conventions that have been adopted elsewhere, and may be rather platform-dependent. They behave differently on Win32 and unix, and I don't know what RiscOS does about them. Tim?
Pipe handles are definitely needed, and given Squeak's current approach to files and sockets, these should be treated as file handles. Anonymous pipes don't have path names, so the process of opening them is of course different. Pipes are very platform-dependent, and I'm not sure this belongs in the core plugins. I'll see if I can think of a way to do this that is not too "evil" ;)
After giving this a little more thought, I can see two feasible ways to expose the required functionality while still allowing OSPP to exist as as optional external plugin.
1) Move primitiveCreatePipe, primitiveGetStdIn, primitiveGetStdOut, and primitiveGetStdErr out of the OSPP plugins and into FilePlugin, and sort out the platform differences in the support code.
2) Have FilePlugin provide a primitiveOpenHandle that could be invoked in a manner similar to the current StandardFileStream>>primOpen:writable:, except that the supplied parameter would be something representing a (HANDLE) or (FILE *), e.g. "primOpenHandle: aByteArray ofLength: 4 writable: true".
Option #1 is undesirable because both the concepts and the implementations are quite platform-specific. But it is workable as long as the VM maintainers don't mind, and as long as at least the Unix and Win32 implementations actually get done.
Option #2 would work well, but does not address the security concern. If spoofing a fileID is a security concern, then spoofing a HANDLE would presumably be equally bad. But it does at least get rid of the outside manipulation of private data structures.
In either case it would be helpful to add a primitiveFileHandle that answers the platform-specific (HANDLE) or (FILE *) corresponding to the certified, non-fake fileID. This would be used by platform-specific extensions (e.g. file locking, fcntl functions) and would eliminate the need for knowledge of the SQFile struct in other plugins. Ditto for the SocketPlugin, if only for the sake of uniformity.
From my own point of view, either of these would be OK. I also have no
objection to leaving things as they are, although I would prefer that the various FilePlugin implementations use the same sessionIdentifier. The workaround I've had to use for the last few years is truly awful.
Dave
On 3-Jan-06, at 2:21 PM, Andreas Raab wrote:
tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
OK, that is up to andreas - I couldn't commit it if I wanted to.
Now that's interesting. I can fix that easy enough but I'm curious as to the reason for having a "global" session ID instead of a local one. I always found it advantageous that (in all likelyhood) you couldn't use the session ID for a file interchangeably with the session ID of a socket. And I must say that I find the idea of exposing what essentially amounts to a "VM secret" to other plugins quite peculiar.
Can somebody remind what the intent of that change was? I'm sure there must've been some reason for it but I think I've missed the discussion of that feature.
We've discussed it several times on this list and others since (believe it or not, I was amazed) November. November 2001 that is...
Part of my liking of the idea is reducing code duplication and the concomitant likelihood of mis-duplication - think of it as a service the VM core offers plugins. Why is hiding one plugin's session ID useful? I can't immediately think of anything devious one could usefully do with this secret info.
Obviously one doesn't have to use this facility but I can't see any reason not to.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A low level language is one whose programs require attention to the irrelevant.
tim Rowledge wrote:
Can somebody remind what the intent of that change was? I'm sure there must've been some reason for it but I think I've missed the discussion of that feature.
We've discussed it several times on this list and others since (believe it or not, I was amazed) November. November 2001 that is...
Oh, I believe it. Coincidentally, this would also be the exact time frame when I moved from the US to Germany and probably wasn't paying close attention ;-)
Part of my liking of the idea is reducing code duplication and the concomitant likelihood of mis-duplication - think of it as a service the VM core offers plugins. Why is hiding one plugin's session ID useful? I can't immediately think of anything devious one could usefully do with this secret info.
Actually it really isn't all that secret (which is part of the problem). If you were to manifacture a file handle you could inspect an existing one and copy the session ID out of it. The one thing it does do is preventing mixing up file handles with other I/O handles if they have the same size and different magic numbers.
Obviously one doesn't have to use this facility but I can't see any reason not to.
True. Although I think it is really valuable if a plugin can verify that a handle is genuine (e.g., actually produced by that plugin). See the following message.
Cheers, - Andreas
On Tue, Jan 03, 2006 at 01:34:35PM -0800, tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
platforms-Cross-vm-sqMemoryAccess.h.diff - Add missing memory access macros (the inline functions are complete, but some of the corresponding macros are missing).
OK, made the changes. Am I hallucinating or should we really make the #define oopForPointer(ptr) ((sqInt)(ptr)) read #define oopForPointer(ptr) ((sqInt)(ptr-sqMemoryBase)) instead?
Looks like another of those annoying cut-n-paste errors. Apparently that particular macro has not not been getting a lot of heavy use.
Dave
Tim,
Some belated followup on this. The patches for sqMemoryAccess.h do not seem to have been committed to SVN as of SVN 1352. I'm attaching a copy of this patch along with the others that I currently apply to the platforms tree in order to do a Unix VM build.
Unrelated to the patch I sent for sqMemoryAccess.h, but affecting the same source file, you had asked if the oopForPointer macro was defined wrong and I had replied that I thought that it did look wrong. However, I just got around to trying a build with the macro defined as: #define oopForPointer(ptr) ((sqInt)((ptr) - sqMemoryBase))
and this does *not* work (blows up with a compiler error). So I don't know what is right here, but you should not make that change until somebody figures it out.
For reference, here is a summary of the patches (attached) that I am currently using. Two patches are required, and the other two are nice to have.
platforms-unix-vm-sqUnixCharConv.c.diff - Provide missing sqGetFilenameFromString() function. This function must be present in order to build a unix VM.
platforms-Cross-vm-sqMemoryAccess.h.diff - Missing memory access macros. Required for Linux build.
platforms-unix-vm-sqUnixExternalPrims.c.diff - Provide meaningful error message if e.g. vm-display-X11 fails to load for some reason. Nice to have but not required for successful VM build.
platforms-unix-config-config.h.in.diff - Define #SQAIO_H. Required for OSPP only, nice to have.
I'll note also that I cannot build an MPEG plugin for Unix with the current sources, but I assume this a new issue related to John's recent upgrades and I have not looked into it yet.
Dave
On Tue, Jan 03, 2006 at 01:34:35PM -0800, tim Rowledge wrote:
On 29-Dec-05, at 12:00 PM, David T. Lewis wrote:
Attachments (in zip file):
platforms-win32-plugins-FilePlugin-sqWin32FilePrims.c.diff - Change Win32 FilePlugin to use session ID from the interpreter rather than generate its own value (for consistency, and also so as not to break OSPP for Win32).
OK, that is up to andreas - I couldn't commit it if I wanted to.
platforms-Cross-vm-sqMemoryAccess.h.diff - Add missing memory access macros (the inline functions are complete, but some of the corresponding macros are missing).
OK, made the changes. Am I hallucinating or should we really make the #define oopForPointer(ptr) ((sqInt)(ptr)) read #define oopForPointer(ptr) ((sqInt)(ptr-sqMemoryBase)) instead?
platforms-unix-vm-sqUnixExternalPrims.c.diff - Provide meaningful console error message if an external module (e.g. vm-display-X11) fails to load.
That's one for Ian.
platforms-unix-config-config.h.in.diff - Add a definition to unix/ config/config.in to #define SQAIO_H "sqaio.h" due to renaming aio.h to sqaio.h. Permits OSPP and AIO plugins to be backward compatible with older source trees. (Note to Tim: RiscOS uses a copy of aio.h in its platform tree, should sqaio.h be moved to Cross?)
Well, maybe it should. I suppose if it is used by two or more platforms it makes a reasonable candidate for Cross. It is another posixy rather than ansi-ish file though.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Never write software that anthropomorphizes the machine. They hate that.
On 12-Mar-06, at 9:35 AM, David T. Lewis wrote:
Tim,
Some belated followup on this. The patches for sqMemoryAccess.h do not seem to have been committed to SVN as of SVN 1352. I'm attaching a copy of this patch along with the others that I currently apply to the platforms tree in order to do a Unix VM build.
Unrelated to the patch I sent for sqMemoryAccess.h, but affecting the same source file, you had asked if the oopForPointer macro was defined wrong and I had replied that I thought that it did look wrong. However, I just got around to trying a build with the macro defined as: #define oopForPointer(ptr) ((sqInt)((ptr) - sqMemoryBase))
and this does *not* work (blows up with a compiler error). So I don't know what is right here, but you should not make that change until somebody figures it out.
Weird; I can't spot any differences with the memory access header. The version I have (updated today from squeakvm.org) seems correct to me. Can you double check and if there are diffs send me the *whole* file. I don't read unix diffs very well.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- Several nuts over fruitcake minimum.
vm-dev@lists.squeakfoundation.org
-
Alan Sibley -
Andreas Raab -
Bert Freudenberg -
David T. Lewis -
Ian Piumarta -
John M McIntosh -
tim Rowledge