I'm making a start at building the RISC OS StackVM and running into some fun problems.
First one was the code in VMMaker class>generateConfiguration that builds directory names by means of evil URI related methods. Strange and ugly results occur for RISC OS and tracing the code reveals many strange and ugly things happening for every platform - it's just that on an OS where something like / is acceptable as a root directory it sort of ends up ok. RISC OS does not do that and I'm a little surprised that it doesn't cause problems on Windows too. My perfectly nice SDFS:RISCOSPi.$ root gets mangled to something like /SDFS%3ARISCOSPi%2F%2a/ or whatever. A simple fix for that is to change to "(FileDirectory default directoryNamed:'foo/bar') fullName"
My next problem is that RISC OS must do the minor manglings seen in methods such as RiscOSVMMaker>export:forExternalPlugin: in order to get all its ducks in a row. Right now we have an explicit creation of a CrossPlatformVMMaker in the #createCodeGenerator method; I'll see if over-riding #forPlatform: will work. There simply isn't any way I can squeeze things within the constraints of the Cross way of making files for everything; RISC OS is just too different.
In CCodeGenerator>generateInlineCppIfElse:asArgument:on:indent: we have" aStream ensureCr" which appears to upset a NullStream. Adding #ensureCr to the list in VMPluginCodeGenerator>emitCCodeOn:doInlining:doAssertions: gets us past that but I'm surprised it hasn't bitten people before.
More news as it happens from the frontline of this spectacular battlefield engagement...
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: AII: Add Insult to Injury
On Tue, Apr 09, 2013 at 03:25:52PM -0700, tim Rowledge wrote:
I'm making a start at building the RISC OS StackVM and running into some fun problems.
First one was the code in VMMaker class>generateConfiguration that builds directory names by means of evil URI related methods. Strange and ugly results occur for RISC OS and tracing the code reveals many strange and ugly things happening for every platform - it's just that on an OS where something like / is acceptable as a root directory it sort of ends up ok. RISC OS does not do that and I'm a little surprised that it doesn't cause problems on Windows too. My perfectly nice SDFS:RISCOSPi.$ root gets mangled to something like /SDFS%3ARISCOSPi%2F%2a/ or whatever. A simple fix for that is to change to "(FileDirectory default directoryNamed:'foo/bar') fullName"
On a related (?) note, I would like to ask if the RISC OS build could be changed to allow all plugins (whether internal or external) to be generated to a single source directory, i.e.
src/plugins/FilePlugin/ src/plugins/OSProcessPlugin/
Instead of:
src/vm/intplugins/FilePlugin/ src/plugins/OSProcessPlugin/
I think that we have had agreement from Eliot and Andreas (who moved to this structure for the Cog work) as well as Ian (who also strongly encouraged it and plans to make the change in trunk). But I don't know if the flattened directory structure might cause problems for RISC OS.
The VMM support is in place if we are ready to make this change:
VMMaker class>>useSinglePluginsDirectory:
So, question to Tim - would it cause problems for you if we move to a single directory for internal and external generated plugins?
Dave
On 09-04-2013, at 8:41 PM, "David T. Lewis" lewis@mail.msen.com wrote:
On a related (?) note, I would like to ask if the RISC OS build could be changed to allow all plugins (whether internal or external) to be generated to a single source directory, i.e.
src/plugins/FilePlugin/ src/plugins/OSProcessPlugin/
Instead of:
src/vm/intplugins/FilePlugin/ src/plugins/OSProcessPlugin/
Absolutely no problem at all; I only ever used external plugins anyway so all it means to me is a simple name change somewhere in the verbiage-tangle that is the makefile.
I can't even remember with any precision why we had separate internal/external directories originally; I think it was something Andreas used as a way to discriminate something about building the plugins before the code was made to be the same for both ways? Like a lot of the apparent strangeness of how VMMaker works it was a matter of compromise between several idiotic OS's (remember this was 15 years ago) being maintained by several headstrong, irritable, opinionated geeks, dealing with a codebase that was up till then largely managed as an ad hoc patching of a set of 'files' kept as comments in the image and local platform specialisations on individuals' machines.
I originally split things up into so many places as away of keeping different things tidy, because, well, tidy. Keeping generated code separate from static code made it reasonable to programmatically delete all the generated code to clean out and re-generate, as well as test the file date and use that to work out which ones might need re-creating. (The extra little pain there was Andreas insisting that he wanted Windows generated code to go into the same place as his platform specific code) I also wanted to avoid copying static code files - which might have made makefiles simpler - since you just know that one will edit the wrong copy, delete it and then wonder what the hell happened to break all your work. Oh and of course the Squeak file copying method was utterly fucked for any system that had any sort of meta-data associated with a file, like dates, permissions, what have you.
I would have replicated the directory layout that Eliot used for BrouHaHa, with nice distinct directories for all the parts and a directory of links to all the files actually needed for a particular version - except only unices had working links back then and of course Squeak had no link related stuff to allow creating such directories. I always assumed both would come pretty soon. So much for *that* prediction.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim If at first you don't succeed, call it version 1.0
David,
when you make this change (if you haven't already) you might also integrate the Cog-style plugin glue, e.g. instead of using interpreterProxy->foo the generator always uses foo and either links directly to foo or keeps it's own local function pointer to foo which is initialized in setInterpreter:. It's faster for internal plugins and a little faster for external plugins. ANSI C removed the need for a syntactic difference between a direct function call and a call through a function pointer hence...
towards the start of the file you see e.g.
#if !defined(SQUEAK_BUILTIN_PLUGIN) static sqInt (*byteSizeOf)(sqInt oop); ... static sqInt (*storeIntegerofObjectwithValue)(sqInt index, sqInt oop, sqInt integer); #else /* !defined(SQUEAK_BUILTIN_PLUGIN) */ extern sqInt byteSizeOf(sqInt oop); ... extern sqInt storeIntegerofObjectwithValue(sqInt index, sqInt oop, sqInt integer);
extern #endif struct VirtualMachine* interpreterProxy; ...
in the middle you see destBitsSize = byteSizeOf(destBits);
and in setInterpreter: you see
#if !defined(SQUEAK_BUILTIN_PLUGIN) byteSizeOf = interpreterProxy->byteSizeOf; ... storeIntegerofObjectwithValue = interpreterProxy->storeIntegerofObjectwithValue; #endif /* !defined(SQUEAK_BUILTIN_PLUGIN) */
On Tue, Apr 9, 2013 at 8:41 PM, David T. Lewis lewis@mail.msen.com wrote:
On Tue, Apr 09, 2013 at 03:25:52PM -0700, tim Rowledge wrote:
I'm making a start at building the RISC OS StackVM and running into some
fun problems.
First one was the code in VMMaker class>generateConfiguration that
builds directory names by means of evil URI related methods. Strange and ugly results occur for RISC OS and tracing the code reveals many strange and ugly things happening for every platform - it's just that on an OS where something like / is acceptable as a root directory it sort of ends up ok. RISC OS does not do that and I'm a little surprised that it doesn't cause problems on Windows too. My perfectly nice SDFS:RISCOSPi.$ root gets mangled to something like /SDFS%3ARISCOSPi%2F%2a/ or whatever. A simple fix for that is to change to "(FileDirectory default directoryNamed:'foo/bar') fullName"
On a related (?) note, I would like to ask if the RISC OS build could be changed to allow all plugins (whether internal or external) to be generated to a single source directory, i.e.
src/plugins/FilePlugin/ src/plugins/OSProcessPlugin/
Instead of:
src/vm/intplugins/FilePlugin/ src/plugins/OSProcessPlugin/
I think that we have had agreement from Eliot and Andreas (who moved to this structure for the Cog work) as well as Ian (who also strongly encouraged it and plans to make the change in trunk). But I don't know if the flattened directory structure might cause problems for RISC OS.
The VMM support is in place if we are ready to make this change:
VMMaker class>>useSinglePluginsDirectory:
So, question to Tim - would it cause problems for you if we move to a single directory for internal and external generated plugins?
Dave
Hi Eliot,
Sorry I missed this email, I did not mean to ignore you.
Yes I'll look into this, thanks for the explanation.
Dave
On Wed, Apr 10, 2013 at 02:09:34PM -0700, Eliot Miranda wrote:
David,
when you make this change (if you haven't already) you might also
integrate the Cog-style plugin glue, e.g. instead of using interpreterProxy->foo the generator always uses foo and either links directly to foo or keeps it's own local function pointer to foo which is initialized in setInterpreter:. It's faster for internal plugins and a little faster for external plugins. ANSI C removed the need for a syntactic difference between a direct function call and a call through a function pointer hence...
towards the start of the file you see e.g.
#if !defined(SQUEAK_BUILTIN_PLUGIN) static sqInt (*byteSizeOf)(sqInt oop); ... static sqInt (*storeIntegerofObjectwithValue)(sqInt index, sqInt oop, sqInt integer); #else /* !defined(SQUEAK_BUILTIN_PLUGIN) */ extern sqInt byteSizeOf(sqInt oop); ... extern sqInt storeIntegerofObjectwithValue(sqInt index, sqInt oop, sqInt integer);
extern #endif struct VirtualMachine* interpreterProxy; ...
in the middle you see destBitsSize = byteSizeOf(destBits);
and in setInterpreter: you see
#if !defined(SQUEAK_BUILTIN_PLUGIN) byteSizeOf = interpreterProxy->byteSizeOf; ... storeIntegerofObjectwithValue = interpreterProxy->storeIntegerofObjectwithValue; #endif /* !defined(SQUEAK_BUILTIN_PLUGIN) */
On Tue, Apr 9, 2013 at 8:41 PM, David T. Lewis lewis@mail.msen.com wrote:
On Tue, Apr 09, 2013 at 03:25:52PM -0700, tim Rowledge wrote:
I'm making a start at building the RISC OS StackVM and running into some
fun problems.
First one was the code in VMMaker class>generateConfiguration that
builds directory names by means of evil URI related methods. Strange and ugly results occur for RISC OS and tracing the code reveals many strange and ugly things happening for every platform - it's just that on an OS where something like / is acceptable as a root directory it sort of ends up ok. RISC OS does not do that and I'm a little surprised that it doesn't cause problems on Windows too. My perfectly nice SDFS:RISCOSPi.$ root gets mangled to something like /SDFS%3ARISCOSPi%2F%2a/ or whatever. A simple fix for that is to change to "(FileDirectory default directoryNamed:'foo/bar') fullName"
On a related (?) note, I would like to ask if the RISC OS build could be changed to allow all plugins (whether internal or external) to be generated to a single source directory, i.e.
src/plugins/FilePlugin/ src/plugins/OSProcessPlugin/
Instead of:
src/vm/intplugins/FilePlugin/ src/plugins/OSProcessPlugin/
I think that we have had agreement from Eliot and Andreas (who moved to this structure for the Cog work) as well as Ian (who also strongly encouraged it and plans to make the change in trunk). But I don't know if the flattened directory structure might cause problems for RISC OS.
The VMM support is in place if we are ready to make this change:
VMMaker class>>useSinglePluginsDirectory:
So, question to Tim - would it cause problems for you if we move to a single directory for internal and external generated plugins?
Dave
-- best, Eliot
On 10 April 2013 05:41, David T. Lewis lewis@mail.msen.com wrote:
On Tue, Apr 09, 2013 at 03:25:52PM -0700, tim Rowledge wrote:
I'm making a start at building the RISC OS StackVM and running into some fun problems.
First one was the code in VMMaker class>generateConfiguration that builds directory names by means of evil URI related methods. Strange and ugly results occur for RISC OS and tracing the code reveals many strange and ugly things happening for every platform - it's just that on an OS where something like / is acceptable as a root directory it sort of ends up ok. RISC OS does not do that and I'm a little surprised that it doesn't cause problems on Windows too. My perfectly nice SDFS:RISCOSPi.$ root gets mangled to something like /SDFS%3ARISCOSPi%2F%2a/ or whatever. A simple fix for that is to change to "(FileDirectory default directoryNamed:'foo/bar') fullName"
On a related (?) note, I would like to ask if the RISC OS build could be changed to allow all plugins (whether internal or external) to be generated to a single source directory, i.e.
src/plugins/FilePlugin/ src/plugins/OSProcessPlugin/
Instead of:
src/vm/intplugins/FilePlugin/ src/plugins/OSProcessPlugin/
+1
it is already like that in CMakeVMMaker. I don't remember how i did it, but generated code placed in the way like you proposing:
[~/projects/cog/sig-cog/src]: ls -l total 16 -rw-r--r--@ 1 sig staff 826 Feb 15 14:54 examplePlugins.ext -rw-r--r--@ 1 sig staff 83 Feb 15 14:54 examplePlugins.int drwxr-xr-x 49 sig staff 1666 Jan 8 15:39 plugins drwxr-xr-x 13 sig staff 442 Feb 15 14:54 vm
[~/projects/cog/sig-cog/src]: ls -l plugins total 0 drwxr-xr-x 3 sig staff 102 Feb 15 14:54 ADPCMCodecPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 AioPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 AsynchFilePlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 B2DPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 B3DAcceleratorPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 BMPReadWriterPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 BitBltPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 ClipboardExtendedPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 CroquetPlugin drwxr-xr-x 3 sig staff 102 Feb 15 14:54 DSAPrims ....
I think that we have had agreement from Eliot and Andreas (who moved to this structure for the Cog work) as well as Ian (who also strongly encouraged it and plans to make the change in trunk). But I don't know if the flattened directory structure might cause problems for RISC OS.
The VMM support is in place if we are ready to make this change:
VMMaker class>>useSinglePluginsDirectory:
So, question to Tim - would it cause problems for you if we move to a single directory for internal and external generated plugins?
Dave
I have a whole 8 methods that I changed to make generating sources appear to work on RISC OS, so I'll attempt to submit them later.
Next question; what sort of changes were made to platform code to work with the new core? I've noted the mention of the heartbeat ticker, which is actually something that may seriously benefit RISC OS and is probably something that ought to have been put in the plain VM years ago.
Oh and I've been trying to make sense of the global struct related changes. Umm…. a bit confused. Help.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: DTFS: Dump Tape to Floor and Shred
Hi Tim,
On Wed, Apr 10, 2013 at 12:43 PM, tim Rowledge tim@rowledge.org wrote:
I have a whole 8 methods that I changed to make generating sources appear to work on RISC OS, so I'll attempt to submit them later.
Next question; what sort of changes were made to platform code to work with the new core? I've noted the mention of the heartbeat ticker, which is actually something that may seriously benefit RISC OS and is probably something that ought to have been put in the plain VM years ago.
The main changes (on the stack vm) are - a heartbeat. preferrably this is a thread in a spin-loop, blocking on a delay, and forcing a poll when it unblocks. see http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/unix/vm/sqUnixHear... http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/win32/vm/sqWin32He.... but for this to work the heartbeat thread must have a higher priority than the VM thread and on linux only the superuser can create real-time threads that have settable priorities, so.... there-on one has to use an interval timer, which sucks as it a) interrupts system calls (SA_RESTART is essential but third-party libraries may be phased by the interval timer) and b) it's a pain to debug through cuz gdb will stop in the signal handler until one says "handle 14 ignore", or some such.
- a 64-bit microsecond clock as the basis of time, see the heartbeat files.
- a thread-safe external semaphore registry, see http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/Cross/vm/sqExterna...
- some atomic access primitives used to implement both the 64-bit microsecond clock and the external semaphore registry in a lock-free form. see http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/Cross/vm/%7BsqAtom...
There's also some Qwaq/teleplace/Term left-overs in the form of a ticker thread for running multi-media processing in parallel with the VM. See sqTicker.c, but I wouldn't worry about that.
There's also some experimental stuff to do with support for a multi-threaded VM along the lines of e.g. the S# or Python VMs where any number of threads can share the VM, but only one is actually running the VM at any one time, and hand-over between threads is carefully mediated. See various COGMTVM defines in the VM.
Oh and I've been trying to make sense of the global struct related changes.
Umm…. a bit confused. Help.
Ah.... well, one of my directions in modifying Slang for Cog/Qwaq was to defer decisions until compile time, getting Slang to generate source which could determine - whether a plugin was internal or external - whether a plugin was included - whether to use the global struct or flat variables could be chosen when compiling the VM, not when generating the source. It makes the generated C source cross-platform, cuts down on duplication, and makes it much easier to run experiments to measure if the global struct is faster or not.
So... to defer the choice of global struct to compile time we...
a) wrap all references to variable which might be in the global struct in the GIV macro (Global Interpreter Variable), see CCodeGeneratorGlobalStructure>>#returnPrefixFromVariable:, so in the source you see e.g.
sqInt interpret(void) { DECL_MAYBE_SQ_GLOBAL_STRUCT register sqInt currentBytecode CB_REG; register char* localFP FP_REG; register char* localIP IP_REG; ...
if (GIV(stackLimit) == 0) { /* begin initStackPagesAndInterpret */
b) define GIV optionally, and output the variables that could be in the struct in a block with a suitable prefix. It looks like:
/*** Variables ***/ #if SQ_USE_GLOBAL_STRUCT # define _iss /* define in-struct static as void */ static struct foo { #else # define _iss static #endif _iss char * stackPointer; ... _iss sqInt theUnknownShort; #undef _iss #if SQ_USE_GLOBAL_STRUCT } fum; # define DECL_MAYBE_SQ_GLOBAL_STRUCT register struct foo * foo = &fum; # define DECL_MAYBE_VOLATILE_SQ_GLOBAL_STRUCT volatile register struct foo * foo = &fum; # define GIV(interpreterInstVar) (foo->interpreterInstVar) #else # define DECL_MAYBE_SQ_GLOBAL_STRUCT /* oh, no mr bill! */ # define DECL_MAYBE_VOLATILE_SQ_GLOBAL_STRUCT /* oh no, mr bill! */ # define GIV(interpreterInstVar) interpreterInstVar #endif #if SQ_USE_GLOBAL_STRUCT static struct foo * foo = &fum; #endif
which is effectively either
static struct foo { char * stackPointer; .... char * stackPointer; } fum;
#define GIV(v) (foo->v)
or
static char * stackPointer; ... static sqInt theUnknownShort; #define GIV(v) v
c) define an optional macro for declaring a local reference to foo in functions that need it, so you see
sqInt interpret(void) { DECL_MAYBE_SQ_GLOBAL_STRUCT
and there's either
# define DECL_MAYBE_SQ_GLOBAL_STRUCT register struct foo * foo = &fum; # define DECL_MAYBE_VOLATILE_SQ_GLOBAL_STRUCT volatile register struct foo * foo = &fum;
or
# define DECL_MAYBE_SQ_GLOBAL_STRUCT /* oh, no mr bill! */ # define DECL_MAYBE_VOLATILE_SQ_GLOBAL_STRUCT /* oh no, mr bill! */
Then in your makefile you can do -DUSE_GLOBAL_STRUCT=1 if it floats your boat. This is a great idea on PPC but doesn't work on x86. I'd be interested to know which is better on ARM.
Make sense? If not, uncork a Bowmore and read at leisure...
Hmm, interesting stuff…
On 10-04-2013, at 6:05 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
The main changes (on the stack vm) are
- a heartbeat. preferrably this is a thread in a spin-loop, blocking on a delay, and forcing a poll when it unblocks.
What I can do reasonably well is have a millisecond (actually, arbitrary, non-regular intervals can be done, up to about 1MHz) tick that keeps a counter, flips a flag or similar. I *can't* run any code in the application since it is not knowable if the application will be mapped into memory at that point and so it might jump to an arbitrary point in some other application. Fun, eh? So, for example, I could have a known memory address that is read (or even branched through) and the tick would have changed it to affect the execution path. Might be some cache issues to sort out but my current mSec timer works ok. So if there is some place that can check a single flag for whether an process check is needed and that gets run at appropriate times, then we're good. Essentially I can add an interrupt handler in user space but it can only play in its own sandbox; main application code can then read/set that info as well. I'll dig around users of checkHighPriorityTickees() etc.
- a 64-bit microsecond clock as the basis of time, see the heartbeat files.
'halfAMo'? Seriously; 'halfAMo'? I knew there was a reason working with you is fun.
a thread-safe external semaphore registry, see http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/Cross/vm/sqExterna...
some atomic access primitives used to implement both the 64-bit microsecond clock and the external semaphore registry in a lock-free form. see http://www.squeakvm.org/svn/squeak/branches/Cog/platforms/Cross/vm/%7BsqAtom...
Yeah, can probably do that.
There's also some Qwaq/teleplace/Term left-overs in the form of a ticker thread for running multi-media processing in parallel with the VM. See sqTicker.c, but I wouldn't worry about that.
I won't.
There's also some experimental stuff to do with support for a multi-threaded VM along the lines of e.g. the S# or Python VMs where any number of threads can share the VM, but only one is actually running the VM at any one time, and hand-over between threads is carefully mediated. See various COGMTVM defines in the VM.
Threads? We don't do no threads man, way too establishment.
Then in your makefile you can do -DUSE_GLOBAL_STRUCT=1 if it floats your boat. This is a great idea on PPC but doesn't work on x86. I'd be interested to know which is better on ARM.
John McI did the original global struct stuff around '99 and we adopted it in '02, and I reworked it to allow a single register to be global (Yay for global register variables and for Harry Wossname putting them into NorCroft CC in 1988) and point to the table; you might not believe how many instructions that saved on an ARM. I *think* it made the system something like 30% faster overall. So yeah, it's better.
Make sense? If not, uncork a Bowmore and read at leisure…
Coffee works better for me. And toasted cheese scones.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Dopeler effect (n): The tendency of stupid ideas to seem smarter when they come at you rapidly.
One more detail, you missed Eliot: - in Cog, image side C code generator uses single class for all 3 platforms.
vm-dev@lists.squeakfoundation.org