I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/89.html - do please comment on the relevant page, or even here.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Oxymorons: New classic
Hi Tim,
I agree, but I cannot follow with what you see can be done with the saved bits. You say "The 10 released bits can be used for a variety of purposes including marking immutable objects, fixed address objects, proxy objects etc.". However, none of those three examples seem to be related as you only spare bits in compiled methods, but not in the general object header. What have I missed?
Cheers, Adrian
On Jun 30, 2008, at 20:34 , tim Rowledge wrote:
I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/ 89.html - do please comment on the relevant page, or even here.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Oxymorons: New classic
On 30-Jun-08, at 11:52 AM, Adrian Lienhard wrote:
Hi Tim,
I agree, but I cannot follow with what you see can be done with the saved bits. You say "The 10 released bits can be used for a variety of purposes including marking immutable objects, fixed address objects, proxy objects etc.". However, none of those three examples seem to be related as you only spare bits in compiled methods, but not in the general object header. What have I missed?
Nothing. I am an idiot. I was confusing different headers. Ignore everything after "variety of purposes"....
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "Bother" said Pooh, as the Vice Squad took his GIFS ..
2008/6/30 tim Rowledge tim@rowledge.org:
I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/89.html - do please comment on the relevant page, or even here.
This is not the only things which i don't like. For instance look into ObjectMemory>>fetchClassOf: it fetching class depending on object header. Either from compact classes array (by index), or from slot, reserved for class reference. I think eliminating compact classes would simplify the access to object's class and free us some bits again: type mask bit (at least).
It also will make fetching a class oop faster and uniform, so there will be no branching. Fetching a class from Compact classes array, requires: getting oop at index in special objects array, and then getting oop from this object:
class := fetchPointer: ccIndex - 1 ofObject: (self fetchPointer: CompactClasses ofObject: specialObjectsOop)
A drawback of getting rid of compact classes is additional memory cost per each instance of classes which were compact.
On 30-Jun-08, at 12:15 PM, Igor Stasenko wrote:
2008/6/30 tim Rowledge tim@rowledge.org:
I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/ 89.html - do please comment on the relevant page, or even here.
This is not the only things which i don't like. For instance look into ObjectMemory>>fetchClassOf: it fetching class depending on object header. Either from compact classes array (by index), or from slot, reserved for class reference. I think eliminating compact classes would simplify the access to object's class and free us some bits again: type mask bit (at least).
It also will make fetching a class oop faster and uniform, so there will be no branching. Fetching a class from Compact classes array, requires: getting oop at index in special objects array, and then getting oop from this object:
class := fetchPointer: ccIndex - 1 ofObject: (self fetchPointer: CompactClasses ofObject: specialObjectsOop)
A drawback of getting rid of compact classes is additional memory cost per each instance of classes which were compact.
Good points but perhaps for another thread?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Every bug you find is the last one.
On Tue, Jul 1, 2008 at 6:34 AM, tim Rowledge tim@rowledge.org wrote:
I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/89.html - do please comment on the relevant page, or even here.
Does every numbered prim also have a name or named equivalent?
I'll be rewriting the Kernel -* classes for the next few versions of SecureSqueak, so I could go through and replace these as I find them.
Also, if the compact classes only exist for space concerns, then as far as I'm concerned, they can go. Memory is cheap, and if you really want to save memory then write a 16-bit block-based VM (*)!
Gulik.
(*) http://gulik.pbwiki.com/Block-based+virtual+machine (disclaimer: random ideas only).
On 30-Jun-08, at 6:02 PM, Michael van der Gulik wrote:
On Tue, Jul 1, 2008 at 6:34 AM, tim Rowledge tim@rowledge.org wrote: I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/ 89.html - do please comment on the relevant page, or even here.
Does every numbered prim also have a name or named equivalent?
It's just a prim, ie a function somewhere.
I'm proposing that we name all those prims that are currently numbered - primitive 1 would become "primitiveAdd" in a similar manner to say "primitiveDisablePowerManager". This would allow all prims to be built as plugin components, thereby allowing on the fly replacement of those plugins (making due allowance for handling retained state etc, of course). (It's sometihng I've been wanting to see done since 1988; I had discussions with Alan Schiffman & Peter Deutsch about it way before I even move to ParcPlace). We would lose the primitive table, saving a 8 kb or so. Oh and the *really* obsoleteIndexPrimitiveTable, saving another 8Kb or so.
Some code gets simplified, some space is saved, some flexibility is added, some bits are freed up in the CM header. THe downside is someone has to do some work.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Advanced design: Upper management doesn't understand it.
2008/7/1 tim Rowledge tim@rowledge.org:
On 30-Jun-08, at 6:02 PM, Michael van der Gulik wrote:
On Tue, Jul 1, 2008 at 6:34 AM, tim Rowledge tim@rowledge.org wrote: I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/89.html - do please comment on the relevant page, or even here.
Does every numbered prim also have a name or named equivalent?
It's just a prim, ie a function somewhere.
I'm proposing that we name all those prims that are currently numbered - primitive 1 would become "primitiveAdd" in a similar manner to say "primitiveDisablePowerManager". This would allow all prims to be built as plugin components, thereby allowing on the fly replacement of those plugins (making due allowance for handling retained state etc, of course). (It's sometihng I've been wanting to see done since 1988; I had discussions with Alan Schiffman & Peter Deutsch about it way before I even move to ParcPlace). We would lose the primitive table, saving a 8 kb or so. Oh and the *really* obsoleteIndexPrimitiveTable, saving another 8Kb or so.
err... a primitive names, will add about same or even more space (in fact it will make VM and image both little bigger ). So its unlikely that new VM will have less memory footprint, but more likely that it would be simplified a bit comparing to old ways how it works with primitives.
Some code gets simplified, some space is saved, some flexibility is added, some bits are freed up in the CM header. THe downside is someone has to do some work.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Advanced design: Upper management doesn't understand it.
On 30-Jun-08, at 8:27 PM, Igor Stasenko wrote:
err... a primitive names, will add about same or even more space (in fact it will make VM and image both little bigger ). So its unlikely that new VM will have less memory footprint, but more likely that it would be simplified a bit comparing to old ways how it works with primitives.
Well, about 200 methods would get an extra literal (to put the name in) and about 200 symbols would be interned (for the names), so yes it would cost a little image space there. The VM size would be affected by what the C compiler did; sometimes simplifying code a little makes for a significant size change as CSE etc comes into play. The plugins are obviously different in structure to plain functions in an executable - no idea whether they are consistently bigger or smaller. Probably depends a bit on the OS? Of course, your image may well not use all the ex-numbered prims and thus may not load some of the new plugins. If the VM is properly (in my opinion) built to have all plugins external then the actual working size might be a fair bit smaller.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A)bort, R)etry, I)gnore, V)alium?
2008/7/1 tim Rowledge tim@rowledge.org:
On 30-Jun-08, at 8:27 PM, Igor Stasenko wrote:
err... a primitive names, will add about same or even more space (in fact it will make VM and image both little bigger ). So its unlikely that new VM will have less memory footprint, but more likely that it would be simplified a bit comparing to old ways how it works with primitives.
Well, about 200 methods would get an extra literal (to put the name in) and about 200 symbols would be interned (for the names), so yes it would cost a little image space there. The VM size would be affected by what the C compiler did; sometimes simplifying code a little makes for a significant size change as CSE etc comes into play.
You forgot , that primitive names are appear in VM as literals too. Just to be able to lookup for a primitive function by its name instead of index. But forget it.. Size is really not an issue, if we can focus on making VM more clean without obsolete stuff. The worst thing is image compatibility: you can't run images which using numbered primitives anymore. Or again, you would need to keep an index->name translation table for backward compatibility :)
The plugins are obviously different in structure to plain functions in an executable - no idea whether they are consistently bigger or smaller. Probably depends a bit on the OS? Of course, your image may well not use all the ex-numbered prims and thus may not load some of the new plugins. If the VM is properly (in my opinion) built to have all plugins external then the actual working size might be a fair bit smaller.
Why external? If you declare a primitive like:
<primitive: 'foo' module: ''>
it will look for a primitive in VM itself. No need in putting things in standalone plugin. There are many primitives which assume that they are in VM code, and also many others that assume that they built as internal only. There will be a lot of C discrepancy if you try to compile some plugins/primitives either as standalone plugin or external plugin :)
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A)bort, R)etry, I)gnore, V)alium?
You forgot , that primitive names are appear in VM as literals too. Just to be able to lookup for a primitive function by its name instead of index. But forget it.. Size is really not an issue, if we can focus on making VM more clean without obsolete stuff.
please this is important to move on and remove dust.
The worst thing is image compatibility: you can't run images which using numbered primitives anymore. Or again, you would need to keep an index->name translation table for backward compatibility :)
It's just a prim, ie a function somewhere.
I'm proposing that we name all those prims that are currently numbered - primitive 1 would become "primitiveAdd" in a similar manner to say "primitiveDisablePowerManager". This would allow all prims to be built as plugin components, thereby allowing on the fly replacement of those plugins (making due allowance for handling retained state etc, of course). (It's sometihng I've been wanting to see done since 1988; I had discussions with Alan Schiffman & Peter Deutsch about it way before I even move to ParcPlace). We would lose the primitive table, saving a 8 kb or so. Oh and the *really* obsoleteIndexPrimitiveTable, saving another 8Kb or so.
So do it.
Stef
Some code gets simplified, some space is saved, some flexibility is added, some bits are freed up in the CM header. THe downside is someone has to do some work.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Advanced design: Upper management doesn't understand it.
Sorry to reply to Steph's reply to Tim's original. I didn't get the original (because I had Anouncements checked by mistake inj my squeak-dev mail settings). I'm really replying to Tim's message here.
On Tue, Jul 1, 2008 at 9:27 AM, stephane ducasse stephane.ducasse@free.fr wrote:
It's just a prim, ie a function somewhere.
I'm proposing that we name all those prims that are currently numbered - primitive 1 would become "primitiveAdd" in a similar manner to say "primitiveDisablePowerManager". This would allow all prims to be built as plugin components, thereby allowing on the fly replacement of those plugins (making due allowance for handling retained state etc, of course). (It's sometihng I've been wanting to see done since 1988; I had discussions with Alan Schiffman & Peter Deutsch about it way before I even move to ParcPlace). We would lose the primitive table, saving a 8 kb or so. Oh and the *really* obsoleteIndexPrimitiveTable, saving another 8Kb or so.
Gettng rid of both obsolete tables is a really good thing (and I've already done it).
Getting rid of indexed primitives for the core does not save anything. There still has to be a table somewhere that maps the primitive name to the primitive function. All you do is replace a table mapping indices to functions with one mapping strings to functions. You can't reply on the OS to do the lookup because you won't be able to strip VM executables if you do and that really costs space.
But more importantly I think its pointless. The bulk of primitives in the system are named, and they're in plugins and that all works fine, even though lookup is a little involved and slow. But I see nothing wrong with the core language primitives being indexed. They are pretty well fixed with unchanged semantics and unchanged references in the system (e.g. SmallInteger>>#+ is where you find <primitive: 1> and has always been where you find it). The table is quick to index and simple.
That said I think its worth cutting down the set of indexed primitives to a minimum:
Object primtives (class, ==, ~~, become: Control primitives (blocks, perform, executeMethod, findUnwind, terminateTo) Process primitives (signal, wait, resume, suspend, yield) Accessing primitives (at:, at:put:, size, instVarAt:, instVarAt:put:) Instantiation primitives (new, new:, shallowCopy) SmallInteger primitives, Float primitives, LargeInteger primitives (although LargeInteger prims are arguable) Mirror primitives (_object:instVarAt:, _objectClass: etc which the executuon simulation machinery needs to use instead of instVarAt: & class etc to be able to simulate proxy execution correctly)
Primitives that absolutely should not be indexed primitives include things like (90 primitiveMousePoint) (91 primitiveTestDisplayDepth) "Blue Book: primitiveCursorLocPut" (92 primitiveSetDisplayMode) "Blue Book: primitiveCursorLink" (93 primitiveInputSemaphore) (94 primitiveGetNextEvent) "Blue Book: primitiveSampleInterval" (95 primitiveInputWord) (98 primitiveStoreImageSegment) (99 primitiveLoadImageSegment) (101 primitiveBeCursor) (102 primitiveBeDisplay) (103 primitiveScanCharacters) (105 primitiveStringReplace) (106 primitiveScreenSize) (107 primitiveMouseButtons) (108 primitiveKbdNext) (109 primitiveKbdPeek) (126 primitiveDeferDisplayUpdates) (127 primitiveShowDisplayRect) (133 primitiveSetInterruptKey) (134 primitiveInterruptSemaphore) (140 primitiveBeep) (141 primitiveClipboardText) (142 primitiveVMPath) (231 primitiveForceDisplayUpdate) (232 primitiveFormPrint) (233 primitiveSetFullScreen)
as these are to do with the broader context, not the core execution engine.
I would go further and rip out support for named primitives from the VM and put it up in the image where it belongs. I would build the machinery for looking up named primitives in the image and have a primitive that allowed one to slam a function address into a method.
For me the VM should include an execution engine, a small and fast FFI and nothing else. One needs some VM support to bootstrap the FFI. i.e. the VM must include a primitive to load a platform library and another to lookup a name in it. But other than that everything can be up in the image. When one invokes a method with a primitive that has not been bound to a function yet then the primitive fails and the image's primitive failure code looks up the primitive's name and retries the call or reports an error if the library or name can't be found.
The indexed primitive table should then come down to under 256 entries, (512 if you include primitiveLoadInstVar) leaving those bits to be used for more useful things like a larger literal range.
But that's just my opinion.
So do it.
Stef
Some code gets simplified, some space is saved, some flexibility is added, some bits are freed up in the CM header. THe downside is someone has to do some work.
:) back to my day job...
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Advanced design: Upper management doesn't understand it.
Am 01.07.2008 um 21:25 schrieb Eliot Miranda:
I would go further and rip out support for named primitives from the VM and put it up in the image where it belongs. I would build the machinery for looking up named primitives in the image and have a primitive that allowed one to slam a function address into a method.
For me the VM should include an execution engine, a small and fast FFI and nothing else. One needs some VM support to bootstrap the FFI. i.e. the VM must include a primitive to load a platform library and another to lookup a name in it. But other than that everything can be up in the image. When one invokes a method with a primitive that has not been bound to a function yet then the primitive fails and the image's primitive failure code looks up the primitive's name and retries the call or reports an error if the library or name can't be found.
Well, until now FFI was intentionally optional. Making it mandatory would immensely widen the Smalltalk-System interface, in fact, there would be no clearly defined interface anymore. Until now, Squeak is a relatively safe place, bad crashes are rare etc. Whenever you deal with FFI the next crash is only minutes away. Also, you could expect that the image would soon fill up with lots of platform-specific code. But more importantly, it would make sand-boxing impossible, as any code in the image can do anything to your machine if it has access to FFI.
So it would simplify the VM at the expense of added complexity in the image. Which may of course fit the bill for Croquet which already relies on FFI, and except for Etoys nobody really uses the sand boxing anyway.
- Bert -
On Tue, Jul 1, 2008 at 12:49 PM, Bert Freudenberg bert@freudenbergs.de wrote:
Am 01.07.2008 um 21:25 schrieb Eliot Miranda:
I would go further and rip out support for named primitives from the VM and put it up in the image where it belongs. I would build the machinery for looking up named primitives in the image and have a primitive that allowed one to slam a function address into a method.
For me the VM should include an execution engine, a small and fast FFI and nothing else. One needs some VM support to bootstrap the FFI. i.e. the VM must include a primitive to load a platform library and another to lookup a name in it. But other than that everything can be up in the image. When one invokes a method with a primitive that has not been bound to a function yet then the primitive fails and the image's primitive failure code looks up the primitive's name and retries the call or reports an error if the library or name can't be found.
Well, until now FFI was intentionally optional. Making it mandatory would immensely widen the Smalltalk-System interface, in fact, there would be no clearly defined interface anymore. Until now, Squeak is a relatively safe place, bad crashes are rare etc. Whenever you deal with FFI the next crash is only minutes away. Also, you could expect that the image would soon fill up with lots of platform-specific code. But more importantly, it would make sand-boxing impossible, as any code in the image can do anything to your machine if it has access to FFI.
So it would simplify the VM at the expense of added complexity in the image. Which may of course fit the bill for Croquet which already relies on FFI, and except for Etoys nobody really uses the sand boxing anyway.
One doesn't have to *use* the FFI. If the FFI isn't exposed via a primitive then no FFI. One can still have named primitives supported by the image and not by the VM and not use the FFI. To call a named primitive in a primitive plugin the following sequence occurs:
the method containing a named primitive spec is activated and the primitive call fails because its function pointer is null. the failure code extracts the plugin name and invokes a primitive to load the plugin library the failure code extracts the primitive name and uses the lookup primitive to find the function in the loaded plugin library the failure code uses a primitive to slam the function pointer into the method the failure code uses the executeMethodWithArgs primitive to retry the bound named primitive method
So the FFI is an optional extra. One needs four primitives, load library, lookup name in library, insert primitive function pointer. and executemethodWithArgs (thanks Tim!). Slamming the function into the method could also be done using, say, objectAt:.
So one can still have a nice small safe VM and have no direct support for named primitives in the VM.
- Bert -
On 1-Jul-08, at 1:20 PM, Eliot Miranda wrote:
One doesn't have to *use* the FFI. If the FFI isn't exposed via a primitive then no FFI. One can still have named primitives supported by the image and not by the VM and not use the FFI. To call a named primitive in a primitive plugin the following sequence occurs:
the method containing a named primitive spec is activated and the primitive call fails because its function pointer is null. the failure code extracts the plugin name and invokes a primitive to load the plugin library the failure code extracts the primitive name and uses the lookup primitive to find the function in the loaded plugin library the failure code uses a primitive to slam the function pointer into the method the failure code uses the executeMethodWithArgs primitive to retry the bound named primitive method
So the FFI is an optional extra. One needs four primitives, load library, lookup name in library, insert primitive function pointer. and executemethodWithArgs (thanks Tim!). Slamming the function into the method could also be done using, say, objectAt:.
So one can still have a nice small safe VM and have no direct support for named primitives in the VM.
Leaving aside my like of all-named-prims, I *like* this enhancement to the support for named prims.
It would slightly complicate the post-prim-call code in each method because you would need to handle the failed-to-find-prim case as well as all the prim-failed cases. It would be helped by an old idea that I'm pretty sure eliot has plans for anyway (as indeed I have written about a few times) to add primitive error return values. For those that haven't heard of them, this is just a way of having the first temp in a context involved in a prim call be a slot for a return value from the prim if any error occurs. This means you can actually know wtf went wrong instead of guessing - "ooh, was the first arg a SmallInteger? Um, if not, was it a Float that might round nicley? Err, was the second arg a ByteArray with the first byte = 255?" etc. Instead you get "ah, errorValue was 1, so we have to explode the reactor core in order to stop the Dreen from eating the children". Much nicer.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim CITOKATE - Criticism is the Only Known Antidote to Error
On Tue, Jul 1, 2008 at 1:34 PM, tim Rowledge tim@rowledge.org wrote:
On 1-Jul-08, at 1:20 PM, Eliot Miranda wrote:
One doesn't have to *use* the FFI. If the FFI isn't exposed via a primitive then no FFI. One can still have named primitives supported by the image and not by the VM and not use the FFI. To call a named primitive in a primitive plugin the following sequence occurs:
the method containing a named primitive spec is activated and the primitive call fails because its function pointer is null. the failure code extracts the plugin name and invokes a primitive to load the plugin library the failure code extracts the primitive name and uses the lookup primitive to find the function in the loaded plugin library the failure code uses a primitive to slam the function pointer into the method the failure code uses the executeMethodWithArgs primitive to retry the bound named primitive method
So the FFI is an optional extra. One needs four primitives, load library, lookup name in library, insert primitive function pointer. and executemethodWithArgs (thanks Tim!). Slamming the function into the method could also be done using, say, objectAt:.
So one can still have a nice small safe VM and have no direct support for named primitives in the VM.
Leaving aside my like of all-named-prims, I *like* this enhancement to the support for named prims.
It would slightly complicate the post-prim-call code in each method because you would need to handle the failed-to-find-prim case as well as all the prim-failed cases. It would be helped by an old idea that I'm pretty sure eliot has plans for anyway (as indeed I have written about a few times) to add primitive error return values. For those that haven't heard of them, this is just a way of having the first temp in a context involved in a prim call be a slot for a return value from the prim if any error occurs. This means you can actually know wtf went wrong instead of guessing - "ooh, was the first arg a SmallInteger? Um, if not, was it a Float that might round nicley? Err, was the second arg a ByteArray with the first byte = 255?" etc. Instead you get "ah, errorValue was 1, so we have to explode the reactor core in order to stop the Dreen from eating the children". Much nicer.
I did primitive error codes at Cadence and they'll very probably b making it into Cog real soon now. They're simpler than VisualWorks', being only symbols. So extracting more information, such as a related error code requires a subsequent call. But I think the work I'm doing right now on eliminating pushRemappableOop:/popRemappableOop will enable me to have a structured object with a name and parameters, which is more generally useful.
A nice thing is that the code is forwards and backewards compatible. One can use the VM to run older images. One can run images that contain the primitive error code on older VMs, where one simply gets a nil error code on primitive failure.
I did primitive error codes at Cadence and they'll very probably b making it into Cog real soon now. They're simpler than VisualWorks', being only symbols. So extracting more information, such as a related error code requires a subsequent call. But I think the work I'm doing right now on eliminating pushRemappableOop:/ popRemappableOop will enable me to have a structured object with a name and parameters, which is more generally useful.
So.... what is the anticipated way that will work? I'm not sure I see the need for a two step process anyway.
The version we did at Interval in 98 was very simple, providing a call for prims to stuff some object (we used SmallInts in prcatice, but anything was allowed) into the first slot in the context that got activated on the fail. We extended the primitive declaration pragma to allow optional naming of the tempvar - default was 'errorValue' I think - and that was it. Code following the prim call could use or ignore the temp.
A nice thing is that the code is forwards and backewards compatible. One can use the VM to run older images. One can run images that contain the primitive error code on older VMs, where one simply gets a nil error code on primitive failure.
Sure, if the temp is used by return-code-unaware methods in an older image then it can't make much difference because it would be assumed nil as normal. Running new images on older VMs isn't something we've ever much bothered with for Squeak so don't worry about it.
I'm going to guess that "work I'm doing right now on eliminating pushRemappableOop:/popRemappableOop" relates to getting allocations out of primitives as much as possible, right? That would be nice.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "Bother," said Pooh, reading his bank statement from Barings.
On Tue, Jul 1, 2008 at 2:23 PM, tim Rowledge tim@rowledge.org wrote:
I did primitive error codes at Cadence and they'll very probably b making it into Cog real soon now. They're simpler than VisualWorks', being only symbols. So extracting more information, such as a related error code requires a subsequent call. But I think the work I'm doing right now on eliminating pushRemappableOop:/popRemappableOop will enable me to have a structured object with a name and parameters, which is more generally useful.
So.... what is the anticipated way that will work? I'm not sure I see the need for a two step process anyway.
The version we did at Interval in 98 was very simple, providing a call for prims to stuff some object (we used SmallInts in prcatice, but anything was allowed) into the first slot in the context that got activated on the fail. We extended the primitive declaration pragma to allow optional naming of the tempvar - default was 'errorValue' I think - and that was it. Code following the prim call could use or ignore the temp.
The compiler is modified to recognise <primitive: integer errorCode: identifier> and <primitive: identifier module: identifier errorCode: identifier> and convert this into one additional temp and generate a long storeTemp as the first instruction of the method.
The VM is modified to, on primitive failure, check for the long storeTemp and stores the error code if it sees it.
Old images lack the storeTemp so the VM does not store the value. Old VMs do perform the storeTemp, but because the storeTemp is storing into the stack top it is effectively a noop. e.g. if the primitive looks like:
*primitiveAdoptInstance:* *anObject*** <*primitive:*** 160 error: *ec***> *ec*** == *#'no modification'*** ifTrue: [*^****NoModificationError*** signal: *self*** message: (*Message*** selector: *#primitiveAdoptInstance:*** arguments: {*anObject***})]. *self*** primitiveFailed
then the header says the method has 2 temps (anObject & ec). The VM initializes the context with stackp pointing at the second temp (ec). The merthod starts with a long storeTemp: 2. So the nil ec gets stored into itself. The compiler generates a long storeTemp to make it quicker to check for the method starting with a storeTemp.
A nice thing is that the code is forwards and backewards compatible. One can use the VM to run older images. One can run images that contain the primitive error code on older VMs, where one simply gets a nil error code on primitive failure.
Sure, if the temp is used by return-code-unaware methods in an older
image then it can't make much difference because it would be assumed nil as normal. Running new images on older VMs isn't something we've ever much bothered with for Squeak so don't worry about it.
I'm going to guess that "work I'm doing right now on eliminating pushRemappableOop:/popRemappableOop" relates to getting allocations out of primitives as much as possible, right? That would be nice.
No. This relates to not having to use pushRemappableOop:/popRemappableOop because it is extremely error-prone. For example can you spot the bug in this:
*primitivePerform* | *performSelector*** *newReceiver*** *selectorIndex*** *lookupClass*** *performMethod*** | *performSelector*** *:=* *messageSelector***. *performMethod*** *:=* *newMethod***. *messageSelector*** *:=* *self*** stackValue: *argumentCount*** - 1. *newReceiver*** *:=* *self*** stackValue: *argumentCount***.
*"NOTE: the following lookup may fail and be converted to #doesNotUnderstand:, so we must adjust argumentCount and slide args now, so that would work."***
*"Slide arguments down over selector"*** *argumentCount*** *:=* *argumentCount*** - 1. *selectorIndex*** *:=* *self*** stackPointerIndex - *argumentCount***. *self*** transfer: *argumentCount*** fromIndex: *selectorIndex*** + 1 ofObject: *activeContext*** toIndex: *selectorIndex*** ofObject: *activeContext***. *self*** pop: 1. *lookupClass*** *:=* *self*** fetchClassOf: *newReceiver***. *self*** findNewMethodInClass: *lookupClass***.
*"Only test CompiledMethods for argument count - other objects will have to take their chances"*** (*self*** isCompiledMethod: *newMethod***) ifTrue: [*self*** success: (*self*** argumentCountOf: *newMethod***) = *argumentCount***].
*self*** successful ifTrue: [*self*** executeNewMethodFromCache. *"Recursive xeq affects successFlag"*** *self*** initPrimCall] ifFalse: [*"Slide the args back up (sigh) and re-insert the selector. "*** 1 to: *argumentCount*** do: [:*i*** | *self*** storePointer: *argumentCount*** - *i*** + 1 + * selectorIndex*** ofObject: *activeContext*** withValue: (*self*** fetchPointer: *argumentCount*** - *i*** + *selectorIndex*** ofObject: *activeContext***)]. *self*** unPop: 1. *self*** storePointer: *selectorIndex*** ofObject: *activeContext*** withValue: *messageSelector***. *argumentCount*** *:=* *argumentCount*** + 1. *newMethod*** *:=* *performMethod***. *messageSelector*** *:=* *performSelector***]
Don't look further. try and spot the bug first.
It starts with *performSelector*** *:=* *messageSelector***. *performMethod*** *:=* *newMethod***. and ends with *newMethod*** *:=* *performMethod***. *messageSelector*** *:=* *performSelector which is only executed if there is a doesNotUnderstand:.* A doesNotUnderstand: may cause a GC in createActualMessage as it creates the message argument for doesNotUnderstand:. So this bug only bytes if a perform is not understood when memory is on the verge of exhaustion.
In the context-to-stack-mapping VM I'm working on the VM may do up to a page worth of context allocations on e.g. trying to store the sender of a context. Rather than try and plug all the holes it is much safer to restrict garbage collection to happening between bytecodes (probably on sends and backward branches). To do this the GC has to maintain a reserve for the VM which is about the size of two stack pages, or probably 2k bytes, and the VM has to defer incremental collections from allocations until the send or backward branch following.
On 1-Jul-08, at 3:07 PM, Eliot Miranda wrote:
The compiler is modified to recognise <primitive: integer errorCode: identifier> and <primitive: identifier module: identifier errorCode: identifier> and convert this into one additional temp
Yup, that's what we did...
and generate a long storeTemp as the first instruction of the method.
The VM is modified to, on primitive failure, check for the long storeTemp and stores the error code if it sees it.
Old images lack the storeTemp so the VM does not store the value. Old VMs do perform the storeTemp, but because the storeTemp is storing into the stack top it is effectively a noop. e.g. if the primitive looks like:
...but that is very clever as a way of handling back-compat. We didn't have to care about that at all so it got no attention.
I'm going to guess that "work I'm doing right now on eliminating pushRemappableOop:/popRemappableOop" relates to getting allocations out of primitives as much as possible, right? That would be nice.
No. This relates to not having to use pushRemappableOop:/ popRemappableOop because it is extremely error-prone. For example can you spot the bug in this:
primitivePerform
[snip]
A doesNotUnderstand: may cause a GC in createActualMessage as it creates the message argument for doesNotUnderstand:. So this bug only bytes if a perform is not understood when memory is on the verge of exhaustion.
Ouch! What fun.
In the context-to-stack-mapping VM I'm working on the VM may do up to a page worth of context allocations on e.g. trying to store the sender of a context. Rather than try and plug all the holes it is much safer to restrict garbage collection to happening between bytecodes (probably on sends and backward branches). To do this the GC has to maintain a reserve for the VM which is about the size of two stack pages, or probably 2k bytes, and the VM has to defer incremental collections from allocations until the send or backward branch following.
OK; be aware that there is a pathological case that might impact your code in this area, mostly restricted to non-virtual memory systems. Somewhere in the GC code it will try to grab more memory for forwarding blocks and if none is provided by the OS (as in RISC OS for example) then some of the reserved space will be stolen *without* proper checks and notifications. This can result in the system trying to handle a lowSpace with only a few hundred bytes of free memory. It doesn't go so well after that.... I've been trying to find relevant emails to illustrate better but no luck so far. I'm reasonably sure we never came up with a good solution but the problem surfaced about 4 years ago and just possibly got fixed somewhere.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "Bother" said Piglet, as Pooh smeared him in honey.
OK; be aware that there is a pathological case that might impact your code in this area, mostly restricted to non-virtual memory systems. Somewhere in the GC code it will try to grab more memory for forwarding blocks and if none is provided by the OS (as in RISC OS for example) then some of the reserved space will be stolen *without* proper checks and notifications. This can result in the system trying to handle a lowSpace with only a few hundred bytes of free memory. It doesn't go so well after that.... I've been trying to find relevant emails to illustrate better but no luck so far. I'm reasonably sure we never came up with a good solution but the problem surfaced about 4 years ago and just possibly got fixed somewhere.
Ah, well let me see if I can find them then.
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
Ok, well the discussions date from late April 2005. Some are below.
On Jul 1, 2008, at 11:46 PM, tim Rowledge wrote:
OK; be aware that there is a pathological case that might impact your code in this area, mostly restricted to non-virtual memory systems. Somewhere in the GC code it will try to grab more memory for forwarding blocks and if none is provided by the OS (as in RISC OS for example) then some of the reserved space will be stolen *without* proper checks and notifications. This can result in the system trying to handle a lowSpace with only a few hundred bytes of free memory. It doesn't go so well after that.... I've been trying to find relevant emails to illustrate better but no luck so far. I'm reasonably sure we never came up with a good solution but the problem surfaced about 4 years ago and just possibly got fixed somewhere.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "Bother" said Piglet, as Pooh smeared him in honey.
From: johnmci@smalltalkconsulting.com Subject: initializeMemoryFirstFree
Date: April 28, 2005 10:03:36 PM PDT (CA)
To: tim@sumeru.stanford.edu
initializeMemoryFirstFree Some thoughts.
a) We must have oh say 100,000 bytes free, reduce fwdBlockBytes if need be from it's optimal calculation. b) fwdBlockBytes must be > 100,000, if not die, this is an arbitrary value, not sure what min really should be..
fwdBlockBytes = foo->totalObjectCount & 4294967292U; if (!((foo->memoryLimit - fwdBlockBytes - 100000) >= (firstFree + BaseHeaderSize))) { fwdBlockBytes = foo->memoryLimit - (firstFree + BaseHeaderSize) - 100000; } if (fwdBlockBytes < 0) error("Death no memory");
So this does allow me to see and get the dialog, but then I don't think the right process gets stopped. since I don't have control and I can't click/keyboard and we die. error("Death no memory"); Is it suspending the UI process?
200,000 is also not much headroom, try 1,000,000 for better safety.
&
Code at bottom to test with.
freememory fwdblocksBytes, endOfmemory memoryLimit
254204 759056 128479936 129238992 206044 759056 128479936 129238992 202500 759056 128479936 129238992
Fall under 200,000 gcmove moves some bytes about so we see 5 iterations. Note how fwdblocks changed from 759056 to 859152 and we cap that and give back 102404 bytes of free
102404 859152 128379840 129238992 102404 859152 128379840 129238992 102404 877184 128361808 129238992 102404 877184 128361808 129238992 102404 877184 128361808 129238992
JMMJMM TOSS signal LOWSPACE Yes under the 200,000 so do the signal. JMMJMM BEEEEEEEEEEEPPPPPPPPP This is the lowspace process waking up and running. JMMJMM primitiveSignalAtBytesLeft called lowspace process changed threshold to zero.
watch how we carve away at fwdblock bytes keeping 100K + 4 bytes around
102404 865684 128373308 129238992 102404 871032 128367960 129238992 102404 871484 128367508 129238992 102404 865364 128373628 129238992 102404 871800 128367192 129238992 102404 869176 128369816 129238992 102404 873180 128365812 129238992 102404 871452 128367540 129238992 102404 864308 128374684 129238992 102404 864120 128374872 129238992 102404 870744 128368248 129238992 102404 870164 128368828 129238992 102404 870664 128368328 129238992 102404 866160 128372832 129238992 102404 869940 128369052 129238992 102404 868352 128370640 129238992 102404 864672 128374320 129238992 102404 870100 128368892 129238992 102404 822776 128416216 129238992 102404 872888 128366104 129238992 102404 872348 128366644 129238992 102404 871832 128367160 129238992 102404 870560 128368432 129238992 102404 869212 128369780 129238992 102404 867052 128371940 129238992 102404 867372 128371620 129238992 102404 867116 128371876 129238992 102404 869252 128369740 129238992 102404 867344 128371648 129238992 102404 866688 128372304 129238992 102404 864384 128374608 129238992 102404 863436 128375556 129238992 102404 860632 128378360 129238992 102404 858804 128380188 129238992 102404 860312 128378680 129238992 102404 859980 128379012 129238992 102404 860172 128378820 129238992 102404 859692 128379300 129238992 102404 860108 128378884 129238992 102404 859524 128379468 129238992 102404 858408 128380584 129238992 102404 860200 128378792 129238992 102404 860200 128378792 129238992 102404 859804 128379188 129238992 102404 859424 128379568 129238992 102404 856880 128382112 129238992 102404 854388 128384604 129238992 102404 855116 128383876 129238992 102404 854748 128384244 129238992 102404 855036 128383956 129238992 102404 855688 128383304 129238992 102404 853028 128385964 129238992 102404 852592 128386400 129238992 102404 852292 128386700 129238992 102404 852940 128386052 129238992 102404 852552 128386440 129238992 102404 852624 128386368 129238992 102404 852732 128386260 129238992 102404 852036 128386956 129238992 102404 855304 128383688 129238992 102404 855000 128383992 129238992 102404 854744 128384248 129238992 102404 854668 128384324 129238992 102404 855340 128383652 129238992 102404 856028 128382964 129238992 102404 853628 128385364 129238992 102404 854504 128384488 129238992 102404 854248 128384744 129238992 102404 854852 128384140 129238992 102404 853556 128385436 129238992 102404 854204 128384788 129238992 102404 852532 128386460 129238992 102404 853840 128385152 129238992 102404 853564 128385428 129238992 102404 853088 128385904 129238992 102404 854196 128384796 129238992 102404 854164 128384828 129238992 102404 854176 128384816 129238992 102404 852560 128386432 129238992 102404 854456 128384536 129238992 102404 852580 128386412 129238992 102404 850528 128388464 129238992 102404 839720 128399272 129238992 102404 848844 128390148 129238992 102404 847364 128391628 129238992 102404 847684 128391308 129238992 102404 847044 128391948 129238992 102404 846788 128392204 129238992 102404 847236 128391756 129238992 102404 850816 128388176 129238992 102404 846164 128392828 129238992 102404 839576 128399416 129238992 102404 848688 128390304 129238992 102404 849004 128389988 129238992 102404 848696 128390296 129238992 102404 838656 128400336 129238992 102404 851656 128387336 129238992 102404 797844 128441148 129238992 102404 793268 128445724 129238992 102404 791064 128447928 129238992 102404 841436 128397556 129238992 102404 840376 128398616 129238992 102404 839528 128399464 129238992 102404 786560 128452432 129238992 102404 835652 128403340 129238992 102404 835652 128403340 129238992 102404 783732 128455260 129238992 102404 782336 128456656 129238992 102404 836676 128402316 129238992 102404 832308 128406684 129238992 102404 832068 128406924 129238992 102404 832636 128406356 129238992 102404 833612 128405380 129238992 102404 834320 128404672 129238992 102404 833980 128405012 129238992 102404 835372 128403620 129238992 102404 780488 128458504 129238992 102404 778388 128460604 129238992 102404 826284 128412708 129238992 102404 827288 128411704 129238992 102404 827288 128411704 129238992 102404 827416 128411576 129238992 102404 827416 128411576 129238992 102404 827544 128411448 129238992 102404 827224 128411768 129238992 102404 828588 128410404 129238992 102404 826084 128412908 129238992 102404 774280 128464712 129238992 102404 837180 128401812 129238992 102404 840692 128398300 129238992 102404 841520 128397472 129238992 102404 796540 128442452 129238992 102404 748380 128490612 129238992 102404 700220 128538772 129238992 102404 652060 128586932 129238992 102404 603900 128635092 129238992 102404 555740 128683252 129238992 102404 507580 128731412 129238992 102404 459420 128779572 129238992 102404 411260 128827732 129238992 102404 363100 128875892 129238992 102404 314940 128924052 129238992 102404 266780 128972212 129238992 102404 218620 129020372 129238992 102404 170460 129068532 129238992 102404 122300 129116692 129238992 102404 74140 129164852 129238992
Grind down forward blocks to 32K, note that free goes to 95616, then 47456, then 2500.
95616 32768 129206224 129238992 47456 32768 129206224 129238992 2500 32768 129206224 129238992
LOTS of these....
2500 32768 129206224 129238992 2500 32768 129206224 129238992 16720 32768 129206224 129238992 2592 32768 129206224 129238992 2496 32768 129206224 129238992 2496 32768 129206224 129238992 2680 32768 129206224 129238992 2588 32768 129206224 129238992 2484 32768 129206224 129238992 2668 32768 129206224 129238992 2576 32768 129206224 129238992 2472 32768 129206224 129238992 2656 32768 129206224 129238992 2564 32768 129206224 129238992 2460 32768 129206224 129238992 2644 32768 129206224 129238992 2552 32768 129206224 129238992 2448 32768 129206224 129238992 2632 32768 129206224 129238992 2540 32768 129206224 129238992 2436 32768 129206224 129238992 2620 32768 129206224 129238992 2528 32768 129206224 129238992 2424 32768 129206224 129238992 2608 32768 129206224 129238992 2516 32768 129206224 129238992 2412 32768 129206224 129238992 2596 32768 129206224 129238992 2504 32768 129206224 129238992 2400 32768 129206224 129238992 2584 32768 129206224 129238992 2492 32768 129206224 129238992 2492 32768 129206224 129238992 2400 32768 129206224 129238992 2400 32768 129206224 129238992 2308 32768 129206224 129238992 2308 32768 129206224 129238992 2216 32768 129206224 129238992 2216 32768 129206224 129238992 2124 32768 129206224 129238992 2124 32768 129206224 129238992 2032 32768 129206224 129238992 2032 32768 129206224 129238992 1940 32768 129206224 129238992 1940 32768 129206224 129238992 1848 32768 129206224 129238992 1848 32768 129206224 129238992 1756 32768 129206224 129238992 1756 32768 129206224 129238992 1664 32768 129206224 129238992 1664 32768 129206224 129238992 1572 32768 129206224 129238992 1572 32768 129206224 129238992 1480 32768 129206224 129238992 1480 32768 129206224 129238992 1388 32768 129206224 129238992 1388 32768 129206224 129238992 1296 32768 129206224 129238992 1296 32768 129206224 129238992 1204 32768 129206224 129238992 1204 32768 129206224 129238992 1112 32768 129206224 129238992 1112 32768 129206224 129238992 1020 32768 129206224 129238992 1020 32768 129206224 129238992 928 32768 129206224 129238992 928 32768 129206224 129238992 836 32768 129206224 129238992 836 32768 129206224 129238992 744 32768 129206224 129238992 744 32768 129206224 129238992 652 32768 129206224 129238992 652 32768 129206224 129238992 560 32768 129206224 129238992 560 32768 129206224 129238992 468 32768 129206224 129238992 468 32768 129206224 129238992 376 32768 129206224 129238992 376 32768 129206224 129238992 284 32768 129206224 129238992 284 32768 129206224 129238992 192 32768 129206224 129238992 192 32768 129206224 129238992 8 32768 129206224 129238992 8 32768 129206224 129238992
And we die no space to allocate context record....
int initializeMemoryFirstFree(int firstFree) { register struct foo * foo = &fum; int fwdBlockBytes;
fwdBlockBytes = foo->totalObjectCount & 4294967292U; if (!((foo->memoryLimit - fwdBlockBytes) >= ((firstFree + BaseHeaderSize) + (100 * 1024)))) { fwdBlockBytes = (foo->memoryLimit - (firstFree + BaseHeaderSize)) - (100 * 1024); } if (fwdBlockBytes < (32 * 1024)) { fwdBlockBytes = 32 * 1024; if (!((foo->memoryLimit - fwdBlockBytes) >= (firstFree + BaseHeaderSize))) { fwdBlockBytes = foo->memoryLimit - (firstFree + BaseHeaderSize); } } foo->endOfMemory = foo->memoryLimit - fwdBlockBytes; foo->freeBlock = firstFree; /* begin setSizeOfFree:to: */ longAtput(foo->freeBlock, ((foo->endOfMemory - firstFree) & AllButTypeMask) | HeaderTypeFree); /* begin setSizeOfFree:to: */ longAtput(foo->endOfMemory, (BaseHeaderSize & AllButTypeMask) | HeaderTypeFree); if (DoAssertionChecks) { if (!((foo->freeBlock < foo->endOfMemory) && (foo->endOfMemory < foo-
memoryLimit))) {
error("error in free space computation"); } if (!((foo->endOfMemory + (foo->headerTypeBytes[(longAt(foo-
endOfMemory)) & TypeMask])) == foo->endOfMemory)) {
error("header format must have changed"); } if (!((objectAfter(foo->freeBlock)) == foo->endOfMemory)) { error("free block not properly initialized"); } } }
...
at 200,000 we don't see the signal, because after the full GC at the 200K boundary we gobble up all the memory for the fwdtable, leaving 4 or 6 bytes left. Then we immediately die because we can't allocate the next context record. For a 64MB memory block we've about a MB or so tied up in fwdspace, certainly it wants over 200K after the fullGC.
As you noticed changing the limit to be larger, I think 400,000 was about the cutoff for my test case gave me the debugger. Let me check at 512MB.
On Apr 29, 2005, at 3:29 PM, Tim Rowledge wrote:
OK, I have some slightly off-to-the-side news on this.
A VM with the store-errant-process-oop + an image with dtl's code to make use of that is much more stable if the lowSpaceThreshold is raised a good bit. What is likely happening to your system is that the lowspace is being signalled but the long faringabout with gc attempts means the event ticler is interrupted instead of the UI process 'at fault'. Thus the bad boy goes ahead and messes you up. With the fix in things are a bit more sensible in that the 'right' proces is interupted. Of course, not much help if lots of processes are using up memory!
Still, a code-recursion test - ie fill up memory with contexts - is passable with 'only' half-meg of lowSpaceThreshold. Likewise the fill memory with bitblts. It seems to need 1mb to survive the fill with Links though. This is at least encouraging enouhg to maybe help us find the problem with my tree walker changes that lead down this path in the first pace.
The essence of the problem is that we really wants one byte per object available to the fwdTable. In the worst case, we could have almost all of OM filled with plain Objects - ie 8byte chunks - and so would really wish for ~12% of available memory reserved. So much for direct pointers 'saving the wastedspace of an object table', eh?
more later but I have to dash out for the dog's massage appointment. Really.
tim
Tim Rowledge, tim@sumeru.stanford.edu, http://sumeru.stanford.edu/tim Strange OpCodes: SEXI: Sign EXtend Integer
Apr 30th,, 2005 John M McIntosh wrote:
On Apr 30, 2005, at 8:00 PM, Andreas Raab wrote:
Hi Tim -
After having problem trying to debug some TK4 code that blew up with lowspace problems but never let me catch and debug, I spent some time adding the lowspace-process stuff we recently discussed. I had to make a few alterations to match it up with the latest 64bit clean code but no problems with that part.
What am I missing? I don't remember low-space stuff - I only remember interrupt-related stuff.
There was a mantis bug about low-space issues and some patchs to record which process caused the lowspace signal. Mind this in my opinion is wrong.
Depending upon the exact size of object memory in use the 200kb used as the lowSpaceThreshold can be gobbled up in one swallow by the initializeMemoryFirstFree: method making sure there is a byte per object that survived the markPhase. In using useUpMemory we can get to having 4 bytes of free space when the next allocate is attempted.... Ka-Boom.
Well, so don't eat up the memory. There is no reason why initializeMemoryFirstFree: would have to reserve that much memory - like the comment says the reserve "should" be chosen so that compactions can be done in one pass but there is absolutely no such requirement. Multi-pass compactions have happened in the past and there is nothing wrong with them (in a low-space situation).
This assumes that we really need to have one byte per object of course. The original rationale was to keep the number of compact loops down to eight (see Dan's comment in initializeMemoryFirstFree:) for Alan's large demo image. The nicest solution would be to come up with a way to do our GC & compacting without needing any extra space. Commence headscratching now... John suggested making sure the fwd gets less than the byte-per-object if things are tight, and accpting the extra compaction loops.
Yes. That's the only reasonable way of dealing with it.
What happens is the fwdblocks calculation grabs all the available free memory when it's recalculated after the full GC, the check for this condition actually backs it off to allow one object header free, 4 or 6 bytes I believe, usually you die right away because someone attempts to allocate a new context record and we don't have 98ish bytes free. I gave Tim a change set that attempts to maximise freespace to 100K by reducing fwdblocks down to 32k, once you hit the 32k limit freespace then heads towards zero of course.
Note that once freespace goes under 200,000 we do signal the lowspace semaphore btw.
These changes do require a VM change, but we did notice as Tim points out if you increase the lowspace threshold, say to 1MB in my testing the other night we'll get the semaphore signaled with a current VM, this would not occur before in an unaltered VM.
Bad news- consider Tweak. With lots of processes whizzing away, merely stopping the one that did the allocation and triggered the lowspace is not going to be much good. Stopping everything except the utterly essential stuff to debug the lowspace will be needed. Probably.
Uh, oh. Are you telling me that the "low space stuff" you are referring to above actually suspends the process that triggers the low-space condition? Bad, bad, bad idea. Ever considered that this might be the timer process? The finalization process? Low-space is *not* a per-process condition; suspending the currently running process is something that should be done with great care (if at all).
Please, don't suspend that process - put it away for the image to examine but by all means do NOT suspend it. If you give me a nice clean semaphore signal for Tweak to handle a low-space condition I know perfectly well what to do but if you just suspend a random process which may have absolutely nothing with the low space condition, then, yes, we are in trouble (if this were a tweak scheduler process you'd be totally hosed).
Tim and I were considering to suspend all user processes and others we don't have knowledge of being untouchable, then I pointed out Tweak spawns all these process, what do we do about them? Certainly we can call something to say lowspace Mr Tweak beware...
The Process Browser logic has a table identifying processes of the VM, we assume a process the user created is causing the problem. The earlier fix suggested to stop the process that was running when the lowspace condition occurred, but I doubt you can 100% say that is the process in question and could as you know be the finalization process or other critical task. Still this is not harmful because the evil process in question is still running and will terminate your image in short order.
Cheers,
- Andreas
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
2008/7/2 John M McIntosh johnmci@smalltalkconsulting.com:
[ big snip.... ]
Few thoughts, about how VM could determine what process(es) is memory hungry to kill them w/o mercy, leaving rest intact and avoid dying:
add 'memory policy' slot to Process instance, which can tell VM, is given process can be killed w/o damaging most critical parts of image.
Lets say, a value 0 - do not kill under any circumstances, and values
0 is candidates for killing.
Then we could assign each process a corresponding policy, and when it goes to look for a candidate for killing, VM picks a process with higher 'memory policy' slot value.
The we could have most important processes have value = 0, and UI and forked processes can have value = 100+. More important than UI processes can be in the middle between 0 and 100. Another rule - automatically increment value of fork(ed) process could be useful too. In this way given slot could reflect fork depth so, one can potentially track process spawn chains.
How do you think, is adding new slot to Process instance worth stability? :)
One more thing, which could be useful is informing image about such unconditional termination event. Either by registering a semaphore which will be signaled upon such event, or adding some kind of #onTerminate message, so dying process can handle such event by itself and perform necessary countermeasures :) Or is there already any means how to inform another process , that some other process is terminated?
From my OS practice: OS treats processes as waitable object, so one
could apply following:
proc run. proc wait. "wait till terminated"
This is easy to do, if process can be associated with semaphore instance. And then , in case it terminates, it signals semaphore.
On Wed, Jul 2, 2008 at 3:32 AM, Igor Stasenko siguctua@gmail.com wrote:
2008/7/2 John M McIntosh johnmci@smalltalkconsulting.com:
[ big snip.... ]
Few thoughts, about how VM could determine what process(es) is memory hungry to kill them w/o mercy, leaving rest intact and avoid dying:
add 'memory policy' slot to Process instance, which can tell VM, is given process can be killed w/o damaging most critical parts of image.
There is no need for this to be in the VM. Instead the process that runs once the LowSpaceSemaphore is signalled should be of a very high priority (probably above finalization but below timing priority). It then enumerates the runnable processes (processes waiting on semaphores are resumably not the ones chewing up memory). It then suspends any processes on the runnable queues that meet certain criteria.
Something that *could* be in the VM is accounting of how much space a process consumes. Add a slot to each process known to the VM called e.g. slotsAllocated. The VM computes the slots allocated between context switches. This is cheap to compute because it can compute how much space was allocated since the last garbage collection or process switch simply by subtracting the allocation pointer at the end of the previous GC or process switch from the current allocation pointer. The slots allocated since the last process switch is added to the slot in the old process and the slots allocated count zeroed on each context switch. We then have an accurate measure of how much space each process has allocated.
Wen the low space process runs it simply examines the runnable processes, checking their space allocation. It can either maintain a per-process allocation rate by computing the amount allocated since the last low space signal, or it can zero the per-process allocation count at each low space signal.
This is much more flexible and less arbitrary than having the VM do it.
Similarly a VM that does context-to-stack mapping should be able to cheaply maintain a stack size count per process since it only has to increase or decrease the current process's stack size on each stack page overfow/underflow and context switch. Traversing a stack page's frames to get an accurate count of the number of "contexts" on each page is pretty quick (just a walk of the frame pointer->caller frame pointer chain). Getting an approximation by dividing the used portion of a stack page by the minimum frame size is even quicker.
You could then provide the image with either an accurate stack depth, or a good approximation thereof. That gives the low space process an easy job to identify a potentially infinitely recursive process.
In general it is good design to put mechanism in the VM and keep policy up in the image.
Lets say, a value 0 - do not kill under any circumstances, and values
0 is candidates for killing.
Then we could assign each process a corresponding policy, and when it goes to look for a candidate for killing, VM picks a process with higher 'memory policy' slot value.
The we could have most important processes have value = 0, and UI and forked processes can have value = 100+. More important than UI processes can be in the middle between 0 and 100. Another rule - automatically increment value of fork(ed) process could be useful too. In this way given slot could reflect fork depth so, one can potentially track process spawn chains.
How do you think, is adding new slot to Process instance worth stability? :)
One more thing, which could be useful is informing image about such unconditional termination event. Either by registering a semaphore which will be signaled upon such event, or adding some kind of #onTerminate message, so dying process can handle such event by itself and perform necessary countermeasures :) Or is there already any means how to inform another process , that some other process is terminated?
From my OS practice: OS treats processes as waitable object, so one
could apply following:
proc run. proc wait. "wait till terminated"
This is easy to do, if process can be associated with semaphore instance. And then , in case it terminates, it signals semaphore.
-- Best regards, Igor Stasenko AKA sig.
On Wed, 02 Jul 2008 16:06:42 +0200, Eliot Miranda wrote:
On Wed, Jul 2, 2008 at 3:32 AM, Igor Stasenko wrote:
...
Few thoughts, about how VM could determine what process(es) is memory hungry to kill them w/o mercy, leaving rest intact and avoid dying:
add 'memory policy' slot to Process instance, which can tell VM, is given process can be killed w/o damaging most critical parts of image.
There is no need for this to be in the VM. Instead the process that runs once the LowSpaceSemaphore is signalled should be of a very high priority (probably above finalization but below timing priority). It then enumerates the runnable processes (processes waiting on semaphores are resumably not the ones chewing up memory). It then suspends any processes on the runnable queues that meet certain criteria.
Something that *could* be in the VM is accounting of how much space a process consumes. Add a slot to each process known to the VM called e.g. slotsAllocated. The VM computes the slots allocated between context switches. This is cheap to compute because it can compute how much space was allocated since the last garbage collection or process switch simply by subtracting the allocation pointer at the end of the previous GC or process switch from the current allocation pointer. The slots allocated since the last process switch is added to the slot in the old process and the slots allocated count zeroed on each context switch. We then have an accurate measure of how much space each process has allocated.
This is a very interesting point, and I would like to see memory consumption per process, that'd be great! But I'm not quite sure I understand how to balance the value of that slot, i.e. how freed space is subtracted. Could you elaborate a bit on this? Thank you!
/Klaus
Hi Klaus,
a request: please don't change the subject to Re: foo... because it breaks threading in gmail (and I guess elsewhere). All other replies have kept the subject unchanged.
On Wed, Jul 2, 2008 at 7:22 AM, Klaus D. Witzel klaus.witzel@cobss.com wrote:
On Wed, 02 Jul 2008 16:06:42 +0200, Eliot Miranda wrote:
On Wed, Jul 2, 2008 at 3:32 AM, Igor Stasenko wrote:
...
Few thoughts, about how VM could determine what process(es) is memory
hungry to kill them w/o mercy, leaving rest intact and avoid dying:
add 'memory policy' slot to Process instance, which can tell VM, is given process can be killed w/o damaging most critical parts of image.
There is no need for this to be in the VM. Instead the process that runs once the LowSpaceSemaphore is signalled should be of a very high priority (probably above finalization but below timing priority). It then enumerates the runnable processes (processes waiting on semaphores are resumably not the ones chewing up memory). It then suspends any processes on the runnable queues that meet certain criteria.
Something that *could* be in the VM is accounting of how much space a process consumes. Add a slot to each process known to the VM called e.g. slotsAllocated. The VM computes the slots allocated between context switches. This is cheap to compute because it can compute how much space was allocated since the last garbage collection or process switch simply by subtracting the allocation pointer at the end of the previous GC or process switch from the current allocation pointer. The slots allocated since the last process switch is added to the slot in the old process and the slots allocated count zeroed on each context switch. We then have an accurate measure of how much space each process has allocated.
This is a very interesting point, and I would like to see memory consumption per process, that'd be great! But I'm not quite sure I understand how to balance the value of that slot, i.e. how freed space is subtracted. Could you elaborate a bit on this? Thank you!
The slot only measures allocations. We never subtract freed space. We don't know which processes the space freed during a garbage collection "belongs to" (were allocated by). But we don't need to know this. The heuristic to determine which processes should be suspended to stop run-away space allocation is that if a process has allocated a lot in recent time then it is likely to continue to do so. A process can't create lots of retained objects without allocating lots of objects. So while this scheme may find false positives, processes which allocate a lot but don't retain the storage, but it will find the processes that allocate a lot and do retain the storage allocated. Since the scheme is intended to stop computation before a fatal evet (running ojut of storage) these false positives are acceptable.
Of course this heuristic is easy to fool if one constructs a pool of processes that conspire to pass the task of allocating space amongst themselves, so that by the time the system gets around to suspending an offending process it has passed the responsibility of behaving badly to another process, but that's a very artificial example.
Hi Eliot,
on Wed, 02 Jul 2008 17:40:05 +0200, you wrote:
Hi Klaus,
a request: please don't change the subject to Re: foo... because it breaks threading in gmail (and I guess elsewhere). All other replies have kept the subject unchanged.
No, I didn't change the subject line. I replied to your
- http://lists.squeakfoundation.org/pipermail/squeak-dev/2008-July/129727.html
and here's what the list server received from me
- http://lists.squeakfoundation.org/pipermail/squeak-dev/2008-July/129731.html
and the gmane NNTP service that I'm using shows the same. Perhaps something happened on your gateway. Anyways.
On Wed, Jul 2, 2008 at 7:22 AM, Klaus D. Witzel wrote:
On Wed, 02 Jul 2008 16:06:42 +0200, Eliot Miranda wrote:
...
Something that *could* be in the VM is accounting of how much space a process consumes. Add a slot to each process known to the VM called e.g. slotsAllocated. The VM computes the slots allocated between context switches. This is cheap to compute because it can compute how much space was allocated since the last garbage collection or process switch simply by subtracting the allocation pointer at the end of the previous GC or process switch from the current allocation pointer. The slots allocated since the last process switch is added to the slot in the old process and the slots allocated count zeroed on each context switch. We then have an accurate measure of how much space each process has allocated.
This is a very interesting point, and I would like to see memory consumption per process, that'd be great! But I'm not quite sure I understand how to balance the value of that slot, i.e. how freed space is subtracted. Could you elaborate a bit on this? Thank you!
The slot only measures allocations. We never subtract freed space. We don't know which processes the space freed during a garbage collection "belongs to" (were allocated by). But we don't need to know this. The heuristic to determine which processes should be suspended to stop run-away space allocation is that if a process has allocated a lot in recent time then it is likely to continue to do so. A process can't create lots of retained objects without allocating lots of objects. So while this scheme may find false positives, processes which allocate a lot but don't retain the storage, but it will find the processes that allocate a lot and do retain the storage allocated. Since the scheme is intended to stop computation before a fatal evet (running ojut of storage) these false positives are acceptable.
Okay, I see what you mean. Thank you.
/Klaus
Of course this heuristic is easy to fool if one constructs a pool of processes that conspire to pass the task of allocating space amongst themselves, so that by the time the system gets around to suspending an offending process it has passed the responsibility of behaving badly to another process, but that's a very artificial example.
On Jul 2, 2008, at 7:06 AM, Eliot Miranda wrote:
Something that *could* be in the VM is accounting of how much space a process consumes. Add a slot to each process known to the VM called e.g. slotsAllocated. The VM computes the slots allocated between context switches. This is cheap to compute because it can compute how much space was allocated since the last garbage collection or process switch simply by subtracting the allocation pointer at the end of the previous GC or process switch from the current allocation pointer. The slots allocated since the last process switch is added to the slot in the old process and the slots allocated count zeroed on each context switch. We then have an accurate measure of how much space each process has allocated.
Ya, there is exactly one place in the VM where the processor switch happens. In the past I tinkered with it to add some slots to Process to track wall clock time by process. At that time was I was thinking gee we could count bytes allocated and other things like socket usage etc, thus providing a bit more process consumption statistical data for Squeak threads ala unix processes.
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
On Wed, Jul 2, 2008 at 11:07 AM, John M McIntosh < johnmci@smalltalkconsulting.com> wrote:
[SNIP]
Ya, there is exactly one place in the VM where the processor switch happens. In the past I tinkered with it to add some slots to Process to track wall clock time by process. At that time was I was thinking gee we could count bytes allocated and other things like socket usage etc, thus providing a bit more process consumption statistical data for Squeak threads ala unix processes.
Please maintain an option that keeps process context switches as fast as possible. I'm interested in Erlang style architectures and how they might be implemented in Smalltalk. Hydra+Cog+Newspeak would make a really nice platform for Erlang-style applications.
-david
2008/7/2 Eliot Miranda eliot.miranda@gmail.com:
On Wed, Jul 2, 2008 at 3:32 AM, Igor Stasenko siguctua@gmail.com wrote:
2008/7/2 John M McIntosh johnmci@smalltalkconsulting.com:
[ big snip.... ]
Few thoughts, about how VM could determine what process(es) is memory hungry to kill them w/o mercy, leaving rest intact and avoid dying:
add 'memory policy' slot to Process instance, which can tell VM, is given process can be killed w/o damaging most critical parts of image.
There is no need for this to be in the VM. Instead the process that runs once the LowSpaceSemaphore is signalled should be of a very high priority (probably above finalization but below timing priority). It then enumerates the runnable processes (processes waiting on semaphores are resumably not the ones chewing up memory). It then suspends any processes on the runnable queues that meet certain criteria.
Well, its more flexible but less stable approach. You need to be sure, that there is someone who still listening for this semaphore, and where is guarantees that process which waits on this semaphore will care to clean anything? Of course, in same way, there is no guarantees that process with memory policy == 0 will not eat up all available space :) So, you are right, we can put decision point into image rather in VM.
Something that *could* be in the VM is accounting of how much space a process consumes. Add a slot to each process known to the VM called e.g. slotsAllocated. The VM computes the slots allocated between context switches. This is cheap to compute because it can compute how much space was allocated since the last garbage collection or process switch simply by subtracting the allocation pointer at the end of the previous GC or process switch from the current allocation pointer. The slots allocated since the last process switch is added to the slot in the old process and the slots allocated count zeroed on each context switch. We then have an accurate measure of how much space each process has allocated.
Wen the low space process runs it simply examines the runnable processes, checking their space allocation. It can either maintain a per-process allocation rate by computing the amount allocated since the last low space signal, or it can zero the per-process allocation count at each low space signal.
This is much more flexible and less arbitrary than having the VM do it.
Similarly a VM that does context-to-stack mapping should be able to cheaply maintain a stack size count per process since it only has to increase or decrease the current process's stack size on each stack page overfow/underflow and context switch. Traversing a stack page's frames to get an accurate count of the number of "contexts" on each page is pretty quick (just a walk of the frame pointer->caller frame pointer chain). Getting an approximation by dividing the used portion of a stack page by the minimum frame size is even quicker.
You could then provide the image with either an accurate stack depth, or a good approximation thereof. That gives the low space process an easy job to identify a potentially infinitely recursive process.
In general it is good design to put mechanism in the VM and keep policy up in the image.
This is what i wanted to avoid: let developer decide what process(es) should stay and what can die instead of stupid machine :)
Killing processes based on measuring how much space it allocates is bad practice, because you really don't have any clues why given process allocated so much space and where its used/involved, and is it safe to kill such process or not. A 'memory policy' i described is much better approach for solving such problems. By finding process with highest possible memory policy slot value and killing it, not the process which consuming memory a lot. So, the developers can control in order of preference, what can be killed in case of problem, and what should stay at any circumstances. Collecting a memory allocation statistics per running process is useful (for measuring hard limits in running system, for example), but not in situation when you reached memory space limits. Because it can't give answer what process should be killed, because process which allocated most of memory is just process which allocated most of memory. This is not means that something wrong with it. Some other process could covertly sit near, allocating few bytes between each GC cycle , step by step increasing allocated memory with useless stuff, and after weeks of stable work you will reach low memory issue. And in such situation, killing process which allocated most in between full GCs will buy you nothing. Thats why i think that counting on memory usage is bad criteria for making decision on what to kill or not.
2008/7/2 Igor Stasenko siguctua@gmail.com:
Oh, and i forgot to add: - signaling a low space semaphore awakes process which kindly asking user to stop play stupid and do something before it will be too late.
A 'memory policy' scheme , which i described before is exactly for the cases, when its too late to let the user decide what to do. Its when you have only two choices left: kill something, or kill VM.
On 2-Jul-08, at 12:12 AM, John M McIntosh wrote:
Ok, well the discussions date from late April 2005. Some are below.
I also found
On 27-Apr-05, at 7:39 PM, Tim Rowledge wrote:
In message a4473bfbe3e462c864d0ef9b9853282e@smalltalkconsulting.com John M McIntosh johnmci@smalltalkconsulting.com wrote:
question without issues. Lately we added some changes by Andreas for correct weak array handling, some changes to how become: works, and my work in VM GC statistical data, so I cann't say which is at fault, if any...
I'm currently hip-deep in similar excrement after merging in the new lowspace- process handling and the gc instrumentation/weak pointer stuff. It _looks_ as if something is getting twisted in the general gc area since the free block size ends up being set to 4. As in '4' not 4k or 4mb, just 4. That quite unsurprisingly upsets the sufficientSpaceToAllocate: code and we get an Earth- shattering kaboom. Once my head has stopped spinning I'll try the lowspace changes without the gc changes.
All to try to make it possible to debug some stupid problem in TK4 tree walking....
Which I think refers pretty directly to the 'secret theft' issue. Ah at last - I think this is probably the email where I tried to explain the problem-
On 30-Apr-05, at 7:00 PM, Tim Rowledge wrote:
After having problem trying to debug some TK4 code that blew up with lowspace problems but never let me catch and debug, I spent some time adding the lowspace-process stuff we recently discussed. I had to make a few alterations to match it up with the latest 64bit clean code but no problems with that part.
After building a VM I started testing with some of the methods in SystemDictionary 'memory space' - in particular #useUpMemory. It is perhaps fortunate that I did since the other #useUp* methods pretty much work once the lowspace process is caught by the vm and passed up to the image. After a _lot_ of head scratching by John & I we found that with a gazillion tiny objects (Links are the smallest possible objects that can exist on their own, plain Objects would have to be contained in a collection and so would cost the same 3 words per object) cause a catastrophic GC explosion. What happens is that memory fills up until we get to signal lowspace and then we are in danger. Depending upon the exact size of object memory in use the 200kb used as the lowSpaceThreshold can be gobbled up in one swallow by the initializeMemoryFirstFree: method making sure there is a byte per object that survived the markPhase. In using useUpMemory we can get to having 4 bytes of free space when the next allocate is attempted.... Ka-Boom. Expanding the lowSpaceThreshold (along with the VM changes to report the process and avoid the accidental problem of interrupting eventTickler) to a couple of mb makes it ok on my machine and the threshold can be a lot lower with the other tests that create bigger (hence fewer) objects (hence smaller fwdTable needs). In the worst case, we could have a very large OM filled with very small objects all surviving markPhase; in such a case we would need an additional 1/12 of OM available for the fwdTable. So for a 30Mb objectmemory we ought to set the lowSPaceThreshold to 30/13 => 2.31Mb + actual space needed to run the notifier/debugger etc for reasonable safety. Or hide the 2.31 Mb away so the image never even knows it is there. If you are using virtual memory and a limit of 512Mb then you should perhaps secrete 40Mb some where safe.
This assumes that we really need to have one byte per object of course. The original rationale was to keep the number of compact loops down to eight (see Dan's comment in initializeMemoryFirstFree:) for Alan's large demo image. The nicest solution would be to come up with a way to do our GC & compacting without needing any extra space. Commence headscratching now... John suggested making sure the fwd gets less than the byte-per-object if things are tight, and accpting the extra compaction loops.
Good news- with the vm change and 2Mb lowSpaceThreshold I can probably go back and find my TK4 problem(s).
Bad news- consider Tweak. With lots of processes whizzing away, merely stopping the one that did the allocation and triggered the lowspace is not going to be much good. Stopping everything except the utterly essential stuff to debug the lowspace will be needed. Probably.
It's really depressing reading mail from back then and even earlier. There are long discussions from 2002 about changing the image format for version 4, clean compiled methods, closures........
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- LE ROI EST MORT. JIVE LE ROI - The King is dead. No kidding.
+1 to moving named primitive into image
+1 to make use of primitive returning value (this would affect the code generator, but hell, its way better to have a notion why it failed than simply guessing in the dark room, like any currently primitive failure code does)
+1 to eliminating pushRemappableOop:/popRemappableOop . (A question however, how you would handle heap overflows, when evil primitive generates massive object or lots of tiny objects?)
(a little OT .. )
Another question, is how would you sync there changes with Hydra (if you having plans of course)? I have many changes in CodeGenerator and some in VMMaker , to simplify my day's job :)
In fact, it would be good to make a clean refactoring of it: - make inst vars of Interpreter/ObjectMemory be the interpreter instance state - make class vars be the part of VM global state - make pool vars be the constants - same could be applied to methods (instance/class side)
Currently things are little messy, since i made incremental changes to existing model.
Another thing, which doesn't makes me sleep well, is that by using thread specific storage, it would be possible to avoid passing interpreter instance to each function/primitive. Too bad, GCC (on windows) support a __thread__ specifier only in latest release , which is 'technology preview' how they calling it :) If we could use this, it would be possible to make Hydra nearly as fast as current Squeak VM. And there is no point to argue do we have right to use thread specific storage or not: consider that Hydra can run on platforms which can support threads. And since we already using threads, why we can't use thread-specific storage.
... (stopping OT)
2008/7/1 Eliot Miranda eliot.miranda@gmail.com:
On Tue, Jul 1, 2008 at 1:34 PM, tim Rowledge tim@rowledge.org wrote:
On 1-Jul-08, at 1:20 PM, Eliot Miranda wrote:
One doesn't have to *use* the FFI. If the FFI isn't exposed via a primitive then no FFI. One can still have named primitives supported by the image and not by the VM and not use the FFI. To call a named primitive in a primitive plugin the following sequence occurs:
the method containing a named primitive spec is activated and the primitive call fails because its function pointer is null. the failure code extracts the plugin name and invokes a primitive to load the plugin library the failure code extracts the primitive name and uses the lookup primitive to find the function in the loaded plugin library the failure code uses a primitive to slam the function pointer into the method the failure code uses the executeMethodWithArgs primitive to retry the bound named primitive method
So the FFI is an optional extra. One needs four primitives, load library, lookup name in library, insert primitive function pointer. and executemethodWithArgs (thanks Tim!). Slamming the function into the method could also be done using, say, objectAt:.
So one can still have a nice small safe VM and have no direct support for named primitives in the VM.
Leaving aside my like of all-named-prims, I *like* this enhancement to the support for named prims.
It would slightly complicate the post-prim-call code in each method because you would need to handle the failed-to-find-prim case as well as all the prim-failed cases. It would be helped by an old idea that I'm pretty sure eliot has plans for anyway (as indeed I have written about a few times) to add primitive error return values. For those that haven't heard of them, this is just a way of having the first temp in a context involved in a prim call be a slot for a return value from the prim if any error occurs. This means you can actually know wtf went wrong instead of guessing - "ooh, was the first arg a SmallInteger? Um, if not, was it a Float that might round nicley? Err, was the second arg a ByteArray with the first byte = 255?" etc. Instead you get "ah, errorValue was 1, so we have to explode the reactor core in order to stop the Dreen from eating the children". Much nicer.
I did primitive error codes at Cadence and they'll very probably b making it into Cog real soon now. They're simpler than VisualWorks', being only symbols. So extracting more information, such as a related error code requires a subsequent call. But I think the work I'm doing right now on eliminating pushRemappableOop:/popRemappableOop will enable me to have a structured object with a name and parameters, which is more generally useful.
A nice thing is that the code is forwards and backewards compatible. One can use the VM to run older images. One can run images that contain the primitive error code on older VMs, where one simply gets a nil error code on primitive failure.
On Tue, Jul 1, 2008 at 4:12 PM, Igor Stasenko siguctua@gmail.com wrote:
+1 to moving named primitive into image
+1 to make use of primitive returning value (this would affect the code generator, but hell, its way better to have a notion why it failed than simply guessing in the dark room, like any currently primitive failure code does)
+1 to eliminating pushRemappableOop:/popRemappableOop . (A question however, how you would handle heap overflows, when evil primitive generates massive object or lots of tiny objects?)
It is much the same as it is now. A large allocation attempt (e.g. in primitiveNewWithArg) either succeed, possibly causing a garbage collection very soon after, or fails. The garbage collector is allowed to run in primitiveNewWithArg et al. But allocations at any other time cannot cause the garbage collector to run and are known to fit within the extra memory reserved for the interpreter. These allocations always succeed but may exceed some threshold which will cause a garbage collection very soon thereafter.
Memory will look something like this: -> start of heap ... lots of objects ... -> young start ... not so many objects ... -> free start/eden ... small amount of free space (a few hundred k) ... -> scavenge threshold ... free memory (megabytes)... -> VM reserve start .... very few k (e.g. 8k) ... -> fwd block reserve -> end
The allocation pointer is free start/eden. A new object is allocated here and advances free start/eden. We will dispense with freeBlock and its header for speed.
primitiveNewWithArg (et al) can allocate up to VM reserve start. An attempt beyond that can trigger a garbage collection but the retry cannot exceed VM reserve start.
Any other allocation can exceed VM reserve start but not fwd block reserve. Exceeding fwd block reserve is an internal fatal error that should not happen if VM reserve is large enough.
Any allocation that pushes free start/eden beyond scavenge threshold sets a flag that will cause the VM to perform an incrementalGC on the next send or backward branch.
The net result is that pointers only move during primitives that do substantial allocations (primitiveNewWithArg, anything that allocates lots of objects) and only these primitives need to use pushRemappableOop:/popRemappableOop.
The details can vary but the essentials are to arrange that for allocations in the course of execution (e.g. creating a block,creating the message argument of a doesNotUnderstand:, creating a primitive failure value, flushing a stack page to heap contexts, etc, etc) the garage collector will not run and so the VM does not need to manage pointers in local variables. It can assume these values will not change and remain valid until the next send or backward branch.
(a little OT .. )
Another question, is how would you sync there changes with Hydra (if you having plans of course)?
This *must* be done. I expect we'll spend quite a lot of time communicating to make this happen. I am about to extend the translator for the stack interpreter. But basically its my job to fit Cog into Hydra.
My own OT:
BTW, did you see my comment about renaming Hydra? I think "String" is a much better name. Strings are composed of threads. Smalltalk comes from Alan Kay's distaste for the kinds of names people were using for programming languages in the 60's & 70's, names like Zeus and Thor. Hence Smalltalk. Hydra is another mythic name. Cog is a small element in a larger whole (and a cool advert by Honda). Anyway, think it over...
I have many changes in CodeGenerator and some in VMMaker , to simplify
my day's job :)
Could you email me your current VMMaker package, or a link to it? And could you summarise the changes you've made and why? I'll do the same.
In fact, it would be good to make a clean refactoring of it:
- make inst vars of Interpreter/ObjectMemory be the interpreter instance
state
- make class vars be the part of VM global state
- make pool vars be the constants
- same could be applied to methods (instance/class side)
Currently things are little messy, since i made incremental changes to existing model.
Another thing, which doesn't makes me sleep well, is that by using thread specific storage, it would be possible to avoid passing interpreter instance to each function/primitive. Too bad, GCC (on windows) support a __thread__ specifier only in latest release , which is 'technology preview' how they calling it :) If we could use this, it would be possible to make Hydra nearly as fast as current Squeak VM. And there is no point to argue do we have right to use thread specific storage or not: consider that Hydra can run on platforms which can support threads. And since we already using threads, why we can't use thread-specific storage.
You need is an abstraction layer, e.g. implemented with macros that insulates the Slang code form the platform thread system details. You then implement the abstraction layer as thinly as possible. I've done this with the threaded API extension of the VW FFI. Its not hard. You may be able to find GPL versions out there.
best Eliot
... (stopping OT)
2008/7/1 Eliot Miranda eliot.miranda@gmail.com:
On Tue, Jul 1, 2008 at 1:34 PM, tim Rowledge tim@rowledge.org wrote:
On 1-Jul-08, at 1:20 PM, Eliot Miranda wrote:
One doesn't have to *use* the FFI. If the FFI isn't exposed via a primitive then no FFI. One can still have named primitives supported
by the
image and not by the VM and not use the FFI. To call a named primitive
in a
primitive plugin the following sequence occurs:
the method containing a named primitive spec is activated and the primitive call fails because its function pointer is null. the failure code extracts the plugin name and invokes a primitive to
load
the plugin library the failure code extracts the primitive name and uses the lookup primitive to find the function in the loaded plugin library the failure code uses a primitive to slam the function pointer into the method the failure code uses the executeMethodWithArgs primitive to retry the bound named primitive method
So the FFI is an optional extra. One needs four primitives, load library, lookup name in library, insert primitive function pointer. and executemethodWithArgs (thanks Tim!). Slamming the function into the
method
could also be done using, say, objectAt:.
So one can still have a nice small safe VM and have no direct support
for
named primitives in the VM.
Leaving aside my like of all-named-prims, I *like* this enhancement to
the
support for named prims.
It would slightly complicate the post-prim-call code in each method because you would need to handle the failed-to-find-prim case as well as
all
the prim-failed cases. It would be helped by an old idea that I'm pretty sure eliot has plans for anyway (as indeed I have written about a few
times)
to add primitive error return values. For those that haven't heard of
them,
this is just a way of having the first temp in a context involved in a
prim
call be a slot for a return value from the prim if any error occurs.
This
means you can actually know wtf went wrong instead of guessing - "ooh,
was
the first arg a SmallInteger? Um, if not, was it a Float that might
round
nicley? Err, was the second arg a ByteArray with the first byte = 255?"
etc.
Instead you get "ah, errorValue was 1, so we have to explode the reactor core in order to stop the Dreen from eating the children". Much nicer.
I did primitive error codes at Cadence and they'll very probably b making
it
into Cog real soon now. They're simpler than VisualWorks', being only symbols. So extracting more information, such as a related error code requires a subsequent call. But I think the work I'm doing right now on eliminating pushRemappableOop:/popRemappableOop will enable me to have a structured object with a name and parameters, which is more generally useful.
A nice thing is that the code is forwards and backewards compatible. One can use the VM to run older images. One can run images that contain the primitive error code on older VMs, where one simply gets a nil error code
on
primitive failure.
-- Best regards, Igor Stasenko AKA sig.
2008/7/2 Eliot Miranda eliot.miranda@gmail.com:
On Tue, Jul 1, 2008 at 4:12 PM, Igor Stasenko siguctua@gmail.com wrote:
+1 to moving named primitive into image
+1 to make use of primitive returning value (this would affect the code generator, but hell, its way better to have a notion why it failed than simply guessing in the dark room, like any currently primitive failure code does)
+1 to eliminating pushRemappableOop:/popRemappableOop . (A question however, how you would handle heap overflows, when evil primitive generates massive object or lots of tiny objects?)
It is much the same as it is now. A large allocation attempt (e.g. in primitiveNewWithArg) either succeed, possibly causing a garbage collection very soon after, or fails. The garbage collector is allowed to run in primitiveNewWithArg et al. But allocations at any other time cannot cause the garbage collector to run and are known to fit within the extra memory reserved for the interpreter. These allocations always succeed but may exceed some threshold which will cause a garbage collection very soon thereafter.
Memory will look something like this: -> start of heap ... lots of objects ... -> young start ... not so many objects ... -> free start/eden ... small amount of free space (a few hundred k) ... -> scavenge threshold ... free memory (megabytes)... -> VM reserve start .... very few k (e.g. 8k) ... -> fwd block reserve -> end
The allocation pointer is free start/eden. A new object is allocated here and advances free start/eden. We will dispense with freeBlock and its header for speed.
primitiveNewWithArg (et al) can allocate up to VM reserve start. An attempt beyond that can trigger a garbage collection but the retry cannot exceed VM reserve start.
Any other allocation can exceed VM reserve start but not fwd block reserve. Exceeding fwd block reserve is an internal fatal error that should not happen if VM reserve is large enough.
Any allocation that pushes free start/eden beyond scavenge threshold sets a flag that will cause the VM to perform an incrementalGC on the next send or backward branch.
The net result is that pointers only move during primitives that do substantial allocations (primitiveNewWithArg, anything that allocates lots of objects) and only these primitives need to use pushRemappableOop:/popRemappableOop.
The details can vary but the essentials are to arrange that for allocations in the course of execution (e.g. creating a block,creating the message argument of a doesNotUnderstand:, creating a primitive failure value, flushing a stack page to heap contexts, etc, etc) the garage collector will not run and so the VM does not need to manage pointers in local variables. It can assume these values will not change and remain valid until the next send or backward branch.
Agreed, most of primitives can't pass over 8k hard limit, except those which excessively deal with i/o (like taking a screenshot). Good, i found that both socket and file prims using an array argument passed as buffer, where plugin would store received data. I know at least single place where it can create a ByteArray of arbitrary size (but this is introduced in Hydra, and we will get to it ;) ).
In any case, removing pushRemappableOop:/popRemappableOop pattern would require to overview all places where its used, so at least for core plugins, which live with regular VM this could be done in single swift cut'n'throw away manner.
(a little OT .. )
Another question, is how would you sync there changes with Hydra (if you having plans of course)?
This *must* be done. I expect we'll spend quite a lot of time communicating to make this happen. I am about to extend the translator for the stack interpreter. But basically its my job to fit Cog into Hydra.
My own OT:
BTW, did you see my comment about renaming Hydra? I think "String" is a much better name. Strings are composed of threads. Smalltalk comes from Alan Kay's distaste for the kinds of names people were using for programming languages in the 60's & 70's, names like Zeus and Thor. Hence Smalltalk. Hydra is another mythic name. Cog is a small element in a larger whole (and a cool advert by Honda). Anyway, think it over...
Naming things is not my best talent. :) A name "Hydra" is not really my idea (originally it came from Andreas colleague afaik). I can say that i like it: Hydra has many heads. But if you find it too pretentious, hmm. A "String" is directly opposite to it.. Its too plain boring & gray :) There can be many criteria how to pick a name: ones care that name should sell itself, others want to put meaning in it (to conform idea). I personally prefer names which simply sound good. :)
I have many changes in CodeGenerator and some in VMMaker , to simplify my day's job :)
Could you email me your current VMMaker package, or a link to it? And could you summarise the changes you've made and why? I'll do the same.
A package available at http://jabberwocky.croquetproject.org:8889/HydraVM. Just made snapshot.
Yes, i would write a summary, meanwhile you can read a summary about Hydra here: http://squeakvm.org/~sig/hydravm/devnotes.html
In fact, it would be good to make a clean refactoring of it:
- make inst vars of Interpreter/ObjectMemory be the interpreter instance
state
- make class vars be the part of VM global state
- make pool vars be the constants
- same could be applied to methods (instance/class side)
Currently things are little messy, since i made incremental changes to existing model.
Another thing, which doesn't makes me sleep well, is that by using thread specific storage, it would be possible to avoid passing interpreter instance to each function/primitive. Too bad, GCC (on windows) support a __thread__ specifier only in latest release , which is 'technology preview' how they calling it :) If we could use this, it would be possible to make Hydra nearly as fast as current Squeak VM. And there is no point to argue do we have right to use thread specific storage or not: consider that Hydra can run on platforms which can support threads. And since we already using threads, why we can't use thread-specific storage.
You need is an abstraction layer, e.g. implemented with macros that insulates the Slang code form the platform thread system details. You then implement the abstraction layer as thinly as possible. I've done this with the threaded API extension of the VW FFI. Its not hard. You may be able to find GPL versions out there.
best Eliot
... (stopping OT)
Igor Stasenko wrote:
2008/7/2 Eliot Miranda eliot.miranda@gmail.com:
BTW, did you see my comment about renaming Hydra? I think "String" is a much better name. Strings are composed of threads. Smalltalk comes from Alan Kay's distaste for the kinds of names people were using for programming languages in the 60's & 70's, names like Zeus and Thor. Hence Smalltalk. Hydra is another mythic name. Cog is a small element in a larger whole (and a cool advert by Honda). Anyway, think it over...
Naming things is not my best talent. :) A name "Hydra" is not really my idea (originally it came from Andreas colleague afaik). I can say that i like it: Hydra has many heads. But if you find it too pretentious, hmm. A "String" is directly opposite to it.. Its too plain boring & gray :)
My alternative to Hydra would be "Couscous" [1] - lots of little things and you can have some decent Smalltalk with it ;-)
[1] http://en.wikipedia.org/wiki/Couscous
Cheers, - Andreas
-----Original Message----- From: squeak-dev-bounces@lists.squeakfoundation.org [mailto:squeak-dev- bounces@lists.squeakfoundation.org] On Behalf Of Igor Stasenko Sent: Tuesday, July 01, 2008 9:26 PM To: The general-purpose Squeak developers list Subject: Re: [squeak-dev] The Primitive: I am not a number- I am a named prim!- SqueakPeople article
2008/7/2 Eliot Miranda eliot.miranda@gmail.com:
My own OT:
BTW, did you see my comment about renaming Hydra? I think "String" is a much better name. Strings are composed of threads. Smalltalk comes from Alan Kay's distaste for the kinds of names people were using for programming languages in the 60's & 70's, names like Zeus and Thor. Hence Smalltalk. Hydra is another mythic name. Cog is a small element in a larger whole (and a cool advert by Honda). Anyway, think it over...
Naming things is not my best talent. :) A name "Hydra" is not really my idea (originally it came from Andreas colleague afaik). I can say that i like it: Hydra has many heads. But if you find it too pretentious, hmm. A "String" is directly opposite to it.. Its too plain boring & gray :)
Well, how about "Tapestry". It is made of threads and is not boring.
There can be many criteria how to pick a name: ones care that name should sell itself, others want to put meaning in it (to conform idea). I personally prefer names which simply sound good. :)
Terry
=========================================================== Terry Raymond Crafted Smalltalk 80 Lazywood Ln. Tiverton, RI 02878 (401) 624-4517 traymond@craftedsmalltalk.com http://www.craftedsmalltalk.com ===========================================================
Quoting Terry Raymond squeak-craft@cox.net:
Well, how about "Tapestry". It is made of threads and is not boring.
Wow, that hits the spot. (And we already have warp, but not weft :-) ) A tapestry is a colourful weaving together of treads to make a whole and often to tell a story, epic or otherwise. It has purpose, into the future, and is often a product of many people. Plus, different parts of it are made by different teams.
That suggests: Quilt
These are made up of separate pieces of cloth (packages, classes?), sometimes many copies of identical pieces, sometimes not, assembled into a larger whole. The pieces are sewn together, but often there is an addition layer (thread) of sewing that winds its way through all of the pieces. Each quilt has an overall design.
Further, each piece of cloth may be a separate design, can be new or old, or can be a part of a previous creation. Each piece of cloth is made up of threads woven together.
So, let's make our bed, and lie in it ;-)
David
I think "String" is a much better name. Strings are composed of threads.
But the word "string" is so overloaded, especially in computer science. It seemed to me that Igor was looking for a name that would evoke "multi-headedness", and "Hydra" certainly does that. I don't think it falls into the same category as names like "Zeus" or "Thor", which I assume were symptomatic of power fantasies with no further meaning. It's the lack of meaning that I see as the problem, I think the mythic quality is otherwise irrelevant.
While were at it, though...
Smalltalk comes from Alan Kay's distaste for the kinds of names people were using for programming languages in the 60's & 70's, names like Zeus and Thor. Hence Smalltalk.
I think "Smalltalk" was a terrible choice. Pretty much everyone to whom I've mentioned it thinks it's too long, and it immediately makes them think of annoying obligatory chit-chat, something they hate (as either speaker or listener).
And how unfortunately ironic would it be to act with such deference to an authority figure expressing his distaste for authority figures? :) I don't think we should treat Alan Kay like a god, either...
Oh, and Cog is a great name. :)
-C
-- Craig Latta improvisational musical informaticist www.netjam.org Smalltalkers do: [:it | All with: Class, (And love: it)]
2008/7/2 Craig Latta craig@netjam.org:
I think "String" is a much better name. Strings are composed of threads.
But the word "string" is so overloaded, especially in computer science.It seemed to me that Igor was looking for a name that would evoke "multi-headedness", and "Hydra" certainly does that. I don't think it falls into the same category as names like "Zeus" or "Thor", which I assume were symptomatic of power fantasies with no further meaning. It's the lack of meaning that I see as the problem, I think the mythic quality is otherwise irrelevant.
Yeah, Hydra could (small)talk a lot faster, because it having many heads.
- there is no Spoon nor a fork, simply because Hydra swallowed them both :)
While were at it, though...Smalltalk comes from Alan Kay's distaste for the kinds of names people were using for programming languages in the 60's & 70's, names like Zeus and Thor. Hence Smalltalk.
I think "Smalltalk" was a terrible choice. Pretty much everyone to whomI've mentioned it thinks it's too long, and it immediately makes them think of annoying obligatory chit-chat, something they hate (as either speaker or listener).
I don't share your point here. A name is quite relevant to computing: if we going to talk with computers then to understand each other we need something in common - a language. Small language, which we can understand both and talk on it :)
And that's the most why i like smalltalk - it is small. It is inherently easy to learn and talk on it. Maybe there other language(s) which would make programmer's life as easy as smalltalk does. But i didn't met them yet in my life.
And how unfortunately ironic would it be to act with such deference toan authority figure expressing his distaste for authority figures? :) I don't think we should treat Alan Kay like a god, either...
Oh, and Cog is a great name. :)-C
-- Craig Latta improvisational musical informaticist www.netjam.org Smalltalkers do: [:it | All with: Class, (And love: it)]
I don't share your point here. A name is quite relevant to computing: if we going to talk with computers then to understand each other we need something in common - a language. Small language, which we can understand both and talk on it :)
And that's the most why i like smalltalk - it is small. It is inherently easy to learn and talk on it. Maybe there other language(s) which would make programmer's life as easy as smalltalk does. But i didn't met them yet in my life.
That's all true, but it also indicates to me that you're not familiar with the original meaning of the word "smalltalk" in English. :) It has strongly negative baggage.
-C
-- Craig Latta improvisational musical informaticist www.netjam.org Smalltalkers do: [:it | All with: Class, (And love: it)]
On Thu, Jul 3, 2008 at 12:50 PM, Craig Latta craig@netjam.org wrote:
I don't share your point here. A name is quite relevant to computing: if we going to talk with computers then to understand each other we need something in common - a language. Small language, which we can understand both and talk on it :)
And that's the most why i like smalltalk - it is small. It is inherently easy to learn and talk on it. Maybe there other language(s) which would make programmer's life as easy as smalltalk does. But i didn't met them yet in my life.
That's all true, but it also indicates to me that you're not familiarwith the original meaning of the word "smalltalk" in English. :) It has strongly negative baggage.
It does? First time I heard; I always thought it was an important social skill: http://howto.lifehack.org/wiki/Small_Talk
Gulik.
Am 03.07.2008 um 03:14 schrieb Michael van der Gulik:
On Thu, Jul 3, 2008 at 12:50 PM, Craig Latta craig@netjam.org wrote:
I don't share your point here. A name is quite relevant to
computing:
if we going to talk with computers then to understand each other we need something in common - a language. Small language, which we can understand both and talk on it :)
And that's the most why i like smalltalk - it is small. It is inherently easy to learn and talk on it. Maybe there other
language(s)
which would make programmer's life as easy as smalltalk does. But i didn't met them yet in my life.
That's all true, but it also indicates to me that you're notfamiliar with the original meaning of the word "smalltalk" in English. :) It has strongly negative baggage.
It does? First time I heard; I always thought it was an important social skill: http://howto.lifehack.org/wiki/Small_Talk
Hehe - but it's the very definition of geeks/nerds to lack "social skills", they have a hard time doing small talk, they feel chatting about irrelevant things is a waste of time. Hence the word has a negative taste in these circles. And the percentage of them among computer professional is higher than average.
Even amongst Squeakers the percentage of people lovingly decorating their images, cuddling it with updates and patches over years is dwindling. The "real programmers" who just load packages to get work done in "throw-away" images are in the majority now.
- Bert -
On Jul 3, 2008, at 3:39 AM, Bert Freudenberg wrote:
Am 03.07.2008 um 03:14 schrieb Michael van der Gulik:
On Thu, Jul 3, 2008 at 12:50 PM, Craig Latta craig@netjam.org wrote:
I don't share your point here. A name is quite relevant to
computing:
if we going to talk with computers then to understand each other we need something in common - a language. Small language, which we can understand both and talk on it :)
And that's the most why i like smalltalk - it is small. It is inherently easy to learn and talk on it. Maybe there other
language(s)
which would make programmer's life as easy as smalltalk does. But i didn't met them yet in my life.
That's all true, but it also indicates to me that you're not familiar with the original meaning of the word "smalltalk" in English. :) It has strongly negative baggage.
It does? First time I heard; I always thought it was an important social skill: http://howto.lifehack.org/wiki/Small_Talk
Hehe - but it's the very definition of geeks/nerds to lack "social skills", they have a hard time doing small talk, they feel chatting about irrelevant things is a waste of time. Hence the word has a negative taste in these circles. And the percentage of them among computer professional is higher than average.
Even amongst Squeakers the percentage of people lovingly decorating their images, cuddling it with updates and patches over years is dwindling. The "real programmers" who just load packages to get work done in "throw-away" images are in the majority now.
That's an interesting observation; I hadn't thought of it. I've definitely moved into the "real programmers" camp; I used to have an image that was basically a multimedia concept-map (http://en.wikipedia.org/wiki/Concept_map ). Today I have a very slightly customized image that I clone throwaway copies from. A big part of the problem was my inability to migrate my content into a newer Squeak version.
(as an aside, I still have hopes that Sophie could step into that role for me, but it currently feels too heavyweight... it's more suited for "real multimedia authoring" than for supporting cognitive processes in real-time)
I wonder if there is any causal link between the trend toward throwaway images, and the larger societal trend toward more disposable objects in general (thinking of the free inkjet printers used as a loss-leader for expensive ink sales... ugh!)
Cheers, Josh
- Bert -
Joshua Gargus a écrit :
That's an interesting observation; I hadn't thought of it. I've definitely moved into the "real programmers" camp; I used to have an image that was basically a multimedia concept-map (http://en.wikipedia.org/wiki/Concept_map). Today I have a very slightly customized image that I clone throwaway copies from. A big part of the problem was my inability to migrate my content into a newer Squeak version.
(as an aside, I still have hopes that Sophie could step into that role for me, but it currently feels too heavyweight... it's more suited for "real multimedia authoring" than for supporting cognitive processes in real-time)
I wonder if there is any causal link between the trend toward throwaway images, and the larger societal trend toward more disposable objects in general (thinking of the free inkjet printers used as a loss-leader for expensive ink sales... ugh!)
Cheers, Josh
Ah yes, the cloning model is very limited. Who diddn't ever dream of a sexual reproduction of images. Wouldn't the children of Croquet and Sophie be viable?
Nicolas
Hehe - but it's the very definition of geeks/nerds to lack "social skills", they have a hard time doing small talk, they feel chatting about irrelevant things is a waste of time. Hence the word has a negative taste in these circles. And the percentage of them among computer professional is higher than average.
Heh, that may also be true, but I wasn't talking about those people. :) And I'm not saying that striking up a conversation with people about superficial things isn't an important skill, I'm saying that most people, geeks or otherwise, find it annoying and would prefer to engage in meaningful discussion. Surely the geeks here can find plenty of URLs to support this. :)
But apparently we disagree, probably best to leave it there. :) I've spoken my peace: "Smalltalk" is an awful name.
-C
-- Craig Latta improvisational musical informaticist www.netjam.org Smalltalkers do: [:it | All with: Class, (And love: it)]
Am 03.07.2008 um 20:32 schrieb Craig Latta:
Hehe - but it's the very definition of geeks/nerds to lack "social skills", they have a hard time doing small talk, they feel chatting about irrelevant things is a waste of time. Hence the word has a negative taste in these circles. And the percentage of them among computer professional is higher than average.
Heh, that may also be true, but I wasn't talking about thosepeople. :) And I'm not saying that striking up a conversation with people about superficial things isn't an important skill, I'm saying that most people, geeks or otherwise, find it annoying and would prefer to engage in meaningful discussion. Surely the geeks here can find plenty of URLs to support this. :)
But apparently we disagree, probably best to leave it there. :)I've spoken my peace: "Smalltalk" is an awful name.
How does "Squeak" stack up then?
*ducks and runs*
- Bert -
On 3-Jul-08, at 11:46 AM, Bert Freudenberg wrote:
How does "Squeak" stack up then?
*ducks and runs*
That would be 'Qwak' :-)
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Quality assurance: A way to ensure you never deliver shoddy goods accidentally.
On Thu, Jul 3, 2008 at 2:54 PM, tim Rowledge tim@rowledge.org wrote:
On 3-Jul-08, at 11:46 AM, Bert Freudenberg wrote:
How does "Squeak" stack up then?
*ducks and runs*
That would be 'Qwak' :-)
Tim - u really, really missed your calling - quit your day job and go do standup NOW!
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Quality assurance: A way to ensure you never deliver shoddy goods accidentally.
2008/7/3 Craig Latta craig@netjam.org:
I don't share your point here. A name is quite relevant to computing: if we going to talk with computers then to understand each other we need something in common - a language. Small language, which we can understand both and talk on it :)
And that's the most why i like smalltalk - it is small. It is inherently easy to learn and talk on it. Maybe there other language(s) which would make programmer's life as easy as smalltalk does. But i didn't met them yet in my life.
That's all true, but it also indicates to me that you're not familiarwith the original meaning of the word "smalltalk" in English. :) It has strongly negative baggage.
Heh, yes, i'm not native english speaker. :)
...
which would make programmer's life as easy as smalltalk does. But i didn't met them yet in my life.
That's all true, but it also indicates to me that you're notfamiliar with the original meaning of the word "smalltalk" in English. :) It has strongly negative baggage.
-C
-- Craig Latta
For a name of any valuable public thing (as languages and technologies) same care as making a trademark or brand should be taken. Not less.
Once that is understood and accepted there are some known guidelines to create a name. People don't think backwards by default, so unless you are "selling" negative experiences one should NEVER use negative evocation.
By the way evocation is not the only technique for names. Here you have a nice reading from (another) Igor that should expand your mindset on this regard: http://www.igorinternational.com/process/naming-guide-product-company-names....
Have a smart synaptic activity and found something positive and brilliant. Don’t accommodate yourself on less than that,
all the best,
Sebastian
On Wed, Jul 2, 2008 at 4:01 AM, Craig Latta craig@netjam.org wrote:
It seemed to me that Igor was looking for a name that would evoke "multi-headedness", and "Hydra" certainly does that. I don't think it falls into the same category as names like "Zeus" or "Thor", which I assume were symptomatic of power fantasies with no further meaning. It's the lack of meaning that I see as the problem, I think the mythic quality is otherwise irrelevant.
While were at it, though...
http://en.wikipedia.org/wiki/Briareos_Hecatonchires http://en.wikipedia.org/wiki/Hecatonchires
but it's long :-)
Smalltalk comes from Alan Kay's distaste for the kinds of names people were using for programming languages in the 60's & 70's, names like Zeus and Thor. Hence Smalltalk.
I think "Smalltalk" was a terrible choice. Pretty much everyone to whomI've mentioned it thinks it's too long, and it immediately makes them think of annoying obligatory chit-chat, something they hate (as either speaker or listener).
And how unfortunately ironic would it be to act with such deference toan authority figure expressing his distaste for authority figures? :) I don't think we should treat Alan Kay like a god, either...
Oh, and Cog is a great name. :)-C
-- Craig Latta improvisational musical informaticist www.netjam.org Smalltalkers do: [:it | All with: Class, (And love: it)]
On Jul 1, 2008, at 5:06 PM, Eliot Miranda wrote:
Any other allocation can exceed VM reserve start but not fwd block reserve. Exceeding fwd block reserve is an internal fatal error that should not happen if VM reserve is large enough.
Any allocation that pushes free start/eden beyond scavenge threshold sets a flag that will cause the VM to perform an incrementalGC on the next send or backward branch.
The net result is that pointers only move during primitives that do substantial allocations (primitiveNewWithArg, anything that allocates lots of objects) and only these primitives need to use pushRemappableOop:/popRemappableOop.
The details can vary but the essentials are to arrange that for allocations in the course of execution (e.g. creating a block,creating the message argument of a doesNotUnderstand:, creating a primitive failure value, flushing a stack page to heap contexts, etc, etc) the garage collector will not run and so the VM does not need to manage pointers in local variables. It can assume these values will not change and remain valid until the next send or backward branch.
Back when Tim and I looked at allocation failures a few years back I had built a VM where it would slide the end of young space into the forwarding block reserve area. That worked quite nicely and you could chew memory right below the number of bytes for needed for the next MethodContext at which point the VM would crash. However that never went into VMMaker. The real issue is deciding how to stop the process that is fatally consuming memory. But that's a different issue.
As for the set a flag for an IGC, also see forceTenureFlag via primitiveForceTenure which some memory policies use to solve the problem with excessive slot scanning via the root table.
-- = = = ======================================================================== John M. McIntosh johnmci@smalltalkconsulting.com Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com = = = ========================================================================
On 1-Jul-08, at 12:25 PM, Eliot Miranda wrote:
[snip]
Gettng rid of both obsolete tables is a really good thing (and I've already done it).
Good
Getting rid of indexed primitives for the core does not save anything. There still has to be a table somewhere that maps the primitive name to the primitive function. All you do is replace a table mapping indices to functions with one mapping strings to functions. You can't reply on the OS to do the lookup because you won't be able to strip VM executables if you do and that really costs space.
Sure; there's always got to be some way of looking up things. It will always take some space and time. But surely trading space/time for more flexible and smarter is what Smalltalk is all about?
But more importantly I think its pointless. The bulk of primitives in the system are named, and they're in plugins and that all works fine, even though lookup is a little involved and slow. But I see nothing wrong with the core language primitives being indexed. They are pretty well fixed with unchanged semantics and unchanged references in the system (e.g. SmallInteger>>#+ is where you find <primitive: 1> and has always been where you find it). The table is quick to index and simple.
The downside in my opinion is that you have an early bound lookup that also requires a different mechanism to support it. Named prims allow for a later binding, replacement dynamically, and removes one of the lookup mechanisms. As an example, we could put the at: & at:put: prims into a plugin; then have one version that uses a cache of recently at:'d objects and one that doesn't. It would be possible to swap between them at runtime. Why? No idea, it's just an example, damnit!
[snip]
Primitives that absolutely should not be indexed primitives include things like (90 primitiveMousePoint) (91 primitiveTestDisplayDepth) "Blue Book: primitiveCursorLocPut" (92 primitiveSetDisplayMode) "Blue Book: primitiveCursorLink" (93 primitiveInputSemaphore) (94 primitiveGetNextEvent) "Blue Book: primitiveSampleInterval" (95 primitiveInputWord) (98 primitiveStoreImageSegment) (99 primitiveLoadImageSegment) (101 primitiveBeCursor) (102 primitiveBeDisplay) (103 primitiveScanCharacters) (105 primitiveStringReplace) (106 primitiveScreenSize) (107 primitiveMouseButtons) (108 primitiveKbdNext) (109 primitiveKbdPeek) (126 primitiveDeferDisplayUpdates) (127 primitiveShowDisplayRect) (133 primitiveSetInterruptKey) (134 primitiveInterruptSemaphore) (140 primitiveBeep) (141 primitiveClipboardText) (142 primitiveVMPath) (231 primitiveForceDisplayUpdate) (232 primitiveFormPrint) (233 primitiveSetFullScreen)
as these are to do with the broader context, not the core execution engine.
I'd certainly agree with the principle.
I would go further and rip out support for named primitives from the VM and put it up in the image where it belongs. I would build the machinery for looking up named primitives in the image and have a primitive that allowed one to slam a function address into a method.
For me the VM should include an execution engine, a small and fast FFI and nothing else.
An interesting idea in a completely different direction. I like a lot about it, though I still think that the core prims ought to be named and live in plugins, including the FFI. Obviously there is a gnarly bit where you can't load the FFI plugin if you rely on FFI to do the plugin loading, but what the hell, there's a bunch of us and we're smart[1].
A system built as you describe would be extremely nice. No argument there. Nonetheless I think that late-bound and rebindable prims along with replaceable VM internals would also have some interesting value. I doubt it's impossible to do both.
tim [1] sideways reference to a .sigline "The theorists tell us that this isimpossible, but what hell, there are three of us and we're smart" -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "How many Motie Watchmakers does it take to change a lightbulb?" "One. Four to change the lightbulb and seventeen to convert the old bulb into an escape capsule for all the others."
On Tue, Jul 1, 2008 at 1:21 PM, tim Rowledge tim@rowledge.org wrote:
On 1-Jul-08, at 12:25 PM, Eliot Miranda wrote:
[snip]
Gettng rid of both obsolete tables is a really good thing (and I've already done it).
Good
Getting rid of indexed primitives for the core does not save anything. There still has to be a table somewhere that maps the primitive name to the primitive function. All you do is replace a table mapping indices to functions with one mapping strings to functions. You can't reply on the OS to do the lookup because you won't be able to strip VM executables if you do and that really costs space.
Sure; there's always got to be some way of looking up things. It will always take some space and time. But surely trading space/time for more flexible and smarter is what Smalltalk is all about?
Smalltalk is about a lot of things. One is minimality (everything's an object). Use Occam's razor. Named primitives for the core prims is a flourish one doesn't need. As my first wife was fond of saying "Don't sweat the petty stuff. Pet the sweaty stuff."
More seriously, the VM should be as minimal as possible and I've shown (I think) that e.g. named primitive support can be lifted into the image and be much more flexible and extensible there. But the same can't be done with the core primitives. We can't have a circularity were the system needs to use core primitives to bind core primitives. So (I tell myself) keep it simple stupid. Just use an index.
I thought I was the only one to have watched the prisoner :) I'm not a compiler - I'm a free programmer (Context was C++)
Stef
On Jun 30, 2008, at 8:34 PM, tim Rowledge wrote:
I am trying to make time to post a few articles on some future directions I consider interesting for the core system.
First one is up at http://people.squeakfoundation.org/article/ 89.html - do please comment on the relevant page, or even here.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Oxymorons: New classic
squeak-dev@lists.squeakfoundation.org
-
Adrian Lienhard -
Andreas Raab -
Bert Freudenberg -
Craig Latta -
Damien Pollet -
David Pennell -
dpharris＠telus.net -
Eliot Miranda -
Igor Stasenko -
John M McIntosh -
Joshua Gargus -
Klaus D. Witzel -
Laurence Rozier -
Michael van der Gulik -
nicolas cellier -
Sebastian Sastre -
stephane ducasse -
Terry Raymond -
tim Rowledge