Branch: refs/heads/Cog
Home: https://github.com/OpenSmalltalk/opensmalltalk-vm
Commit: 6df629d160c99e374ac6e9d2bde230281867e77b
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/6df629d160c99e374a…
Author: Eliot Miranda <eliot.miranda(a)gmail.com>
Date: 2018-10-04 (Thu, 04 Oct 2018)
Changed paths:
M build.linux64x64/bochsx64/conf.COG
M build.linux64x64/bochsx64/conf.COG.dbg
M build.linux64x64/bochsx86/conf.COG
M platforms/unix/plugins/BochsIA32Plugin/Makefile.inc
M platforms/unix/plugins/BochsX64Plugin/Makefile.inc
Log Message:
-----------
Remove spurious -m32 directives from the linux Bochs plugin makefiles.
Add -fPIC to the support library builds.
**NOTE:** This service has been marked for deprecation: https://developer.github.com/changes/2018-04-25-github-services-deprecation/
Functionality will be removed from GitHub.com on January 31st, 2019.
Hi everybody,
I'm forwarding an email I sent to Eliot as he suggested. It is about
changes I'm doing to the VM to get type information dynamically.
Thanks Eliot!
Hernan.
---------- Forwarded message ---------
From: Hernan Wilkinson <hernan.wilkinson(a)10pines.com>
Date: Thu, Oct 4, 2018 at 7:09 AM
Subject: Re: Help for a crazy idea I have
To: Eliot Miranda <eliot.miranda(a)gmail.com>
Hi Eliot!
thank you for your answer.
I look at Slots as you suggested. They could work but, at least as far I
could see and play with them, they would not work for all the cases I have
in mind, for example I could not use slots to keep the type of temporary
variables, parameters or methods return, etc. If I'm mistaken please let me
know.
So, I continued with my idea and changed ClassDescription adding a new
inst. var. called "instanceVariablesTypes", that points to an array of
arrays. Each position of the first one points to the types of the inst.
var. whose index correspond the position. Very simple and it works fine.
Performance impact is very low. At the end of the email is the code I wrote
at the VM in case you want to see it.
Now I'm working on the temporary variables types using
AdditionalMethodState.
But the main reason I'm writing to you again is because I found a rare
behavior and an error when running the simulator with the Cuis image. I'm
using the StackInterpreter. The problems are are:
1) The simulator running a Cuis image does more scavengings than running
the simulator with a Squeak image, even thought the Cuis image size is 9 mb
and the Squeak one is 50 mb. (I know the image size has nothing to do with
the young space size). So, is there a way to configure the young space size
or a way to see why it triggers so many scavengings?
2) When I added the inst. var "instanceVariableTypes" to ClassDescription,
some pointer ends up invalid. Doing the same with Squeak works fine.
When adding that inst var in an Cuis image using the simulator, an
assertion stops the simulator execution. There is another detail, this does
not happen with 32 bit Cuis images/vm, only with 64 bit Cuis images. Do you
know what could be happening? (I'm attaching the simulator's stack report
just is case you have time to look at it, but it is very simple to
reproduce).
3) What is the difference between "<primitive: 105 error: ec>" and just
"<primitive: 105>" ?
I had to replace in Cuis all the first one with the second one because the
simulator did not work with methods using the first one (example:
Array>>replaceFrom: start to: stop with: replacement startingAt: repStart)
Anyway, it is very interesting what I'm achieving with just only keeping
the types for inst. vars. I could significantly improve auto complete in
the class browser for example, do some type checking, etc.
I'll keep you posted with the changes I'm making.
If you could help me with the 3 problems I mentioned above it would be
great :-)
Cheers!
Hernan
-----------------
Here is the code I changed in the VM:
storeAndPopReceiverVariableBytecode
| rcvr top instVarIndex |
rcvr := self receiver.
top := self internalStackTop.
instVarIndex := currentBytecode bitAnd: 7.
self internalPop: 1.
objectMemory
storePointerImmutabilityCheck: instVarIndex
ofObject: rcvr
withValue: top.
self fetchNextBytecode.
self keepInstanceVariableTypeInformationFor: top in: rcvr at: instVarIndex.
"<--- added this"
extendedStoreBytecodePop: popBoolean
| descriptor variableType variableIndex value rcvr |
<inline: true>
descriptor := self fetchByte.
variableType := descriptor >> 6 bitAnd: 3.
variableIndex := descriptor bitAnd: 63.
value := self internalStackTop.
popBoolean ifTrue: [ self internalPop: 1 ].
variableType = 0 ifTrue:
[rcvr := self receiver.
objectMemory storePointerImmutabilityCheck: variableIndex ofObject: rcvr
withValue: value.
self keepInstanceVariableTypeInformationFor: value in: rcvr at:
variableIndex. "<--- added this"
^ self fetchNextBytecode].
variableType = 1 ifTrue:
[ self fetchNextBytecode.
^self temporary: variableIndex in: localFP put: value].
variableType = 3 ifTrue:
[self storeLiteralVariable: variableIndex withValue: value.
^ self fetchNextBytecode].
self error: 'illegal store'
keepInstanceVariableTypeInformationFor: anAssignedObject in: rcvr at:
instVarIndex
| assignedObjectClass assignedObjectClassTag instVarTypes rcvrClass
rcvrClassTag types |
"(objectMemory isForwarded: rcvr) ifTrue: [ self halt ]."
rcvrClassTag := objectMemory fetchClassTagOf: rcvr.
self deny: (objectMemory isForwardedClassTag: rcvrClassTag).
rcvrClass := objectMemory classForClassTag: rcvrClassTag.
self deny: rcvrClass isNil.
"(objectMemory isForwarded: top) ifTrue: [ self halt ]."
assignedObjectClassTag := objectMemory fetchClassTagOf: anAssignedObject.
self deny: (objectMemory isForwardedClassTag: assignedObjectClassTag).
assignedObjectClass := objectMemory classForClassTag:
assignedObjectClassTag.
self deny: assignedObjectClass isNil.
instVarTypes := objectMemory followObjField: 5 ofObject: rcvrClass. "TODO:
use constant instead of 5"
instVarTypes = objectMemory nilObject ifTrue: [ ^self ].
types := objectMemory followObjField: instVarIndex ofObject: instVarTypes.
0 to: 9 do: [ :index | | typeAtIndex | "TODO: use array size and not 9.
Check for types == nil"
typeAtIndex := objectMemory followObjField: index ofObject: types.
typeAtIndex == assignedObjectClass ifTrue: [ ^self ].
typeAtIndex == objectMemory nilObject ifTrue: [ ^objectMemory storePointer:
index ofObject: types withValue: assignedObjectClass ]].
"self halt: 'out of space for types"
On Wed, Sep 19, 2018 at 1:48 PM Eliot Miranda <eliot.miranda(a)gmail.com>
wrote:
> Hi Hernan,
>
> On Sun, Sep 16, 2018 at 1:43 PM Hernan Wilkinson <
> hernan.wilkinson(a)10pines.com> wrote:
>
>> Hi Eliot,
>> I have had this idea for a long time, some guys at the university
>> implemented in pharo using metalinks (Marcus's compiler capability) but it
>> was a toy implementation... a few days ago I cloned the cog repo to show
>> people at work how easy is to run the Smalltalk vm as a simulation and see
>> how everything is implemented, change it, etc. so I started to implement
>> this idea but of course, I need help :-)
>> The idea is simple: Every time an object is assigned to a variable, I
>> want to keep its class to have "real type inference".
>> The idea is based on the fact that Smalltalk is always running, so
>> "class" information will be generated with real-running code. Of course, we
>> will have to derive type info from that class info, but that is a problem
>> to be solved in the image space.
>> So, I did some experiments and modified the bytecodes that store inst.
>> vars. (#storeAndPopReceiverVariableBytecode and #extendedStoreBytecodePop:)
>> to keep that info, but they doit in a dictionary that belongs to the
>> simulator space.
>> What I want to do is to store that info in a way that should be
>> accesible from the image.
>> I added an inst. var. to ClassDescription called
>> 'instanceVariablesTypes' that it is initialized with an array of the number
>> of inst. vars. of the class and at each index there is a Set (it could be a
>> Dictionary or an Array...) where I want to store the assigned object class
>> to the inst. var
>> The problem I have is that I do not know how to manage those objects
>> from the image space. I need to be able to add objects to the Set that
>> keeps the classes of an inst. var, make the array of instanceVariablesTypes
>> change accordingly to changes in the class, etc.
>> So basically my question are:
>> 1) is there a way to manage a Set or a Dictionary from the vm but that
>> was created at "the image". The method lookup has the code to lookup in a
>> dict., but I have not seen code to create, add, remove elements
>> 2) if there is not, I could do everything with arrays, but to do so I
>> need to know how an array can be instantiated from the vm
>> 3) Any other suggestion?
>>
>
> Yes, I would look at Slots. With Slots you should b e sable to manage
> everything at the image and work on a subset of all the classes. Kernel
> classes should be avoided since one will get into an infinite recursion
> (and indeed using the Vm would be a solution here). The problem with using
> the VM is that one has to use very low-level data structures and tackle the
> JIT to get performance, It is not easy, whereas Slots should get you the
> information you need very very cheaply.
>
>
>>
>> I know this idea would slow down the execution of the vm, but my idea is
>> based on the fact that while programming (development time) the speed is
>> not so critical, so we could have an option to not do this on runtime time
>> (or different vms...)
>> I also know that keeping the type info for inst. vars. is not enough, the
>> same has to be done with methods (parameters, temporaries and return type),
>> class variables, and keeping types for collections should be based per
>> instance and not per class (like generics do in other languages: Array<T>),
>> etc. Also, I do not know how this idea could conflict with jiting (I played
>> with the stack vm only), etc.
>> There are a lot of things to do, but I think having this information
>> available from the image would help a lot on tools like refactoring, syntax
>> highlighting, type checking, etc etc.
>>
>
> Again, using Slots instead of the VM would make it much easier to access
> the information at the image level.
>
>
>>
>> Well, the email is too long already :-)
>> I hope you can send me some ideas to implement this.
>>
>
> Take a look at Slots. If that doesn't work then indeed I can help you do
> it in the VM. But I warn you; it'll be hairy (but fun!! :-) ).
>
>
>>
>> Thanks!
>> Hernan
>>
>> --
>>
>> *Hernán WilkinsonAgile Software Development, Teaching & Coaching*
>> *Phone: +54-011*-4893-2057
>> *Twitter: @HernanWilkinson*
>> *site: http://www.10Pines.com <http://www.10pines.com/>*
>> Address: Alem 896, Floor 6, Buenos Aires, Argentina
>>
>
>
> --
> _,,,^..^,,,_
> best, Eliot
>
--
*Hernán WilkinsonAgile Software Development, Teaching & Coaching*
*Phone: +54-011*-4893-2057
*Twitter: @HernanWilkinson*
*site: http://www.10Pines.com <http://www.10pines.com/>*
Address: Alem 896, Floor 6, Buenos Aires, Argentina
--
*Hernán WilkinsonAgile Software Development, Teaching & Coaching*
*Phone: +54-011*-4893-2057
*Twitter: @HernanWilkinson*
*site: http://www.10Pines.com <http://www.10pines.com/>*
Address: Alem 896, Floor 6, Buenos Aires, Argentina
ClementBera uploaded a new version of VMMaker to project VM Maker:
http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2447.mcz
==================== Summary ====================
Name: VMMaker.oscog-cb.2447
Author: cb
Time: 4 October 2018, 9:41:04.472108 am
UUID: b2f39c11-1d5c-43a9-8925-32eab1591459
Ancestors: VMMaker.oscog-cb.2446
Production VM:
- Fixed a bug in removeSegment: where invalid memory would be accessed when the segment info array was full & a segment removed.
SelectiveCompactor VM:
- Fixed a bug in primitiveResume when a context forwarder is found.
- Fixed SelectiveCompactor postCompaction forwarding logic
-> Just ran SelectiveCompactor with >20Gb workloads without crashes with these fixes.
Might be some redundancy that I need to check in following logic with other code in the VM.
=============== Diff against VMMaker.oscog-cb.2446 ===============
Item was changed:
----- Method: CoInterpreterPrimitives>>primitiveResume (in category 'process primitives') -----
primitiveResume
"Put this process on the scheduler's lists thus allowing it to proceed next time there is
a chance for processes of it's priority level. It must go to the back of its run queue so
as not to preempt any already running processes at this level. If the process's priority
is higher than the current process, preempt the current process."
| proc inInterpreter |
proc := self stackTop. "rcvr"
+ (objectMemory isContext: (objectMemory followField: SuspendedContextIndex ofObject: proc)) ifFalse:
- (objectMemory isContext: (objectMemory fetchPointer: SuspendedContextIndex ofObject: proc)) ifFalse:
[^self primitiveFail].
"We're about to switch process, either to an interpreted frame or a
machine code frame. To know whether to return or enter machine code
we have to know from whence we came. We could have come from the
interpreter, either directly or via a machine code primitive. We could have
come from machine code. The instructionPointer tells us where from:"
inInterpreter := instructionPointer >= objectMemory startOfMemory.
(self resume: proc preemptedYieldingIf: preemptionYields from: CSResume) ifTrue:
[self forProcessPrimitiveReturnToExecutivePostContextSwitch: inInterpreter]
"Personally I would like to check MyList, which should not be one of the elements of the scheduler lists.
But there are awful race conditions in things like should:notTakeMoreThan: that mean we can't.
eem 9/27/2010 23:08. e.g.
| proc myList classLinkedList |
proc := self stackTop.
myList := objectMemory fetchPointer: MyListIndex ofObject: proc.
classLinkedList := self superclassOf: (objectMemory splObj: ClassSemaphore).
((self fetchClassOfNonInt: myList) ~= classLinkedList
and: [objectMemory isContext: (objectMemory fetchPointer: SuspendedContextIndex ofObject: proc)]) ifFalse:
[^self primitiveFail].
''We're about to switch process, either to an interpreted frame or a
machine code frame. To know whether to return or enter machine code
we have to know from whence we came. We could have come from the
interpreter, either directly or via a machine code primitive. We could have
come from machine code. The instructionPointer tells us where from:''
inInterpreter := instructionPointer >= objectMemory startOfMemory.
(self resume: proc preemptedYieldingIf: preemptionYields from: CSResume) ifTrue:
[self forProcessPrimitiveReturnToExecutivePostContextSwitch: inInterpreter]"!
Item was added:
+ ----- Method: SpurMemoryManager>>followClassTable (in category 'selective compaction') -----
+ followClassTable
+ "In addition to postBecomeScanClassTable:, I follow hiddenRootObj and its pages"
+ (self isForwarded: hiddenRootsObj) ifTrue: [hiddenRootsObj := self followForwarded: hiddenRootsObj].
+ 0 to: numClassTablePages - 1 do:
+ [:i| | page |
+ page := self followField: i ofObject: hiddenRootsObj.
+ 0 to: (self numSlotsOf: page) - 1 do:
+ [:j| | classOrNil |
+ classOrNil := self fetchPointer: j ofObject: page.
+ classOrNil ~= nilObj ifTrue:
+ [(self isForwarded: classOrNil) ifTrue:
+ [classOrNil := self followForwarded: classOrNil.
+ self storePointer: j ofObject: page withValue: classOrNil].
+ (self rawHashBitsOf: classOrNil) = 0 ifTrue:
+ [self storePointerUnchecked: j ofObject: page withValue: nilObj.
+ "If the removed class is before the classTableIndex, set the
+ classTableIndex to point to the empty slot so as to reuse it asap."
+ (i << self classTableMajorIndexShift + j) < classTableIndex ifTrue:
+ [classTableIndex := i << self classTableMajorIndexShift + j]]]]].
+ "classTableIndex must never index the first page, which is reserved for classes known to the VM."
+ self assert: classTableIndex >= (1 << self classTableMajorIndexShift).
+ self assert: self validClassTableRootPages.!
Item was added:
+ ----- Method: SpurMemoryManager>>followProcessList (in category 'selective compaction') -----
+ followProcessList
+ "Eagerly patch all process related forwarders, except suspended contexts which are lazily patched in wakeHighestPriority"
+ | scheduler processLists processList proc |
+ scheduler := self followField: ValueIndex ofObject: (self splObj: SchedulerAssociation).
+ self followField: ActiveProcessIndex ofObject: scheduler.
+ processLists := self followField: ProcessListsIndex ofObject: scheduler.
+ 0 to: (self numSlotsOf: processLists) - 1 do: [:i |
+ processList := self followField: i ofObject: processLists.
+ self followField: LastLinkIndex ofObject: processList.
+ proc := self followField: FirstLinkIndex ofObject: processList.
+ [proc = nilObj] whileFalse: [proc := self followField: NextLinkIndex ofObject: proc]].!
Item was changed:
----- Method: SpurSegmentManager>>removeSegment: (in category 'growing/shrinking memory') -----
removeSegment: emptySeg
<var: #emptySeg type: #'SpurSegmentInfo *'>
| i |
i := self indexOfSegment: emptySeg.
self assert: i > 0.
totalHeapSizeIncludingBridges := totalHeapSizeIncludingBridges - emptySeg segSize.
manager sqDeallocateMemorySegmentAt: emptySeg segStart asVoidPointer OfSize: emptySeg segSize.
+ i to: numSegments - 2 do:
- i to: numSegments - 1 do:
[:j| segments at: j put: (segments at: j + 1)].
self cCode: [] inSmalltalk: [segments at: numSegments - 1 put: SpurSegmentInfo new].
numSegments := numSegments - 1.
self bridgeFrom: (self addressOf: (segments at: i - 1))
to: (i <= (numSegments - 1) ifTrue: [self addressOf: (segments at: i)]).
manager setLastSegment: (self addressOf: (segments at: numSegments - 1))!
Item was changed:
----- Method: SpurSelectiveCompactor>>postCompactionAction (in category 'compaction') -----
postCompactionAction
| allFlags |
"For now we don't optimize and just follow everything everywhere on stack and in caches, let's see in the profiler if we need to optimize with those cases. My guess is that this is < 100 microSecond"
manager followSpecialObjectsOop.
allFlags := BecamePointerObjectFlag + BecameActiveClassFlag bitOr: BecameCompiledMethodFlag.
+ "Note: there is not the OldBecameNewFlag"
+ "gcMode flag is cleared after postBecomeAction, reset it."
- "should be gcMode Become - gcMode flag is cleared after postBecomeAction"
manager coInterpreter postBecomeAction: allFlags.
- manager postBecomeScanClassTable: allFlags.
manager coInterpreter setGCMode: GCModeFull.
+ "Special to selective, crazy objects can be forwarded..."
+ "manager postBecomeScanClassTable: allFlags. => Done in followClassTable"
+ manager followClassTable.
+ manager followProcessList.
+ manager followForwardedObjStacks.
+
"Not sure the following are needed...
coInterpreter mapInterpreterOops.
manager mapExtraRoots."
self assert: manager validClassTableHashes.!