Sometimes it's too easy, part II - Vm-dev

25 Jan 2017


      Hi All,
I had stalled in recent days on the new compactor.  The compactor
appears to be working perfectly for me running on Mac OS, but when my
colleague Bob Westergaard built a Newspeak VM for linux it crashed
immediately (our Newspeak images invoke GC as soon as the low space watcher
is spawned, because we're based on an older Squeak release).
Linux crashes whereas for me Mac OS doesn't crash for the combination of
two reasons.  Linux places the heap much higher in the address space,
typically creating heaps that extend into the up[per half of the address
space.  The default type for oops in the VM is saint, which is signed.
So unless unsigned comparisons are used, oops in the upper half of the
address space, which consequently have their sign bit set, will confuse
enumerations, and hence bring the compactor crashing down.
The problem is how to find these unsigned comparisons.  I had taken the
approach of simply finding test cases and running them in the debugger, but
alas the compactor works perfectly in the simulator, because there's not
enough address space to create a simulated heap > 2Gb, so the simulator
works and the generated C doesn't work on Linux.  Running the two side by
side trying to look for where they diverge was not productive given the
difficulty of looking at a 47Mb image containing 875k objects.
And then the subconscious kicked in and I realized I could use Slang to
analyze the parse tree and look for signed comparisons.  The first cut
looked like this.  Yes, the open coding of building the code enrapture and
running the type inference is ugly, but it's not that complicated.  The
important bit at the bottom is
node isSend
 and: [(#(< > <= >=) includes: node selector)
 and: [({node receiver. node args first } anySatisfy:
[:o| (cg typeFor: o in: m)
ifNil: [false]
ifNotNil: [:t| (cg isIntegralCType: t) and: [t first ~= $u]]])]]
SpurPlanningCompactor class>>#identifySignedComparisons
"self identifySignedComparisons"
| vmm cg |
vmm := (VMMaker forPlatform: 'Cross')
interpreterClass: StackInterpreter;
options: #(ObjectMemory Spur32BitMemoryManager).
cg := [vmm buildCodeGeneratorForInterpreter]
on: Notification
do: [:ex|
ex tag == #getVMMaker
ifTrue: [ex resume: vmm]
ifFalse: [ex pass]].
cg vmClass preGenerationHook: cg.
cg inferTypesForImplicitlyTypedVariablesAndMethods.
cg retainMethods: self selectors.
cg prepareMethods.
cg doInlining: true.
self selectors do:
[:sel|
(cg methodNamed: sel) ifNotNil:
[:m|
m parseTree nodesDo:
[:node|
(node isSend
and: [(#(< > <= >=) includes: node selector)
and: [{node receiver. node args first } anySatisfy:
[:o| (cg typeFor: o in: m)
ifNil: [false]
ifNotNil: [:t| (cg isIntegralCType: t) and: [t first ~= $u]]]]]) ifTrue:
[Transcript ensureCr; nextPutAll: sel; space; print: node; flush]]]]
Using the above was a doit to collect the results in a set allowed me to
filter out the noise, and the final method looks like
identifySignedComparisons
"self identifySignedComparisons"
| vmm cg noise |
noise := #('(manager bytesInObject: largestFreeChunk) >= spaceEstimate'
'(self classIndexOf: o*) > self isForwardedObjectClassIndexPun'
'GCModeFull > 0'
'ReceiverIndex + (objectMemory integerValueOf: sp*) < (objectMemory
lengthOf: o*)'
'fmt* < manager firstCompiledMethodFormat'
'fmt* < self firstCompiledMethodFormat'
'fmt* <= 5'
'gcPhaseInProgress > 0'
'i <= finishIndex'
'i >= 0'
'numPointerSlots > 0'
'scavenger rememberedSetSize > 0').
vmm := (VMMaker forPlatform: 'Cross')
interpreterClass: StackInterpreter;
options: #(ObjectMemory Spur32BitMemoryManager).
cg := [vmm buildCodeGeneratorForInterpreter]
on: Notification
do: [:ex|
ex tag == #getVMMaker
ifTrue: [ex resume: vmm]
ifFalse: [ex pass]].
cg vmClass preGenerationHook: cg.
cg inferTypesForImplicitlyTypedVariablesAndMethods.
cg retainMethods: self selectors.
cg prepareMethods.
cg doInlining: true.
self selectors do:
[:sel|
(cg methodNamed: sel) ifNotNil:
[:m|
m parseTree nodesDo:
[:node|
(node isSend
and: [(#(< > <= >=) includes: node selector)
and: [({node receiver. node args first } anySatisfy:
[:o| (cg typeFor: o in: m)
ifNil: [false]
ifNotNil: [:t| (cg isIntegralCType: t) and: [t first ~= $u]]])
and: [noise noneSatisfy: [:n| n match: node printString]]]]) ifTrue:
[Transcript ensureCr; nextPutAll: sel; space; print: node; flush]]]]
and my output is
savedFirstFieldsSpaceInFreeChunk savedFirstFieldsSpace start >= nilObj
unmarkPinnedObjectsAndFindFirstUnpinnedOrFreeEntityFollowing: (segments at:
sweepIndex) segSize + (segments at: sweepIndex) segStart < nextObj
unmarkPinnedObjectsAndFindFirstUnpinnedOrFreeEntityFollowing: nextObj >=
manager endOfMemory
findNextMarkedPinnedAfter: nextObj >= manager endOfMemory
planCompactSavingForwarders toFinger <= (manager startOfObject: o)
planCompactSavingForwarders toFinger <= (manager startOfObject: previousPin)
updatePointers o2 >= firstFreeObject
updatePointers toFinger <= (manager startOfObject: o3)
updatePointers toFinger <= (manager startOfObject: previousPin)
savedFirstFieldsSpaceWasAllocated savedFirstFieldsSpace start >= nilObj
freeFrom:upTo:previousPin: toFinger >= (segments at: i) segStart
freeFrom:upTo:previousPin: seg segSize + seg segStart < limit
freeFrom:upTo:previousPin: start := (self byteAt: pin + 7) = self
numSlotsMask ifTrue: [pin - self baseHeaderSize] ifFalse: [pin] > toFinger
freeFrom:upTo:previousPin: (segments at: sweepIndex) segSize + (segments
at: sweepIndex) segStart < nextObj1
freeFrom:upTo:previousPin: nextObj1 >= manager endOfMemory
freeFrom:upTo:previousPin: nextUnpinned >= limit
freeFrom:upTo:previousPin: nextObj >= manager endOfMemory
unmarkPinned: (segments at: sweepIndex) segSize + (segments at: sweepIndex)
segStart < pinnedObj
compact savedFirstFieldsSpace start >= nilObj
compact savedFirstFieldsSpace start >= nilObj
updateSavedFirstFieldsSpaceIfNecessary savedFirstFieldsSpace start >= nilObj
endCompaction savedFirstFieldsSpace start >= nilObj
updatePointersInInitialImmobileObjects o >= firstFreeObject
copyAndUnmarkMobileObjects toFinger <= (manager startOfObject: o)
copyAndUnmarkMobileObjects toFinger <= (manager startOfObject: previousPin)
copyAndUnmarkMobileObjects bytes + 2 * 8 > availableSpace
copyAndUnmarkMobileObjects availableSpace > 0
releaseSavedFirstFieldsSpace savedFirstFieldsSpace start >= nilObj
updatePointersInMobileObjects toFinger <= (manager startOfObject: o)
updatePointersInMobileObjects toFinger <= (manager startOfObject:
previousPin)
Less than half an hour's work.  Now I can squash those bugs :-)
_,,,^..^,,,_
best, Eliot