Eliot Miranda uploaded a new version of Cog to project VM Maker: http://source.squeak.org/VMMaker/Cog-eem.328.mcz
==================== Summary ====================
Name: Cog-eem.328 Author: eem Time: 15 August 2016, 7:50:11.265382 pm UUID: 09c34c1e-71c5-4a9f-870c-e441d6d24011 Ancestors: Cog-eem.327
Provide Spur32BitPreen to opreen 32-bit images, removing the excessive free space left by Spur's current poor compaction algorithm.
=============== Diff against Cog-eem.327 ===============
Item was added: + SimulatorHarness subclass: #Spur32BitPreen + instanceVariableNames: 'oldHeap newHeap map reverseMap oldInterpreter newInterpreter imageHeaderFlags savedWindowSize' + classVariableNames: '' + poolDictionaries: 'VMObjectIndices' + category: 'Cog-Bootstrapping'! + + !Spur32BitPreen commentStamp: 'eem 8/15/2016 19:49' prior: 0! + A Spur32BitPreen is a simple image rewriter for 32-bit Spru images that eliminates free space and hence shrinks the preened image. Use via + Spur32BitPreen new preenImage: 'spur' + which will produce spur-preen.image and spur-preen.changes from spur.image and spur.changes. + + Instance Variables + imageHeaderFlags: <Integer> + map: <Dictionary> + newHeap: <Spur32BitMMLESimulator> + newInterpreter: <StackInterpreterSimulatorLSB> + oldHeap: <Spur32BitMMLESimulator> + oldInterpreter: <StackInterpreterSimulatorLSB> + reverseMap: <Dictionary> + savedWindowSize: <Integer> + + imageHeaderFlags + - flags word in image header + + map + - map from oops in old image to oops in new image + + newHeap + - the preened heap + + newInterpreter + - the interpreter wrapping the preened heap + + oldHeap + - the heap to be preened + + oldInterpreter + - the interpreter wrapping the heap to be preened + + reverseMap + - map from oops in new image to oops in old image + + savedWindowSize + - screen size word in mage header + !
Item was added: + ----- Method: Spur32BitPreen>>clone: (in category 'bootstrap image') ----- + clone: oldObj + | newObj hash | + newObj := newHeap + allocateSlots: (oldHeap numSlotsOf: oldObj) + format: (oldHeap formatOf: oldObj) + classIndex: (oldHeap classIndexOf: oldObj). + (hash := oldHeap rawHashBitsOf: oldObj) ~= 0 ifTrue: + [newHeap setHashBitsOf: newObj to: hash]. + (oldHeap isImmutable: oldObj) ifTrue: + [newHeap setIsImmutableOf: newObj to: true]. + (oldHeap isPinned: oldObj) ifTrue: + [newHeap setIsPinnedOf: newObj to: true]. + self deny: (oldHeap isRemembered: oldObj). + self deny: (oldHeap isMarked: oldObj). + self deny: (oldHeap isGrey: oldObj). + reverseMap at: newObj put: oldObj. + ^map at: oldObj put: newObj!
Item was added: + ----- Method: Spur32BitPreen>>cloneFreeLists: (in category 'bootstrap image') ----- + cloneFreeLists: oldObj + | newObj | + newObj := newHeap + allocateSlots: newHeap numFreeLists + format: newHeap wordIndexableFormat + classIndex: newHeap wordSizeClassIndexPun. + reverseMap at: newObj put: oldObj. + ^map at: oldObj put: newObj!
Item was added: + ----- Method: Spur32BitPreen>>cloneObjects (in category 'bootstrap image') ----- + cloneObjects + "Clone all normal objects. Of hidden objects only clone the freeLists object and + the classTableRoot and class table pages. In particular, dont clone objStacks. + The refs to the objStacks are nilled out in fillInHeap." + | i freeListsObject | + i := 0. + freeListsObject := oldHeap freeListsObject. + oldHeap allOldSpaceObjectsDo: + [:obj32| + (i := i + 1) >= 100000 ifTrue: + [Transcript nextPut: $:; flush. i := 0]. + obj32 = freeListsObject + ifTrue: + [self cloneFreeLists: obj32] + ifFalse: + [(self shouldClone: obj32) ifTrue: + [self clone: obj32]]]!
Item was added: + ----- Method: Spur32BitPreen>>fillInBitsObject:from: (in category 'bootstrap image') ----- + fillInBitsObject: obj64 from: obj32 + 0 to: (oldHeap numBytesOf: obj32) - 1 do: + [:i| + newHeap + storeByte: i + ofObject: obj64 + withValue: (oldHeap fetchByte: i ofObject: obj32)]!
Item was added: + ----- Method: Spur32BitPreen>>fillInClassTable (in category 'bootstrap image') ----- + fillInClassTable + self fillInPointerObject: (map at: oldHeap classTableRootObj) from: oldHeap classTableRootObj!
Item was added: + ----- Method: Spur32BitPreen>>fillInCompiledMethod:from: (in category 'bootstrap image') ----- + fillInCompiledMethod: newObj from: oldObj + "oldInterpreter printOop: oldObj" + "newInterpreter printOop: newObj" + 0 to: (oldHeap numPointerSlotsOf: oldObj) - 1 do: + [:i| | oldOop newOop | + oldOop := oldHeap fetchPointer: i ofObject: oldObj. + newOop := self map32BitOop: oldOop. + newHeap + storePointerUnchecked: i + ofObject: newObj + withValue: newOop]. + (oldInterpreter startPCOfMethod: oldObj) + to: (oldHeap numBytesOf: oldObj) - 1 + do: [:j| + newHeap + storeByte: j + ofObject: newObj + withValue: (oldHeap fetchByte: j ofObject: oldObj)]!
Item was added: + ----- Method: Spur32BitPreen>>fillInHeap (in category 'bootstrap image') ----- + fillInHeap + | heapEnd freeListsObj | + heapEnd := newHeap freeStart. + newHeap + nilObject: (map at: oldHeap nilObject); + falseObject: (map at: oldHeap falseObject); + trueObject: (map at: oldHeap trueObject); + specialObjectsOop: (map at: oldHeap specialObjectsOop); + lastHash: oldHeap lastHash; + setHiddenRootsObj: (map at: oldHeap classTableRootObj). + newHeap segmentManager + initSegmentForInImageCompilationFrom: newHeap nilObject + to: heapEnd + newHeap bridgeSize. + freeListsObj := newHeap objectAfter: newHeap trueObject. + "Nil-out the free lists." + newHeap + fillObj: freeListsObj numSlots: (newHeap numSlotsOf: freeListsObj) with: 0; + initializeFreeSpacePostLoad: freeListsObj; + initializePostBootstrap; + setEndOfMemory: (newHeap segmentManager bridgeAt: 0) + newHeap baseHeaderSize!
Item was added: + ----- Method: Spur32BitPreen>>fillInObjects (in category 'bootstrap image') ----- + fillInObjects + "interpreter32 printOop: oldObj" + | i | + {oldHeap markStack. oldHeap weaklingStack. oldHeap mournQueue} do: + [:obj| + obj ~= oldHeap nilObject ifTrue: + [map at: obj put: (map at: oldHeap nilObject)]]. + i := 0. + oldHeap allObjectsDo: + [:oldObj| + (i := i + 1) >= 10000 ifTrue: + [Transcript nextPut: $.; flush. i := 0]. + (map at: oldObj ifAbsent: nil) + ifNotNil: + [:newObj| | format | + (newHeap numSlotsOf: newObj) > 0 ifTrue: "filter-out filtered objStack pages" + [format := oldHeap formatOf: oldObj. + (newHeap isPointersFormat: format) + ifTrue: + [self fillInPointerObject: newObj from: oldObj] + ifFalse: + [(newHeap isCompiledMethodFormat: format) + ifTrue: [self fillInCompiledMethod: newObj from: oldObj] + ifFalse: [self fillInBitsObject: newObj from: oldObj]]]] + ifNil: [self assert: (self isUnmappedObject: oldObj)]]!
Item was added: + ----- Method: Spur32BitPreen>>fillInPointerObject:from: (in category 'bootstrap image') ----- + fillInPointerObject: newObj from: oldObj + 0 to: (oldHeap numSlotsOf: oldObj) - 1 do: + [:i| + newHeap + storePointerUnchecked: i + ofObject: newObj + withValue: (self map32BitOop: (oldHeap fetchPointer: i ofObject: oldObj))]!
Item was added: + ----- Method: Spur32BitPreen>>incrementPCField:ofObject:for: (in category 'bootstrap image') ----- + incrementPCField: fieldIndex ofObject: obj64 for: method32 + | value nLits | + value := newHeap fetchPointer: fieldIndex ofObject: obj64. + (newHeap isIntegerObject: value) + ifTrue: + [nLits := oldHeap literalCountOf: method32. + newHeap + storePointerUnchecked: fieldIndex + ofObject: obj64 + withValue: (newHeap integerObjectOf: nLits + LiteralStart * 4 + (newHeap integerValueOf: value))] + ifFalse: + [self assert: (reverseMap at: value) = oldHeap nilObject]!
Item was added: + ----- Method: Spur32BitPreen>>initMaps (in category 'initialize-release') ----- + initMaps + map := Dictionary new: oldHeap memory size // 32. + reverseMap := Dictionary new: oldHeap memory size // 32!
Item was added: + ----- Method: Spur32BitPreen>>isUnmappedObject: (in category 'bootstrap image') ----- + isUnmappedObject: obj32 + "Answer if obj32 is an object that is not cloned by the bootstrap." + ^obj32 = oldHeap freeListsObject + or: [(oldHeap isValidObjStackPage: obj32)]!
Item was added: + ----- Method: Spur32BitPreen>>map32BitOop: (in category 'bootstrap image') ----- + map32BitOop: obj32 + "oldInterpreter printOop: obj32" + ^map + at: obj32 + ifAbsent: + [(oldHeap isImmediateCharacter: obj32) + ifTrue: [newHeap characterObjectOf: (oldHeap characterValueOf: obj32)] + ifFalse: [newHeap integerObjectOf: (oldHeap integerValueOf: obj32)]]!
Item was added: + ----- Method: Spur32BitPreen>>on: (in category 'public access') ----- + on: imageName + (oldInterpreter := StackInterpreterSimulator newWithOptions: #(ObjectMemory Spur32BitMemoryManager)) + openOn: imageName extraMemory: 0. + oldHeap := oldInterpreter objectMemory. + imageHeaderFlags := oldInterpreter getImageHeaderFlags. + savedWindowSize := oldInterpreter savedWindowSize. + newInterpreter := StackInterpreterSimulator newWithOptions: #(ObjectMemory Spur32BitMemoryManager). + newHeap := newInterpreter objectMemory. + newHeap + allocateMemoryOfSize: oldHeap oldSpaceSize + newSpaceSize: 8 * 1024 * 1024 + stackSize: 16 * 1024 + codeSize: 0. + newHeap bootstrapping: true. + self initMaps!
Item was added: + ----- Method: Spur32BitPreen>>preenImage (in category 'public access') ----- + preenImage + self cloneObjects. + self fillInObjects. + self fillInClassTable. + self fillInHeap. + self reportSizes!
Item was added: + ----- Method: Spur32BitPreen>>preenImage: (in category 'public access') ----- + preenImage: imageName + (Smalltalk classNamed: #FileReference) ifNotNil: + [^self preenImageUsingFileReference: imageName]. + (Smalltalk classNamed: #FileDirectory) ifNotNil: + [^self preenImageUsingFileDirectory: imageName]. + self error: 'at a loss as to what file system support to use'!
Item was added: + ----- Method: Spur32BitPreen>>preenImageUsingFileDirectory: (in category 'public access') ----- + preenImageUsingFileDirectory: imageName + | dirName baseName dir | + dirName := FileDirectory dirPathFor: imageName. + baseName := (imageName endsWith: '.image') + ifTrue: [FileDirectory baseNameFor: imageName] + ifFalse: [FileDirectory localNameFor: imageName]. + dir := dirName isEmpty ifTrue: [FileDirectory default] ifFalse: [FileDirectory default on: dirName]. + self on: (dir fullNameFor: baseName, '.image'). + [self preenImage] + on: Halt + do: [:ex| + "suppress halts from the usual suspects (development time halts)" + (#(fullGC compactImage) includes: ex signalerContext sender selector) + ifTrue: [ex resume] + ifFalse: [ex pass]]. + self writeSnapshot: (dir fullNameFor: baseName, '-preen.image') + headerFlags: imageHeaderFlags + screenSize: savedWindowSize. + dir deleteFileNamed: baseName, '-preen.changes'; + copyFileNamed: baseName, '.changes' toFileNamed: baseName, '-preen.changes'!
Item was added: + ----- Method: Spur32BitPreen>>reportSizes (in category 'bootstrap image') ----- + reportSizes + | change oldHeapSize newHeapSize | + oldHeapSize := oldHeap oldSpaceSize. + newHeapSize := newHeap oldSpaceSize. + change := newHeapSize - oldHeapSize / oldHeapSize. + Transcript + nextPutAll: 'done.'; cr; + nextPutAll: 'old heap size: '; nextPutAll: oldHeapSize asStringWithCommas; tab; + nextPutAll: 'initial new heap size: '; nextPutAll: newHeapSize asStringWithCommas; cr; + nextPutAll: 'change: '; print: change * 100.0 maxDecimalPlaces: 2; nextPut: $%; cr; + flush. + newHeapSize := newHeap endOfMemory + - newHeap scavenger eden limit + - newHeap totalFreeListBytes. + change := newHeapSize - oldHeapSize / oldHeapSize. + Transcript + nextPutAll: 'final new heap size: '; nextPutAll: newHeapSize asStringWithCommas; tab; + nextPutAll: 'change: '; print: change * 100.0 maxDecimalPlaces: 2; nextPut: $%; cr; + flush!
Item was added: + ----- Method: Spur32BitPreen>>shouldClone: (in category 'bootstrap image') ----- + shouldClone: obj32 + ^(oldHeap isValidObjStackPage: obj32) not!
Item was added: + ----- Method: Spur32BitPreen>>writeSnapshot:headerFlags:screenSize: (in category 'snapshot') ----- + writeSnapshot: imageFileName headerFlags: headerFlags screenSize: screenSizeInteger + newHeap + checkFreeSpace; + runLeakCheckerForFullGC. + newInterpreter + setImageHeaderFlagsFrom: headerFlags; + setDisplayForm: nil; + setSavedWindowSize: savedWindowSize; + imageName: imageFileName; + writeImageFileIO. + Transcript cr; show: 'Done!!'!
Item was changed: ----- Method: Spur32to64BitBootstrap>>cloneFreeLists: (in category 'bootstrap image') ----- cloneFreeLists: obj32 | obj64 | obj64 := heap64 allocateSlots: heap64 numFreeLists + format: heap64 wordIndexableFormat + classIndex: heap64 wordSizeClassIndexPun. - format: heap64 sixtyFourBitIndexableFormat - classIndex: heap64 sixtyFourBitLongsClassIndexPun. reverseMap at: obj64 put: obj32. ^map at: obj32 put: obj64!
vm-dev@lists.squeakfoundation.org