Vm-dev April 2018

vm-dev@lists.squeakfoundation.org

43 participants
235 discussions

Identify console executable and standard one
by Vincent.Blondeau＠lamresearch.com 27 Apr '18

27 Apr '18

Hi, I would like to know from inside the image whether the ST image is a console executable, e.g. PharoConsole.exe vs Pharo.exe (Related to this pull request: https://github.com/pharo-project/pharo/pull/694). The solution is to add a primitive inside the VM but I have no idea where I should look for... Could someone give me some pointers? Thanks in advance! Cheers, Vincent Blondeau Software Engineer, Software and Controls | Global Product Group Desk +1 510.572.7499 Lam Research Corporation 4650 Cushing Pkwy, Fremont, CA 94538 USA | www.lamresearch.com Connect with Lam Research: Facebook<https://www.facebook.com/LamResearchCorporation> | Twitter<https://twitter.com/lamresearch> | LinkedIn<https://www.linkedin.com/company/lam-research>

5 7

VM Maker: VMMaker.oscog-cb.2379.mcz
by commits＠source.squeak.org 27 Apr '18

27 Apr '18

ClementBera uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2379.mcz ==================== Summary ==================== Name: VMMaker.oscog-cb.2379 Author: cb Time: 27 April 2018, 1:15:44.874393 pm UUID: 92e7c689-d9a4-4d98-bc58-9aa5f10f57e2 Ancestors: VMMaker.oscog-cb.2378 Started implementation of Tracking. Refactor a bit more to share code between selective and tracking. Progress on this topic will take longer time that expected, deadlines are further than expected... =============== Diff against VMMaker.oscog-cb.2378 =============== Item was removed: - SpurSweeper subclass: #SpurAnalysingSweeper - instanceVariableNames: '' - classVariableNames: '' - poolDictionaries: '' - category: 'VMMaker-SpurMemoryManager'! - - !SpurAnalysingSweeper commentStamp: 'cb 4/27/2018 09:45' prior: 0! - Abstract class, in addition to SpurSweeper, while sweeping the heap I annotate segments with occupation rate. This can be used by a compacting algorithm to compact only segments which are not used a lot.! Item was removed: - ----- Method: SpurAnalysingSweeper>>globalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- - globalSweepAndSegmentOccupationAnalysis - self internalGlobalSweepAndSegmentOccupationAnalysis. - manager checkFreeSpace: GCModeFull. - manager unmarkSurvivingObjectsForCompact.! Item was removed: - ----- Method: SpurAnalysingSweeper>>internalGlobalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- - internalGlobalSweepAndSegmentOccupationAnalysis - "Iterate over old space, free unmarked objects, annotate each segment with each occupation" - | currentEntity nextBridge start segmentIndex currentUsed currentUnused | - currentEntity := manager firstObject. - nextBridge := manager segmentManager bridgeAt: 0. - segmentIndex := currentUnused := currentUsed := 0. - [self oop: currentEntity isLessThan: manager endOfMemory] whileTrue: - [currentEntity = nextBridge - ifTrue: - ["End of segment, set occupation" - self - setOccupationAtIndex: segmentIndex - used: currentUsed - unused: currentUnused. - currentUnused := currentUsed := 0. - segmentIndex := segmentIndex + 1. - self unmark: currentEntity. - nextBridge := manager segmentManager bridgeAt: segmentIndex] - ifFalse: - ["In-segment, sweep and compute occupation" - (self canUseAsFreeSpace: currentEntity) - ifTrue: - ["bulkFreeChunkFrom: may change a 1 word header - object to a double word header object" - start := manager startOfObject: currentEntity. - self bulkFreeChunkFrom: currentEntity. - currentEntity := manager objectStartingAt: start. - currentUnused := currentUnused + (manager numSlotsOfAny: currentEntity)] - ifFalse: - [self unmark: currentEntity. - currentUsed := currentUsed + (manager numSlotsOfAny: currentEntity)]]. - currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. - "set last segment (last bridge = endOfMemory)" - self - setOccupationAtIndex: segmentIndex - used: currentUsed - unused: currentUnused.! Item was removed: - ----- Method: SpurAnalysingSweeper>>occupationOf: (in category 'segment access') ----- - occupationOf: segInfo - self subclassResponsibility! Item was removed: - ----- Method: SpurAnalysingSweeper>>setOccupationAtIndex:used:unused: (in category 'segment access') ----- - setOccupationAtIndex: segmentIndex used: used unused: unused - self subclassResponsibility! Item was added: + SpurSweeper subclass: #SpurAnalysingSweeperCompactor + instanceVariableNames: 'segmentToFill' + classVariableNames: 'MaxOccupationForCompaction' + poolDictionaries: '' + category: 'VMMaker-SpurMemoryManager'! + + !SpurAnalysingSweeperCompactor commentStamp: 'cb 4/27/2018 10:48' prior: 0! + Abstract class, in addition to SpurSweeper, while sweeping the heap I annotate segments with occupation rate. This is then used by compacting algorithms to compact only segments which are not used that much.! Item was added: + ----- Method: SpurAnalysingSweeperCompactor class>>declareCVarsIn: (in category 'translation') ----- + declareCVarsIn: aCCodeGenerator + aCCodeGenerator var: 'segmentToFill' type: #'SpurSegmentInfo *'! Item was added: + ----- Method: SpurAnalysingSweeperCompactor class>>initialize (in category 'initialization') ----- + initialize + super initialize. + "If the segment is occupied by more than MaxOccupationForCompaction, + it's not worth compacting it, whatever the rest of the system looks like. + MaxOccupationForCompaction is included in [0;16rFFFF]." + MaxOccupationForCompaction := 16rA000. "Basically if segment is occupied by more than 60%, not worth compacting"! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>allocateSegmentToFill (in category 'segment to fill') ----- + allocateSegmentToFill + | res | + res := manager growOldSpaceByAtLeast: manager growHeadroom. + res ifNil: [self error: 'not enough memory for selective compaction'].! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>assertNoSegmentBeingCompacted (in category 'compaction') ----- + assertNoSegmentBeingCompacted + "Assertion only - no segment is being claimed at this point" + | segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> + 0 to: manager numSegments - 1 do: + [:i| + segInfo := self addressOf: (manager segmentManager segments at: i). + self deny: (self isSegmentBeingCompacted: segInfo)]. + ! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>compact (in category 'api') ----- + compact + self subclassResponsibility! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>compactSegment:freeStart: (in category 'compaction') ----- + compactSegment: segInfo freeStart: initialFreeStart + <var: 'segInfo' type: #'SpurSegmentInfo *'> + | currentEntity fillStart bytesToCopy numSlots bridge | + fillStart := initialFreeStart. + bridge := manager segmentManager bridgeFor: segInfo. + currentEntity := manager objectStartingAt: segInfo segStart. + [self oop: currentEntity isLessThan: bridge] whileTrue: + [(manager isFreeObject: currentEntity) + ifTrue: + ["To avoid confusing too much Spur (especially the leak/free checks), we mark the free chunk as a word object." + manager detachFreeObject: currentEntity. + manager set: currentEntity classIndexTo: manager wordSizeClassIndexPun formatTo: manager wordIndexableFormat] + ifFalse: + ["Copy the object in segmentToFill and replace it by a forwarder." + self assert: (manager isPinned: currentEntity) not. + numSlots := manager numSlotsOfAny: currentEntity. + bytesToCopy := manager bytesInObject: currentEntity. + self assert: (manager objectBytesForSlots: numSlots) = (manager bytesInObject: currentEntity). + manager mem: fillStart asVoidPointer cp: (manager startOfObject: currentEntity) asVoidPointer y: bytesToCopy. + self assert: (manager baseHeader: (manager objectStartingAt: fillStart)) = (manager baseHeader: currentEntity). + self assert: (manager fetchPointer: numSlots - 1 ofObject: (manager objectStartingAt: fillStart)) = (manager fetchPointer: numSlots - 1 ofObject: currentEntity). + manager forward: currentEntity to: (manager objectStartingAt: fillStart). + fillStart := fillStart + (manager objectBytesForSlots: numSlots). + self assert: (manager isForwarded: currentEntity). + self assert: fillStart < (segmentToFill segLimit - manager bridgeSize)]. + currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. + self assert: currentEntity = bridge. + ^ fillStart! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>compactSegmentsToCompact (in category 'compaction') ----- + compactSegmentsToCompact + "Forwards all objects in segments to compact and removes their freechunks" + | segInfo fillStart | + <var: 'segInfo' type: #'SpurSegmentInfo *'> + fillStart := segmentToFill segStart. + + "Removes initial free chunk in segment to fill... (Segment is entirely free)" + manager detachFreeObject: (manager objectStartingAt: fillStart). + + "Compact each segment to compact..." + 0 to: manager numSegments - 1 do: + [:i| + segInfo := self addressOf: (manager segmentManager segments at: i). + (self isSegmentBeingCompacted: segInfo) + ifTrue: [fillStart := self compactSegment: segInfo freeStart: fillStart ]]. + + "Final free chunk in segment to fill..." + manager + addFreeChunkWithBytes: segmentToFill segSize - manager bridgeSize + segmentToFill segStart - fillStart + at: fillStart. + + self postCompactionAction + ! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>computeSegmentsToCompact (in category 'compaction') ----- + computeSegmentsToCompact + "Compute segments to compact: least occupied. + Answers true if at least 1 segment is being compacted." + | canStillClaim aboutToClaim aboutToClaimSegment atLeastOneSegmentToCompact | + <var: 'aboutToClaimSegment' type: #'SpurSegmentInfo *'> + atLeastOneSegmentToCompact := false. + aboutToClaimSegment := self findNextSegmentToCompact. + "Segment to fill is one of the segment compacted last GC. + If no segment were compacted last GC, and that there is + at least one segment to compact, allocate a new one." + aboutToClaimSegment ifNil: [^false]. + segmentToFill ifNil: [self findOrAllocateSegmentToFill]. + canStillClaim := segmentToFill segSize - manager bridgeSize. + [aboutToClaimSegment ifNil: [^atLeastOneSegmentToCompact]. + aboutToClaim := aboutToClaimSegment segSize - manager bridgeSize * ((self occupationOf: aboutToClaimSegment) + 1) // 16rFFFF. "+1 to round up, this is approx" + aboutToClaim < canStillClaim ] whileTrue: + [self markSegmentAsBeingCompacted: aboutToClaimSegment. + atLeastOneSegmentToCompact := true. + canStillClaim := canStillClaim - aboutToClaim. + aboutToClaimSegment := self findNextSegmentToCompact]. + ^atLeastOneSegmentToCompact! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>findAndSetSegmentToFill (in category 'segment to fill') ----- + findAndSetSegmentToFill + | segInfo firstEntity | + <var: 'segInfo' type: #'SpurSegmentInfo *'> + 0 to: manager numSegments - 1 do: + [:i| + segInfo := self addressOf: (manager segmentManager segments at: i). + firstEntity := manager objectStartingAt: segInfo segStart. + ((manager isFreeObject: firstEntity) and: [(manager objectAfter: firstEntity limit: manager endOfMemory) = (manager segmentManager bridgeFor: segInfo)]) + ifTrue: [segmentToFill := segInfo. ^0]]. + ! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>findNextSegmentToCompact (in category 'compaction') ----- + findNextSegmentToCompact + "Answers the next segment to compact or nil if none. + The next segment to compact: + - cannot be segment 0 (Segment 0 has specific objects + (nil, true, etc.) and special size computed at start-up + that we don't want to deal with) + - cannot have a high occupation rate (> MaxOccupationForCompaction)" + | leastOccupied leastOccupiedSegment tempOccupied segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> + leastOccupied := 16rFFFF. + 1 to: manager numSegments - 1 do: + [:i| + segInfo := self addressOf: (manager segmentManager segments at: i). + ((self isSegmentBeingCompacted: segInfo) or: [segInfo containsPinned or: [manager segmentManager isEmptySegment: segInfo] ]) + ifFalse: + [(tempOccupied := self occupationOf: segInfo) <= leastOccupied + ifTrue: [ leastOccupied := tempOccupied. + leastOccupiedSegment := segInfo ]]]. + leastOccupied > MaxOccupationForCompaction ifTrue: [^nil]. + ^ leastOccupiedSegment! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>findOrAllocateSegmentToFill (in category 'segment to fill') ----- + findOrAllocateSegmentToFill + "There was no compacted segments from past GC that we can directly re-use. + We need either to find an empty segment or allocate a new one." + self findAndSetSegmentToFill. + segmentToFill ifNotNil: [^0]. + "No empty segment. We need to allocate a new one" + self allocateSegmentToFill. + "We don't know which segment it is that we've just allocated... So we look for it... This is a bit dumb." + self findAndSetSegmentToFill. + self assert: segmentToFill ~~ nil. + ! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>freeSegment: (in category 'segment access') ----- + freeSegment: segInfo + <var: 'segInfo' type: #'SpurSegmentInfo *'> + manager addFreeChunkWithBytes: segInfo segSize - manager bridgeSize at: segInfo segStart.! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>globalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- + globalSweepAndSegmentOccupationAnalysis + self internalGlobalSweepAndSegmentOccupationAnalysis. + manager checkFreeSpace: GCModeFull. + manager unmarkSurvivingObjectsForCompact.! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>internalGlobalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- + internalGlobalSweepAndSegmentOccupationAnalysis + "Iterate over old space, free unmarked objects, annotate each segment with each occupation" + | currentEntity nextBridge start segmentIndex currentUsed currentUnused | + currentEntity := manager firstObject. + nextBridge := manager segmentManager bridgeAt: 0. + segmentIndex := currentUnused := currentUsed := 0. + [self oop: currentEntity isLessThan: manager endOfMemory] whileTrue: + [currentEntity = nextBridge + ifTrue: + ["End of segment, set occupation" + self + setOccupationAtIndex: segmentIndex + used: currentUsed + unused: currentUnused. + currentUnused := currentUsed := 0. + segmentIndex := segmentIndex + 1. + self unmark: currentEntity. + nextBridge := manager segmentManager bridgeAt: segmentIndex] + ifFalse: + ["In-segment, sweep and compute occupation" + (self canUseAsFreeSpace: currentEntity) + ifTrue: + ["bulkFreeChunkFrom: may change a 1 word header + object to a double word header object" + start := manager startOfObject: currentEntity. + self bulkFreeChunkFrom: currentEntity. + currentEntity := manager objectStartingAt: start. + currentUnused := currentUnused + (manager numSlotsOfAny: currentEntity)] + ifFalse: + [self unmark: currentEntity. + currentUsed := currentUsed + (manager numSlotsOfAny: currentEntity)]]. + currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. + "set last segment (last bridge = endOfMemory)" + self + setOccupationAtIndex: segmentIndex + used: currentUsed + unused: currentUnused.! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>isSegmentBeingCompacted: (in category 'segment access') ----- + isSegmentBeingCompacted: segInfo + <var: 'segInfo' type: #'SpurSegmentInfo *'> + "Swizzle is abused bit 16 isBeingCompacted bits 0-15 occupation" + ^ segInfo swizzle anyMask: 1 << 16! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>markSegmentAsBeingCompacted: (in category 'segment access') ----- + markSegmentAsBeingCompacted: segInfo + <var: 'segInfo' type: #'SpurSegmentInfo *'> + "Swizzle is abused bit 16 isBeingCompacted bits 0-15 occupation" + segInfo swizzle: (segInfo swizzle bitOr: 1 << 16)! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>occupationOf: (in category 'segment access') ----- + occupationOf: segInfo + <var: 'segInfo' type: #'SpurSegmentInfo *'> + "Swizzle is abused bit 16 isBeingCompacted bits 0-15 occupation" + ^segInfo swizzle bitAnd: 16rFFFF! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>postCompactionAction (in category 'compaction') ----- + postCompactionAction + self postForwardingAction! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>postForwardingAction (in category 'compaction') ----- + postForwardingAction + | allFlags | + "For now we don't optimize and just follow everything everywhere on stack and in caches, let's see in the profiler if we need to optimize with those cases. My guess is that this is < 100 microSecond" + manager followSpecialObjectsOop. + allFlags := BecamePointerObjectFlag + BecameActiveClassFlag bitOr: BecameCompiledMethodFlag. + manager coInterpreter postBecomeAction: allFlags. + manager postBecomeScanClassTable: allFlags.! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>postSwizzleAction (in category 'api') ----- + postSwizzleAction + "Since the compact abuses the swizzle field of segment, it needs to be rest after start-up." + | segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> + 0 to: manager numSegments - 1 do: + [:i| + segInfo := self addressOf: (manager segmentManager segments at: i). + segInfo swizzle: 0 ]! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>selectiveCompaction (in category 'compaction') ----- + selectiveCompaction + "Figures out which segments to compact and compact them into segmentToFill" + | atLeastOneSegmentToCompact | + self assertNoSegmentBeingCompacted. + atLeastOneSegmentToCompact := self computeSegmentsToCompact. + "If no compaction we don't pay forwarding cost (stack scan, cache scan, etc.) + and we don't allocate segmentToFill if none available." + atLeastOneSegmentToCompact + ifTrue: + [self assert: segmentToFill ~~ nil. + self compactSegmentsToCompact]. + manager checkFreeSpace: GCModeFull.! Item was added: + ----- Method: SpurAnalysingSweeperCompactor>>setOccupationAtIndex:used:unused: (in category 'segment access') ----- + setOccupationAtIndex: segmentIndex used: used unused: unused + "WARNING: Resets the isCompacted bit" + "Swizzle is abused bit 16 isBeingCompacted bits 0-15 occupation + Setting occupation resets the claim bit" + | occupation segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> + segInfo := self addressOf: (manager segmentManager segments at: segmentIndex). + occupation := used * 16rFFFF // (used + unused). + segInfo swizzle: occupation! Item was changed: + SpurAnalysingSweeperCompactor subclass: #SpurSelectiveCompactor + instanceVariableNames: '' + classVariableNames: '' - SpurAnalysingSweeper subclass: #SpurSelectiveCompactor - instanceVariableNames: 'segmentToFill' - classVariableNames: 'MaxOccupationForCompaction' poolDictionaries: '' category: 'VMMaker-SpurMemoryManager'! + !SpurSelectiveCompactor commentStamp: 'cb 4/27/2018 13:14' prior: 0! - !SpurSelectiveCompactor commentStamp: 'cb 4/27/2018 09:50' prior: 0! SpurSelectiveCompactor compacts memory by selecting the memory segments with the most free space and compacting only those, to limit fragmentation while being really quick to perform. The algorithm is fast mostly because it does not update pointers: they are updated lazily during the next marking phase, so there is no need to read the fields of objects in other memory segments that the one compacted. The algorithm works as follow. First, a global sweep pass iterates over the memory linearly, changing unmarked objects to free chunks and concatenating free chunks. During the global sweep phase, the segments of the heap are analysed to determine the percentage of occupation. Second, the least occupied segments are compacted by copying the remaining live objects into an entirely free segment, called regionToFill (we detail later in the paragraph where regionToFill comes from), changing their values to forwarding objects and marking the free chunks as unavailable (removed from free list and marked as data objects). Third, the next marking phase removes all forwarders. Fourth, at the beginning of the next compaction phase the compacted segments from the previous GC can be entirely marked as free space (No need to check anything inside, there were only forwarders and trash data). One of the compacted segment is then selected as the segmentToFill, others are just marked as free chunks. The compaction is effectively partial, compacting only the most critical segments of the heap to limit fragmentation. Compaction time is crazy low, since a low number of objects are moved and pointer updated is lazily done during the next marking phase, while still preventing memory fragmentation. Now this works well when biasForGC is true, but when performing a snapshot, the compactor is just total crap (we need to figure out a solution). segmentToFill <SegInfo> the segment that will be filled through the copying algorithm ------------------------ Segment abuse: The swizzle field of segInfo is abused by using the low 8 bits for occupation and the 9th bit as isBeingCompacted bit. + TODO: check it seems when memory is decreasing rapidly many empty segment are kept (14 and not 4 - Bug in totalFreeSpace?) ! Item was removed: - ----- Method: SpurSelectiveCompactor class>>declareCVarsIn: (in category 'translation') ----- - declareCVarsIn: aCCodeGenerator - aCCodeGenerator var: 'segmentToFill' type: #'SpurSegmentInfo *'! Item was removed: - ----- Method: SpurSelectiveCompactor class>>initialize (in category 'initialization') ----- - initialize - super initialize. - "If the segment is occupied by more than MaxOccupationForCompaction, - it's not worth compacting it, whatever the rest of the system looks like. - MaxOccupationForCompaction is included in [0;255]." - MaxOccupationForCompaction := 150. "Basically if segment is occupied by more than 60%, not worth compacting"! Item was removed: - ----- Method: SpurSelectiveCompactor>>allocateSegmentToFill (in category 'freeing') ----- - allocateSegmentToFill - | res | - res := manager growOldSpaceByAtLeast: manager growHeadroom. - res ifNil: [self error: 'not enough memory for selective compaction'].! Item was removed: - ----- Method: SpurSelectiveCompactor>>assertNoSegmentBeingCompacted (in category 'compaction') ----- - assertNoSegmentBeingCompacted - "Assertion only - no segment is being claimed at this point" - | segInfo | - <var: 'segInfo' type: #'SpurSegmentInfo *'> - 0 to: manager numSegments - 1 do: - [:i| - segInfo := self addressOf: (manager segmentManager segments at: i). - self deny: (self isSegmentBeingCompacted: segInfo)]. - ! Item was removed: - ----- Method: SpurSelectiveCompactor>>compactSegment:freeStart: (in category 'compaction') ----- - compactSegment: segInfo freeStart: initialFreeStart - <var: 'segInfo' type: #'SpurSegmentInfo *'> - | currentEntity fillStart bytesToCopy numSlots bridge | - fillStart := initialFreeStart. - bridge := manager segmentManager bridgeFor: segInfo. - currentEntity := manager objectStartingAt: segInfo segStart. - [self oop: currentEntity isLessThan: bridge] whileTrue: - [(manager isFreeObject: currentEntity) - ifTrue: - ["To avoid confusing too much Spur (especially the leak/free checks), we mark the free chunk as a word object." - manager detachFreeObject: currentEntity. - manager set: currentEntity classIndexTo: manager wordSizeClassIndexPun formatTo: manager wordIndexableFormat] - ifFalse: - ["Copy the object in segmentToFill and replace it by a forwarder." - self assert: (manager isPinned: currentEntity) not. - numSlots := manager numSlotsOfAny: currentEntity. - bytesToCopy := manager bytesInObject: currentEntity. - self assert: (manager objectBytesForSlots: numSlots) = (manager bytesInObject: currentEntity). - manager mem: fillStart asVoidPointer cp: (manager startOfObject: currentEntity) asVoidPointer y: bytesToCopy. - self assert: (manager baseHeader: (manager objectStartingAt: fillStart)) = (manager baseHeader: currentEntity). - self assert: (manager fetchPointer: numSlots - 1 ofObject: (manager objectStartingAt: fillStart)) = (manager fetchPointer: numSlots - 1 ofObject: currentEntity). - manager forward: currentEntity to: (manager objectStartingAt: fillStart). - fillStart := fillStart + (manager objectBytesForSlots: numSlots). - self assert: (manager isForwarded: currentEntity). - self assert: fillStart < (segmentToFill segLimit - manager bridgeSize)]. - currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. - self assert: currentEntity = bridge. - ^ fillStart! Item was removed: - ----- Method: SpurSelectiveCompactor>>compactSegmentsToCompact (in category 'compaction') ----- - compactSegmentsToCompact - "Forwards all objects in segments to compact and removes their freechunks" - | segInfo fillStart | - <var: 'segInfo' type: #'SpurSegmentInfo *'> - fillStart := segmentToFill segStart. - - "Removes initial free chunk in segment to fill... (Segment is entirely free)" - manager detachFreeObject: (manager objectStartingAt: fillStart). - - "Compact each segment to compact..." - 0 to: manager numSegments - 1 do: - [:i| - segInfo := self addressOf: (manager segmentManager segments at: i). - (self isSegmentBeingCompacted: segInfo) - ifTrue: [fillStart := self compactSegment: segInfo freeStart: fillStart ]]. - - "Final free chunk in segment to fill..." - manager - addFreeChunkWithBytes: segmentToFill segSize - manager bridgeSize + segmentToFill segStart - fillStart - at: fillStart. - - "Follow stack zone and caches..." - self postForwardingAction - ! Item was removed: - ----- Method: SpurSelectiveCompactor>>computeSegmentsToCompact (in category 'compaction') ----- - computeSegmentsToCompact - "Compute segments to claim: least occupied. - Answers true if at least 1 segment is being compacted." - | canStillClaim aboutToClaim aboutToClaimSegment atLeastOneSegmentToCompact | - <var: 'aboutToClaimSegment' type: #'SpurSegmentInfo *'> - atLeastOneSegmentToCompact := false. - aboutToClaimSegment := self findNextSegmentToCompact. - "Segment to fill is one of the segment compacted last GC. - If no segment were compacted last GC, and that there is - at least one segment to compact, allocate a new one." - aboutToClaimSegment ifNil: [^false]. - segmentToFill ifNil: [self findOrAllocateSegmentToFill]. - canStillClaim := segmentToFill segSize - manager bridgeSize. - [aboutToClaimSegment ifNil: [^atLeastOneSegmentToCompact]. - aboutToClaim := aboutToClaimSegment segSize - manager bridgeSize * ((self occupationOf: aboutToClaimSegment) + 1) // 255. "+1 to round up, this is approx" - aboutToClaim < canStillClaim ] whileTrue: - [self markSegmentAsBeingCompacted: aboutToClaimSegment. - atLeastOneSegmentToCompact := true. - canStillClaim := canStillClaim - aboutToClaim. - aboutToClaimSegment := self findNextSegmentToCompact]. - ^atLeastOneSegmentToCompact! Item was removed: - ----- Method: SpurSelectiveCompactor>>findAndSetSegmentToFill (in category 'freeing') ----- - findAndSetSegmentToFill - | segInfo firstEntity | - <var: 'segInfo' type: #'SpurSegmentInfo *'> - 0 to: manager numSegments - 1 do: - [:i| - segInfo := self addressOf: (manager segmentManager segments at: i). - firstEntity := manager objectStartingAt: segInfo segStart. - ((manager isFreeObject: firstEntity) and: [(manager objectAfter: firstEntity limit: manager endOfMemory) = (manager segmentManager bridgeFor: segInfo)]) - ifTrue: [segmentToFill := segInfo. ^0]]. - ! Item was removed: - ----- Method: SpurSelectiveCompactor>>findNextSegmentToCompact (in category 'compaction') ----- - findNextSegmentToCompact - "Answers the next segment to compact or nil if none. - The next segment to compact: - - cannot be segment 0 (Segment 0 has specific objects - (nil, true, etc.) and special size computed at start-up - that we don't want to deal with) - - cannot be be a segment already being compacted. - - cannot contain pinned object (since we're in a copying GC) - - cannot be entirely empty (no need to block that empty segment until next marking phase) - - cannot have a high occupation rate (> MaxOccupationForCompaction)" - | leastOccupied leastOccupiedSegment tempOccupied segInfo | - <var: 'segInfo' type: #'SpurSegmentInfo *'> - leastOccupied := 255. - 1 to: manager numSegments - 1 do: - [:i| - segInfo := self addressOf: (manager segmentManager segments at: i). - ((self isSegmentBeingCompacted: segInfo) or: [segInfo containsPinned or: [manager segmentManager isEmptySegment: segInfo] ]) - ifFalse: - [(tempOccupied := self occupationOf: segInfo) <= leastOccupied - ifTrue: [ leastOccupied := tempOccupied. - leastOccupiedSegment := segInfo ]]]. - leastOccupied > MaxOccupationForCompaction ifTrue: [^nil]. - ^ leastOccupiedSegment! Item was removed: - ----- Method: SpurSelectiveCompactor>>findOrAllocateSegmentToFill (in category 'freeing') ----- - findOrAllocateSegmentToFill - "There was no compacted segments from past GC that we can directly re-use. - We need either to find an empty segment or allocate a new one." - self findAndSetSegmentToFill. - segmentToFill ifNotNil: [^0]. - "No empty segment. We need to allocate a new one" - self allocateSegmentToFill. - "We don't know which segment it is that we've just allocated... So we look for it... This is a bit dumb." - self findAndSetSegmentToFill. - self assert: segmentToFill ~~ nil. - ! Item was removed: - ----- Method: SpurSelectiveCompactor>>freeSegment: (in category 'freeing') ----- - freeSegment: segInfo - <var: 'segInfo' type: #'SpurSegmentInfo *'> - manager addFreeChunkWithBytes: segInfo segSize - manager bridgeSize at: segInfo segStart.! Item was removed: - ----- Method: SpurSelectiveCompactor>>isSegmentBeingCompacted: (in category 'segment access') ----- - isSegmentBeingCompacted: segInfo - <var: 'segInfo' type: #'SpurSegmentInfo *'> - "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation" - ^ segInfo swizzle anyMask: 1 << 8! Item was removed: - ----- Method: SpurSelectiveCompactor>>markSegmentAsBeingCompacted: (in category 'segment access') ----- - markSegmentAsBeingCompacted: segInfo - <var: 'segInfo' type: #'SpurSegmentInfo *'> - "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation" - segInfo swizzle: (segInfo swizzle bitOr: 1 << 8)! Item was removed: - ----- Method: SpurSelectiveCompactor>>occupationOf: (in category 'segment access') ----- - occupationOf: segInfo - <var: 'segInfo' type: #'SpurSegmentInfo *'> - "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation" - ^segInfo swizzle bitAnd: 16rFF! Item was removed: - ----- Method: SpurSelectiveCompactor>>postForwardingAction (in category 'compaction') ----- - postForwardingAction - | allFlags | - "For now we don't optimize and just follow everything everywhere on stack and in caches, let's see in the profiler if we need to optimize with those cases. My guess is that this is < 100 microSecond" - manager followSpecialObjectsOop. - allFlags := BecamePointerObjectFlag + BecameActiveClassFlag bitOr: BecameCompiledMethodFlag. - manager coInterpreter postBecomeAction: allFlags. - manager postBecomeScanClassTable: allFlags.! Item was removed: - ----- Method: SpurSelectiveCompactor>>postSwizzleAction (in category 'api') ----- - postSwizzleAction - "Since the compact abuses the swizzle field of segment, it needs to be rest after start-up." - | segInfo | - <var: 'segInfo' type: #'SpurSegmentInfo *'> - 0 to: manager numSegments - 1 do: - [:i| - segInfo := self addressOf: (manager segmentManager segments at: i). - segInfo swizzle: 0 ]! Item was removed: - ----- Method: SpurSelectiveCompactor>>selectiveCompaction (in category 'compaction') ----- - selectiveCompaction - "Figures out which segments to compact and compact them into segmentToFill" - | atLeastOneSegmentToCompact | - self assertNoSegmentBeingCompacted. - atLeastOneSegmentToCompact := self computeSegmentsToCompact. - "If no compaction we don't pay forwarding cost (stack scan, cache scan, etc.) - and we don't allocate segmentToFill if none available." - atLeastOneSegmentToCompact - ifTrue: - [self assert: segmentToFill ~~ nil. - self compactSegmentsToCompact]. - manager checkFreeSpace: GCModeFull.! Item was removed: - ----- Method: SpurSelectiveCompactor>>setOccupationAtIndex:used:unused: (in category 'segment access') ----- - setOccupationAtIndex: segmentIndex used: used unused: unused - "WARNING: Resets the isCompacted bit" - "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation - Setting occupation resets the claim bit" - | occupation segInfo | - <var: 'segInfo' type: #'SpurSegmentInfo *'> - segInfo := self addressOf: (manager segmentManager segments at: segmentIndex). - occupation := used * 255 // (used + unused). - segInfo swizzle: occupation! Item was changed: + SpurAnalysingSweeperCompactor subclass: #SpurTrackingCompactor - SpurAnalysingSweeper subclass: #SpurTrackingCompactor instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'VMMaker-SpurMemoryManager'! + !SpurTrackingCompactor commentStamp: 'cb 4/27/2018 13:12' prior: 0! - !SpurTrackingCompactor commentStamp: 'cb 4/27/2018 09:58' prior: 0! SpurTrackingCompactor is a derived simplified implementation of Garbage First (G1) algorithm (Java 9 default GC). SpurTrackingCompactor compacts memory by selecting the memory segments with the most free space and compacting only those, to limit fragmentation while being really quick to perform. To update efficiently the references to moved objects, SpurTrackingCompactor uses a per segment remembered table in the form with a card marking scheme, hence when compacting segments, instead of scanning all the heap for pointer updates, it scans only the moved objects and the objects remembered for the segment. Since segments compacted are almost free segments, the remembered table is small upon compaction. This algorithm requires extra GC write barriers and higher aligment in segments for efficient write barrier (bits in the pointer are used to know to which segment an object belongs). + TODO: + followTrackedReference and reference tracking + Implement global card mark with 1 byte per 1024 byte (1 bit has object, 7 bits starting index of object in the range covered) + Implement per segment card mark with 1 bit per 1024 bytes. + Implement cards for 2Gb (2Mb global card and 250kb), allow only segments within the 2 Gb range to avoid issues + - Sweep phase can update global card to avoid changing whole VM + - Write barrier: based on some bits in pointer, figure out which card to dirty and in which segment's card mark + - followTrackReference is commented with idea: find what to follow and heap and follow all. ! Item was added: + ----- Method: SpurTrackingCompactor>>compact (in category 'api') ----- + compact + <inline: #never> + self globalSweepAndSegmentOccupationAnalysis. + self selectiveCompaction. + ! Item was added: + ----- Method: SpurTrackingCompactor>>followAllObjectsInSegmentToFill (in category 'compaction') ----- + followAllObjectsInSegmentToFill + | currentEntity bridge | + bridge := manager segmentManager bridgeFor: segmentToFill. + currentEntity := manager objectStartingAt: segmentToFill segStart. + [self oop: currentEntity isLessThan: bridge] whileTrue: + [((manager isEnumerableObject: currentEntity) and: [manager isPointersNonImm: currentEntity]) + ifTrue: + [0 to: (manager numSlotsOfAny: currentEntity) do: + [:i | self followField: i ofObject: currentEntity]]]. + self assert: currentEntity = bridge.! Item was added: + ----- Method: SpurTrackingCompactor>>followTrackedReferences (in category 'compaction') ----- + followTrackedReferences + "Each segment has a remembered set in the form of a card table. We need to create a local card table, bitAnd it with all compacted segments, then follow all dirty areas" + 1halt. #TODO. ! Item was added: + ----- Method: SpurTrackingCompactor>>freeCompactedSegments (in category 'compaction') ----- + freeCompactedSegments + 0 to: manager numSegments - 1 do: + [:i| self freeSegment: (self addressOf: (manager segmentManager segments at: i))]. + ! Item was added: + ----- Method: SpurTrackingCompactor>>postCompactionAction (in category 'compaction') ----- + postCompactionAction + self followAllObjectsInSegmentToFill. "deal with inner segment references" + self followTrackedReferences. "partial heap scan" + self freeCompactedSegments. + self postForwardingAction.!

1 0

VM Maker: VMMaker.oscog-cb.2378.mcz
by commits＠source.squeak.org 27 Apr '18

27 Apr '18

ClementBera uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2378.mcz ==================== Summary ==================== Name: VMMaker.oscog-cb.2378 Author: cb Time: 27 April 2018, 10:07:17.97412 am UUID: 4dc47e41-2815-44b2-9c16-23caeb9df018 Ancestors: VMMaker.oscog-cb.2377 Improved compactor comments. Introduced SpurTrackingCompactor (Work in progress) which emulates Garbage First compaction behavior. Added SpurAnalysingSweeper as common superclass of SelectiveCompactor and TrackingCompactor. SpurAnalysingSweeper includes APIs to sweep the heap and compute occupation of each segment at the same time, which is used by Selective and Tracking compactors (they compact only the least used segments). =============== Diff against VMMaker.oscog-cb.2377 =============== Item was added: + SpurSweeper subclass: #SpurAnalysingSweeper + instanceVariableNames: '' + classVariableNames: '' + poolDictionaries: '' + category: 'VMMaker-SpurMemoryManager'! + + !SpurAnalysingSweeper commentStamp: 'cb 4/27/2018 09:45' prior: 0! + Abstract class, in addition to SpurSweeper, while sweeping the heap I annotate segments with occupation rate. This can be used by a compacting algorithm to compact only segments which are not used a lot.! Item was added: + ----- Method: SpurAnalysingSweeper>>globalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- + globalSweepAndSegmentOccupationAnalysis + self internalGlobalSweepAndSegmentOccupationAnalysis. + manager checkFreeSpace: GCModeFull. + manager unmarkSurvivingObjectsForCompact.! Item was added: + ----- Method: SpurAnalysingSweeper>>internalGlobalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- + internalGlobalSweepAndSegmentOccupationAnalysis + "Iterate over old space, free unmarked objects, annotate each segment with each occupation" + | currentEntity nextBridge start segmentIndex currentUsed currentUnused | + currentEntity := manager firstObject. + nextBridge := manager segmentManager bridgeAt: 0. + segmentIndex := currentUnused := currentUsed := 0. + [self oop: currentEntity isLessThan: manager endOfMemory] whileTrue: + [currentEntity = nextBridge + ifTrue: + ["End of segment, set occupation" + self + setOccupationAtIndex: segmentIndex + used: currentUsed + unused: currentUnused. + currentUnused := currentUsed := 0. + segmentIndex := segmentIndex + 1. + self unmark: currentEntity. + nextBridge := manager segmentManager bridgeAt: segmentIndex] + ifFalse: + ["In-segment, sweep and compute occupation" + (self canUseAsFreeSpace: currentEntity) + ifTrue: + ["bulkFreeChunkFrom: may change a 1 word header + object to a double word header object" + start := manager startOfObject: currentEntity. + self bulkFreeChunkFrom: currentEntity. + currentEntity := manager objectStartingAt: start. + currentUnused := currentUnused + (manager numSlotsOfAny: currentEntity)] + ifFalse: + [self unmark: currentEntity. + currentUsed := currentUsed + (manager numSlotsOfAny: currentEntity)]]. + currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. + "set last segment (last bridge = endOfMemory)" + self + setOccupationAtIndex: segmentIndex + used: currentUsed + unused: currentUnused.! Item was added: + ----- Method: SpurAnalysingSweeper>>occupationOf: (in category 'segment access') ----- + occupationOf: segInfo + self subclassResponsibility! Item was added: + ----- Method: SpurAnalysingSweeper>>setOccupationAtIndex:used:unused: (in category 'segment access') ----- + setOccupationAtIndex: segmentIndex used: used unused: unused + self subclassResponsibility! Item was changed: CogClass subclass: #SpurCompactor instanceVariableNames: 'manager scavenger coInterpreter' classVariableNames: '' poolDictionaries: 'SpurMemoryManagementConstants VMBasicConstants VMSpurObjectRepresentationConstants' category: 'VMMaker-SpurMemoryManager'! + !SpurCompactor commentStamp: 'cb 4/27/2018 09:38' prior: 0! + Abstract common superclass of all compactors to define apis and simulation variables. - !SpurCompactor commentStamp: 'cb 4/26/2018 13:53' prior: 0! - Common superclass of all compactors to define apis and simulation variables. - IMPORTANT: This defines APIs only, subclassing is prohibited on auxiliary classes (Slang compiler won't find inherited methods). - The full GC in Spur is split in two, the marking phase and the compactor phase. The subclasses of SpurCompactor are implementations of the second phase, so they are called once the marking phase is finished. SpurCompactor is reponsible for: - freeing unmarked objects - compacting the live old space objects (though each subclass define what it does, some spurCompactor may not compact) - unmarking all objects remaining live - updating oops directly referred by the VM when they are moved (remapObj:/shouldRemapObj: thingy) The main apis are the following: - biasForGC/biasForSnapshot: tells the compactor if the GC is performed for snapshots or not, in general we want to compact more aggressively for snapshots to avoid saving large files with many unused space. - compact: main API, should free the unmarked object, unmark the objects remaining live and potentially compact the heap - remapObj:/shouldRemapObj: => Not really sure what this does, it seems it has to do with updating oops directly referred by the VM when they are moved. + - postSwizzleAction: if you want to do something at start-up after swizzle phase (typically useful if your compaction algo uses segInfos) - Instance Variables coInterpreter: <StackInterpreter> + scavenger: <SpurGenerationScavenger> - compactedCopySpace: <SpurNewSpaceSpace> manager: <SpurMemoryManager>! Item was changed: + SpurAnalysingSweeper subclass: #SpurSelectiveCompactor - SpurSweeper subclass: #SpurSelectiveCompactor instanceVariableNames: 'segmentToFill' classVariableNames: 'MaxOccupationForCompaction' poolDictionaries: '' category: 'VMMaker-SpurMemoryManager'! + !SpurSelectiveCompactor commentStamp: 'cb 4/27/2018 09:50' prior: 0! - !SpurSelectiveCompactor commentStamp: 'cb 4/26/2018 13:59' prior: 0! SpurSelectiveCompactor compacts memory by selecting the memory segments with the most free space and compacting only those, to limit fragmentation while being really quick to perform. The algorithm is fast mostly because it does not update pointers: they are updated lazily during the next marking phase, so there is no need to read the fields of objects in other memory segments that the one compacted. The algorithm works as follow. First, a global sweep pass iterates over the memory linearly, changing unmarked objects to free chunks and concatenating free chunks. During the global sweep phase, the segments of the heap are analysed to determine the percentage of occupation. Second, the least occupied segments are compacted by copying the remaining live objects into an entirely free segment, called regionToFill (we detail later in the paragraph where regionToFill comes from), changing their values to forwarding objects and marking the free chunks as unavailable (removed from free list and marked as data objects). Third, the next marking phase removes all forwarders. Fourth, at the beginning of the next compaction phase the compacted segments from the previous GC can be entirely marked as free space (No need to check anything inside, there were only forwarders and trash data). One of the compacted segment is then selected as the segmentToFill, others are just marked as free chunks. The compaction is effectively partial, compacting only the most critical segments of the heap to limit fragmentation. Compaction time is crazy low, since a low number of objects are moved and pointer updated is lazily done during the next marking phase, while still preventing memory fragmentation. Now this works well when biasForGC is true, but when performing a snapshot, the compactor is just total crap (we need to figure out a solution). - IMPORTANT: I could not figure out to make inheritance work so I copied methods from SpurSweeper here. - segmentToFill <SegInfo> the segment that will be filled through the copying algorithm + ------------------------ + + Segment abuse: + The swizzle field of segInfo is abused by using the low 8 bits for occupation and the 9th bit as isBeingCompacted bit. + ! Item was removed: - ----- Method: SpurSelectiveCompactor>>globalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- - globalSweepAndSegmentOccupationAnalysis - self internalGlobalSweepAndSegmentOccupationAnalysis. - manager checkFreeSpace: GCModeFull. - manager unmarkSurvivingObjectsForCompact.! Item was removed: - ----- Method: SpurSelectiveCompactor>>internalGlobalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- - internalGlobalSweepAndSegmentOccupationAnalysis - "Iterate over old space, free unmarked objects, annotate each segment with each occupation" - | currentEntity nextBridge start segmentIndex currentUsed currentUnused | - currentEntity := manager firstObject. - nextBridge := manager segmentManager bridgeAt: 0. - segmentIndex := currentUnused := currentUsed := 0. - [self oop: currentEntity isLessThan: manager endOfMemory] whileTrue: - [currentEntity = nextBridge - ifTrue: - ["End of segment, set occupation" - self - setOccupationAtIndex: segmentIndex - used: currentUsed - unused: currentUnused. - currentUnused := currentUsed := 0. - segmentIndex := segmentIndex + 1. - self unmark: currentEntity. - nextBridge := manager segmentManager bridgeAt: segmentIndex] - ifFalse: - ["In-segment, sweep and compute occupation" - (self canUseAsFreeSpace: currentEntity) - ifTrue: - ["bulkFreeChunkFrom: may change a 1 word header - object to a double word header object" - start := manager startOfObject: currentEntity. - self bulkFreeChunkFrom: currentEntity. - currentEntity := manager objectStartingAt: start. - currentUnused := currentUnused + (manager numSlotsOfAny: currentEntity)] - ifFalse: - [self unmark: currentEntity. - currentUsed := currentUsed + (manager numSlotsOfAny: currentEntity)]]. - currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. - "set last segment (last bridge = endOfMemory)" - self - setOccupationAtIndex: segmentIndex - used: currentUsed - unused: currentUnused.! Item was changed: ----- Method: SpurSelectiveCompactor>>isSegmentBeingCompacted: (in category 'segment access') ----- isSegmentBeingCompacted: segInfo <var: 'segInfo' type: #'SpurSegmentInfo *'> + "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation" - "Swizzle is abused bit 8 isClaimed bits 0-7 occupation" ^ segInfo swizzle anyMask: 1 << 8! Item was changed: ----- Method: SpurSelectiveCompactor>>markSegmentAsBeingCompacted: (in category 'segment access') ----- markSegmentAsBeingCompacted: segInfo <var: 'segInfo' type: #'SpurSegmentInfo *'> + "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation" - "Swizzle is abused bit 8 isClaimed bits 0-7 occupation" segInfo swizzle: (segInfo swizzle bitOr: 1 << 8)! Item was changed: ----- Method: SpurSelectiveCompactor>>occupationOf: (in category 'segment access') ----- occupationOf: segInfo <var: 'segInfo' type: #'SpurSegmentInfo *'> + "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation" - "Swizzle is abused bit 8 isClaimed bits 0-7 occupation" ^segInfo swizzle bitAnd: 16rFF! Item was removed: - ----- Method: SpurSelectiveCompactor>>setOccupation:used:unused: (in category 'segment access') ----- - setOccupation: segInfo used: used unused: unused - <var: 'segInfo' type: #'SpurSegmentInfo *'> - "Swizzle is abused bit 8 isClaimed bits 0-7 occupation - Setting occupation resets the claim bit" - | occupation | - occupation := used * 255 // (used + unused). - segInfo swizzle: occupation! Item was changed: ----- Method: SpurSelectiveCompactor>>setOccupationAtIndex:used:unused: (in category 'segment access') ----- setOccupationAtIndex: segmentIndex used: used unused: unused + "WARNING: Resets the isCompacted bit" + "Swizzle is abused bit 8 isBeingCompacted bits 0-7 occupation - "Swizzle is abused bit 8 isClaimed bits 0-7 occupation Setting occupation resets the claim bit" | occupation segInfo | <var: 'segInfo' type: #'SpurSegmentInfo *'> segInfo := self addressOf: (manager segmentManager segments at: segmentIndex). occupation := used * 255 // (used + unused). segInfo swizzle: occupation! Item was changed: SpurCompactor subclass: #SpurSweeper instanceVariableNames: 'biasForGC' classVariableNames: '' poolDictionaries: '' category: 'VMMaker-SpurMemoryManager'! + !SpurSweeper commentStamp: 'cb 4/27/2018 09:43' prior: 0! - !SpurSweeper commentStamp: 'cb 10/2/2017 11:34' prior: 0! SpurSweeper is a sweep-only algorithm, setting the compactor to SpurSweeper effectively changes the fullGC to a mark-sweep non-moving algorithm. + SpurSweeper is a reference implementation if one wants to evaluate GC performance and compare it to a Mark-Sweep. It's also the only non-moving GC available right now which can be convenient for some experiments. One of the main reason why it was implemented is because advanced compaction algorithm includes a sweep phase (See SelectiveCompactor for example) and SpurSweeper allows to debug the sweep phase separatedly. + ! - SpurSweeper has two main purposes: - 1. SpurSelectiveCompactor includes a sweep algorithm, inherited from SpurSweeper, and SpurSweeper allows to debug it separatedly. - 2. SpurSweeper is a non-moving GC which can be convenient in some cases (For example when accessing objects from C in a concurrent thread called with ThreadedFFI, the C code can access the objects during fullGC since there won't be conflict with object fields being updated while being read). - - For snapshots a non-compacting algortihm does not make sense, hence a more aggressive compactor is called instead (see #compact).! Item was added: + SpurAnalysingSweeper subclass: #SpurTrackingCompactor + instanceVariableNames: '' + classVariableNames: '' + poolDictionaries: '' + category: 'VMMaker-SpurMemoryManager'! + + !SpurTrackingCompactor commentStamp: 'cb 4/27/2018 09:58' prior: 0! + SpurTrackingCompactor is a derived simplified implementation of Garbage First (G1) algorithm (Java 9 default GC). + + SpurTrackingCompactor compacts memory by selecting the memory segments with the most free space and compacting only those, to limit fragmentation while being really quick to perform. To update efficiently the references to moved objects, SpurTrackingCompactor uses a per segment remembered table in the form with a card marking scheme, hence when compacting segments, instead of scanning all the heap for pointer updates, it scans only the moved objects and the objects remembered for the segment. Since segments compacted are almost free segments, the remembered table is small upon compaction. + + This algorithm requires extra GC write barriers and higher aligment in segments for efficient write barrier (bits in the pointer are used to know to which segment an object belongs). + + !

1 0

VM Maker: VMMaker.oscog-cb.2377.mcz
by commits＠source.squeak.org 26 Apr '18

26 Apr '18

ClementBera uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2377.mcz ==================== Summary ==================== Name: VMMaker.oscog-cb.2377 Author: cb Time: 26 April 2018, 8:53:46.138916 pm UUID: 94bd1cb9-5044-44d7-8e84-7e4c341f8855 Ancestors: VMMaker.oscog-cb.2376 Spur ancilliary classes includes now: (self compactorClass withAllSuperclasses copyUpThrough: SpurCompactor) reverse instead of only: self compactorClass Removed a shit load of code duplication thanks to this change. =============== Diff against VMMaker.oscog-cb.2376 =============== Item was changed: ----- Method: SpurCompactor>>postSwizzleAction (in category 'api') ----- postSwizzleAction + "do nothing"! - self subclassResponsibility! Item was changed: ----- Method: SpurCompactor>>printTheBogons: (in category 'debugging') ----- printTheBogons: aBogon + <inline: true> + coInterpreter + print: 'bogon '; printHexnp: aBogon; cr! - self subclassResponsibility! Item was changed: ----- Method: SpurCompactor>>remapObj: (in category 'api') ----- remapObj: objOop + <api> + <inline: false> + ^manager vanillaRemapObj: objOop! - self subclassResponsibility! Item was changed: ----- Method: SpurCompactor>>shouldRemapObj: (in category 'api') ----- shouldRemapObj: objOop + <api> + "Answer if the obj should be scavenged (or simply followed). The method is called + shouldRemapObj: for compatibility with ObjectMemory. Defer to the compactor + to choose the actual test, there being a difference between the vanilla test and + that used with a sliding compactor where objects are not marked as forwarded." + ^manager vanillaShouldRemapObj: objOop! - self subclassResponsibility! Item was changed: ----- Method: SpurMemoryManager class>>ancilliaryClasses: (in category 'translation') ----- ancilliaryClasses: options initializationOptions ifNil: [initializationOptions := options]. + ^{ SpurGenerationScavenger. SpurScavengeLogRecord. SpurSegmentManager. SpurSegmentInfo }, + (self compactorClass withAllSuperclasses copyUpThrough: SpurCompactor) reverse, + SpurNewSpaceSpace withAllSubclasses + + ! - ^{ SpurGenerationScavenger. SpurScavengeLogRecord. SpurSegmentManager. SpurSegmentInfo. self compactorClass }, - SpurNewSpaceSpace withAllSubclasses! Item was removed: - ----- Method: SpurPigCompactor>>postSwizzleAction (in category 'compatibility') ----- - postSwizzleAction - "do nothing" - ! Item was removed: - ----- Method: SpurPigCompactor>>printTheBogons: (in category 'debug printing') ----- - printTheBogons: aBogon - <inline: true> - coInterpreter - print: 'bogon '; printHexnp: aBogon; cr! Item was removed: - ----- Method: SpurPigCompactor>>remapObj: (in category 'gc - scavenge/compact') ----- - remapObj: objOop - "Scavenge or simply follow objOop. Answer the new location of objOop. - The send should have been guarded by a send of shouldRemapOop:. - The method is called remapObj: for compatibility with ObjectMemory." - <api> - <inline: false> - ^manager vanillaRemapObj: objOop! Item was removed: - ----- Method: SpurPigCompactor>>shouldRemapObj: (in category 'gc - scavenge/compact') ----- - shouldRemapObj: objOop - <api> - "Answer if the obj should be scavenged (or simply followed). The method is called - shouldRemapObj: for compatibility with ObjectMemory. Defer to the compactor - to choose the actual test, there being a difference between the vanilla test and - that used with a sliding compactor where objects are not marked as forwarded." - ^manager vanillaShouldRemapObj: objOop! Item was removed: - ----- Method: SpurPlanningCompactor>>postSwizzleAction (in category 'compatibility') ----- - postSwizzleAction - "do nothing" - ! Item was changed: + SpurSweeper subclass: #SpurSelectiveCompactor + instanceVariableNames: 'segmentToFill' - SpurCompactor subclass: #SpurSelectiveCompactor - instanceVariableNames: 'biasForGC segmentToFill' classVariableNames: 'MaxOccupationForCompaction' poolDictionaries: '' category: 'VMMaker-SpurMemoryManager'! !SpurSelectiveCompactor commentStamp: 'cb 4/26/2018 13:59' prior: 0! SpurSelectiveCompactor compacts memory by selecting the memory segments with the most free space and compacting only those, to limit fragmentation while being really quick to perform. The algorithm is fast mostly because it does not update pointers: they are updated lazily during the next marking phase, so there is no need to read the fields of objects in other memory segments that the one compacted. The algorithm works as follow. First, a global sweep pass iterates over the memory linearly, changing unmarked objects to free chunks and concatenating free chunks. During the global sweep phase, the segments of the heap are analysed to determine the percentage of occupation. Second, the least occupied segments are compacted by copying the remaining live objects into an entirely free segment, called regionToFill (we detail later in the paragraph where regionToFill comes from), changing their values to forwarding objects and marking the free chunks as unavailable (removed from free list and marked as data objects). Third, the next marking phase removes all forwarders. Fourth, at the beginning of the next compaction phase the compacted segments from the previous GC can be entirely marked as free space (No need to check anything inside, there were only forwarders and trash data). One of the compacted segment is then selected as the segmentToFill, others are just marked as free chunks. The compaction is effectively partial, compacting only the most critical segments of the heap to limit fragmentation. Compaction time is crazy low, since a low number of objects are moved and pointer updated is lazily done during the next marking phase, while still preventing memory fragmentation. Now this works well when biasForGC is true, but when performing a snapshot, the compactor is just total crap (we need to figure out a solution). IMPORTANT: I could not figure out to make inheritance work so I copied methods from SpurSweeper here. segmentToFill <SegInfo> the segment that will be filled through the copying algorithm ! Item was removed: - ----- Method: SpurSelectiveCompactor>>biasForGC (in category 'api') ----- - biasForGC - biasForGC := true.! Item was removed: - ----- Method: SpurSelectiveCompactor>>biasForSnapshot (in category 'api') ----- - biasForSnapshot - biasForGC := false.! Item was removed: - ----- Method: SpurSelectiveCompactor>>bulkFreeChunkFrom: (in category 'sweep phase - copied from SpurSweeper') ----- - bulkFreeChunkFrom: objOop - "ObjOop is either a freeChunk or an object to free, always in old space. The old space entity before objOop is necessarily a marked object. - Attempts to free as many byte from objOop, looking ahead for multiple freechunks / objects to free in a row" - | bytes start next currentObj | - - "Avoids pathological case, not point in dealing with non-mergeable free chunks, we would remove them and re-add them to the free list." - (self isSingleFreeObject: objOop) ifTrue: [^0]. - - "We free unmarked objects and freechunks next to each others and merge them at the same time" - start := manager startOfObject: objOop. - currentObj := objOop. - bytes := 0. - [bytes := bytes + (manager bytesInObject: currentObj). - self freeEntity: currentObj. - next := manager objectStartingAt: start + bytes. - self canUseNextEntityAsFreeSpace: next] - whileTrue: [currentObj := next]. - - manager addFreeChunkWithBytes: bytes at: start. - - ^ next! Item was removed: - ----- Method: SpurSelectiveCompactor>>canUseAsFreeSpace: (in category 'sweep phase - copied from SpurSweeper') ----- - canUseAsFreeSpace: objOop - <inline: true> - ^ (manager isFreeObject: objOop) or: [(manager isMarked: objOop) not]! Item was removed: - ----- Method: SpurSelectiveCompactor>>canUseNextEntityAsFreeSpace: (in category 'sweep phase - copied from SpurSweeper') ----- - canUseNextEntityAsFreeSpace: next - <inline: true> - ^ (manager oop: next isLessThan: manager endOfMemory) and: [self canUseAsFreeSpace: next]! Item was removed: - ----- Method: SpurSelectiveCompactor>>freeEntity: (in category 'sweep phase - copied from SpurSweeper') ----- - freeEntity: entity - <inline: true> - (manager isFreeObject: entity) - ifFalse: "Freed old space objects are removed from remembered table" - [(manager isRemembered: entity) ifTrue: - [scavenger forgetObject: entity]] - ifTrue: "Merged old space free chunks are removed from free list" - [manager detachFreeObject: entity] - ! Item was removed: - ----- Method: SpurSelectiveCompactor>>isSingleFreeObject: (in category 'sweep phase - copied from SpurSweeper') ----- - isSingleFreeObject: objOop - <inline: true> - | next | - ^ (manager isFreeObject: objOop) and: - [next := manager objectAfter: objOop limit: manager endOfMemory. - (manager oop: next isGreaterThanOrEqualTo: manager endOfMemory) or: [manager isMarked: next]]! Item was removed: - ----- Method: SpurSelectiveCompactor>>printTheBogons: (in category 'debugging') ----- - printTheBogons: aBogon - <inline: true> - coInterpreter - print: 'bogon '; printHexnp: aBogon; cr! Item was removed: - ----- Method: SpurSelectiveCompactor>>remapObj: (in category 'api') ----- - remapObj: objOop - <api> - <inline: false> - ^manager vanillaRemapObj: objOop! Item was removed: - ----- Method: SpurSelectiveCompactor>>shouldRemapObj: (in category 'api') ----- - shouldRemapObj: objOop - <api> - ^manager vanillaShouldRemapObj: objOop! Item was removed: - ----- Method: SpurSelectiveCompactor>>unmark: (in category 'sweep phase - copied from SpurSweeper') ----- - unmark: objOop - self assert: ((manager isMarked: objOop) and: [(manager isFreeObject: objOop) not]). - (manager isSegmentBridge: objOop) ifFalse: [manager setIsMarkedOf: objOop to: false]. - (manager isPinned: objOop) ifTrue: [manager segmentManager notePinned: objOop]! Item was removed: - ----- Method: SpurSweeper>>postSwizzleAction (in category 'api') ----- - postSwizzleAction - "do nothing" - ! Item was removed: - ----- Method: SpurSweeper>>printTheBogons: (in category 'debugging') ----- - printTheBogons: aBogon - <inline: true> - coInterpreter - print: 'bogon '; printHexnp: aBogon; cr! Item was removed: - ----- Method: SpurSweeper>>remapObj: (in category 'api') ----- - remapObj: objOop - <api> - <inline: false> - ^manager vanillaRemapObj: objOop! Item was removed: - ----- Method: SpurSweeper>>shouldRemapObj: (in category 'api') ----- - shouldRemapObj: objOop - <api> - ^manager vanillaShouldRemapObj: objOop!

1 0

VM Maker: VMMaker.oscog-cb.2373.mcz
by commits＠source.squeak.org 26 Apr '18

26 Apr '18

ClementBera uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2373.mcz ==================== Summary ==================== Name: VMMaker.oscog-cb.2373 Author: cb Time: 26 April 2018, 11:11:47.778949 am UUID: 9b389323-2181-4503-a361-d66ad87fa2de Ancestors: VMMaker.oscog-cb.2372 Remove the APIs I added to iterate over free chunks (there was an existing API) Added assertValidFreeObject: to avoid stepping all the time in isValidFreeObject to know what's wrong. Obviously this new method cannot be used in the C code or we will have code in assertion-free VM, leading to the following pattern: self "Sorry stepping over isValidFreeObject all the time was killing me" cCode: [self assert: (self isValidFreeObject: child)] inSmalltalk: [self assertValidFreeObject: child]. Since I now use forwarders in fullGC, adapted heap space integrity check. I was a little bit too aggressive in assertion in detachFreeObject: in last commit, reverted that. And SpurSelectiveCompactor is now working as an alternative compactor to Planning, Pig compactors and Sweeper! So exciting. Still needs some tuning for production use (Mostly snapshots consume high memory). SpurSelectiveCompactor compaction time is crazy low (almost as fast as a Sweep algorithm). I may write yet another compactor since I need to compare SelectiveCompactor with Garbage First multi-remembered table approach for research purpose... =============== Diff against VMMaker.oscog-cb.2372 =============== Item was removed: - ----- Method: SpurMemoryManager>>allOldSpaceFreeChunksDo: (in category 'object enumeration') ----- - allOldSpaceFreeChunksDo: aBlock - <inline: true> - <doNotGenerate> "Could be generated, but used for debug only" - self allOldSpaceFreeChunksFrom: self firstObject do: aBlock! Item was removed: - ----- Method: SpurMemoryManager>>allOldSpaceFreeChunksFrom:do: (in category 'object enumeration') ----- - allOldSpaceFreeChunksFrom: initialObject do: aBlock - <inline: true> - <doNotGenerate> "Could be generated, but used for debug only" - self allOldSpaceEntitiesFrom: initialObject - do: [:objOop| - (self isFreeObject: objOop) ifTrue: - [aBlock value: objOop]]! Item was changed: ----- Method: SpurMemoryManager>>allocateOldSpaceChunkOfExactlyBytes:suchThat: (in category 'free space') ----- allocateOldSpaceChunkOfExactlyBytes: chunkBytes suchThat: acceptanceBlock "Answer a chunk of oldSpace from the free lists that satisfies acceptanceBlock, if one of this size is available, otherwise answer nil. N.B. the chunk is simply a pointer, it has no valid header. The caller *must* fill in the header correctly." <var: #chunkBytes type: #usqInt> | index node next prev child childBytes | <inline: true> "must inline for acceptanceBlock" "for debugging:" "totalFreeOldSpace := self totalFreeListBytes" index := chunkBytes / self allocationUnit. index < self numFreeLists ifTrue: [(freeListsMask anyMask: 1 << index) ifTrue: [(node := freeLists at: index) = 0 ifTrue: [freeListsMask := freeListsMask - (1 << index)] ifFalse: [prev := 0. [node ~= 0] whileTrue: [self assert: node = (self startOfObject: node). self assert: (self isValidFreeObject: node). next := self fetchPointer: self freeChunkNextIndex ofFreeChunk: node. (acceptanceBlock value: node) ifTrue: [prev = 0 ifTrue: [freeLists at: index put: next] ifFalse: [self storePointer: self freeChunkNextIndex ofFreeChunk: prev withValue: next]. totalFreeOldSpace := totalFreeOldSpace - chunkBytes. ^node]. prev := node. node := next]]]. ^nil]. "Large chunk. Search the large chunk list. Large chunk list organized as a tree, each node of which is a list of chunks of the same size. Beneath the node are smaller and larger blocks. When the search ends parent should hold the first chunk of the same size as chunkBytes, or 0 if none." node := 0. child := freeLists at: 0. [child ~= 0] whileTrue: + [self "Sorry stepping over isValidFreeObject all the time was killing me" + cCode: [self assert: (self isValidFreeObject: child)] + inSmalltalk: [self assertValidFreeObject: child]. - [self assert: (self isValidFreeObject: child). childBytes := self bytesInObject: child. childBytes = chunkBytes ifTrue: "size match; try to remove from list at node first." [node := child. [prev := node. node := self fetchPointer: self freeChunkNextIndex ofFreeChunk: node. node ~= 0] whileTrue: [(acceptanceBlock value: node) ifTrue: [self assert: (self isValidFreeObject: node). self storePointer: self freeChunkNextIndex ofFreeChunk: prev withValue: (self fetchPointer: self freeChunkNextIndex ofFreeChunk: node). totalFreeOldSpace := totalFreeOldSpace - chunkBytes. ^self startOfObject: node]]. (acceptanceBlock value: child) ifFalse: [^nil]. "node was right size but unaceptable." next := self fetchPointer: self freeChunkNextIndex ofFreeChunk: child. next = 0 ifTrue: "no list; remove the interior node" [self unlinkSolitaryFreeTreeNode: child] ifFalse: "list; replace node with it" [self inFreeTreeReplace: child with: next]. totalFreeOldSpace := totalFreeOldSpace - chunkBytes. ^self startOfObject: child] ifFalse: "no size match; walk down the tree" [child := self fetchPointer: (childBytes < chunkBytes ifTrue: [self freeChunkLargerIndex] ifFalse: [self freeChunkSmallerIndex]) ofFreeChunk: child]]. ^nil! Item was added: + ----- Method: SpurMemoryManager>>assertValidFreeObject: (in category 'free space') ----- + assertValidFreeObject: objOop + <doNotGenerate> "If you want to generate this you want to use 'self assert: (self isValidFreeObject: objOop)' instead not to generate code in assertion-free VMs" + | chunk | + "duplicated assertions from isValidFreeObject: because I need to know what is wrong not only that it is not valid (I got bored of stepping inside isValidFreeObject:...)" + self assert: (self oop: (self addressAfter: objOop) isLessThanOrEqualTo: endOfMemory). + chunk := self fetchPointer: self freeChunkNextIndex ofFreeChunk: objOop. + self assert: (chunk = 0 or: [self isFreeOop: chunk]). + (self isLargeFreeObject: objOop) ifTrue: [ + "Tree assertions" + chunk := self fetchPointer: self freeChunkParentIndex ofFreeChunk: objOop. + self assert: (chunk = 0 or: [(self isFreeOop: chunk) and: [self isLargeFreeObject: chunk]]). + chunk := self fetchPointer: self freeChunkSmallerIndex ofFreeChunk: objOop. + self assert: (chunk = 0 or: [(self isFreeOop: chunk) and: [self isLargeFreeObject: chunk]]). + chunk := self fetchPointer: self freeChunkLargerIndex ofFreeChunk: objOop. + self assert: (chunk = 0 or: [(self isFreeOop: chunk) and: [self isLargeFreeObject: chunk]]). ]! Item was changed: ----- Method: SpurMemoryManager>>checkHeapFreeSpaceIntegrity (in category 'debug support') ----- checkHeapFreeSpaceIntegrity "Perform an integrity/leak check using the heapMap. Assume clearLeakMapAndMapAccessibleFreeSpace has set a bit at each free chunk's header. Scan all objects in the heap checking that no pointer points to a free chunk and that all free chunks that refer to others refer to marked chunks. Answer if all checks pass." | ok total | <inline: false> <var: 'total' type: #usqInt> ok := true. total := 0. 0 to: self numFreeLists - 1 do: [:i| (freeLists at: i) ~= 0 ifTrue: [(heapMap heapMapAtWord: (self pointerForOop: (freeLists at: i))) = 0 ifTrue: [coInterpreter print: 'leak in free list '; printNum: i; print: ' to non-free '; printHex: (freeLists at: i); cr. self eek. ok := false]]]. "Excuse the duplication but performance is at a premium and we avoid some tests by splitting the newSpace and oldSpace enumerations." self allNewSpaceEntitiesDo: [:obj| | fieldOop | (self isFreeObject: obj) ifTrue: [coInterpreter print: 'young object '; printHex: obj; print: ' is free'; cr. self eek. ok := false] ifFalse: [0 to: (self numPointerSlotsOf: obj) - 1 do: [:fi| fieldOop := self fetchPointer: fi ofObject: obj. (self isNonImmediate: fieldOop) ifTrue: [(heapMap heapMapAtWord: (self pointerForOop: fieldOop)) ~= 0 ifTrue: [coInterpreter print: 'object leak in '; printHex: obj; print: ' @ '; printNum: fi; print: ' = '; printHex: fieldOop; print: ' is free'; cr. self eek. ok := false]]]]]. self allOldSpaceEntitiesDo: [:obj| | fieldOop | (self isFreeObject: obj) ifTrue: [(heapMap heapMapAtWord: (self pointerForOop: obj)) = 0 ifTrue: [coInterpreter print: 'leak in free chunk '; printHex: obj; print: ' is unmapped?!! '; cr. self eek. ok := false]. fieldOop := self fetchPointer: self freeChunkNextIndex ofFreeChunk: obj. (fieldOop ~= 0 and: [(heapMap heapMapAtWord: (self pointerForOop: fieldOop)) = 0]) ifTrue: [coInterpreter print: 'leak in free chunk '; printHex: obj; print: ' @ 0 = '; printHex: fieldOop; print: ' is unmapped'; cr. self eek. ok := false]. (self isLargeFreeObject: obj) ifTrue: [self freeChunkParentIndex to: self freeChunkLargerIndex do: [:fi| fieldOop := self fetchPointer: fi ofFreeChunk: obj. (fieldOop ~= 0 and: [(heapMap heapMapAtWord: (self pointerForOop: fieldOop)) = 0]) ifTrue: [coInterpreter print: 'leak in free chunk '; printHex: obj; print: ' @ '; printNum: fi; print: ' = '; printHex: fieldOop; print: ' is unmapped'; cr. self eek. ok := false]]]. total := total + (self bytesInObject: obj)] ifFalse: [0 to: (self numPointerSlotsOf: obj) - 1 do: [:fi| + (self isForwarded: obj) + ifTrue: + [self assert: fi = 0. "I'm now trying to use forwarders in GC algorithms..." + fieldOop := self fetchPointer: fi ofMaybeForwardedObject: obj] + ifFalse: "We keep #fetchPointer:ofObject: API here for assertions" + [fieldOop := self fetchPointer: fi ofObject: obj]. - fieldOop := self fetchPointer: fi ofObject: obj. (self isNonImmediate: fieldOop) ifTrue: [(heapMap heapMapAtWord: (self pointerForOop: fieldOop)) ~= 0 ifTrue: [coInterpreter print: 'object leak in '; printHex: obj; print: ' @ '; printNum: fi; print: ' = '; printHex: fieldOop; print: ' is free'; cr. self eek. ok := false]]]]]. total ~= totalFreeOldSpace ifTrue: [coInterpreter print: 'incorrect totalFreeOldSpace; expected '; printNum: totalFreeOldSpace; print: ' found '; printNum: total; cr. self eek. ok := false]. ^ok! Item was changed: ----- Method: SpurMemoryManager>>detachFreeObject: (in category 'free space') ----- detachFreeObject: freeChunk "This is a rare operation, so its efficiency isn't critical. Having a valid prev link for tree nodes would help." <inline: false> | chunkBytes result | chunkBytes := self bytesInObject: freeChunk. result := self allocateOldSpaceChunkOfExactlyBytes: chunkBytes suchThat: [:f| f = freeChunk]. self assert: result = (self startOfObject: freeChunk). - "Following is assertion only. Typical problem is that the free structures (tree/list) keep references to detached object somehow" - self cCode: '' inSmalltalk: - [self allOldSpaceFreeChunksDo: - [ :f | self assert: (self isValidFreeObject: f)]]. ! Item was changed: ----- Method: SpurSelectiveCompactor>>compactSegmentsToCompact (in category 'compaction') ----- compactSegmentsToCompact "Forwards all objects in segments to compact and removes their freechunks" | freeStart | freeStart := segmentToFill segStart. + + "Removes initial free chunk in segment to fill... (Segment is entirely free)" + manager detachFreeObject: (manager objectStartingAt: freeStart). + + "Compact each segment to compact..." 0 to: manager numSegments - 1 do: [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). (self isSegmentBeingCompacted: segInfo) ifTrue: [freeStart := self compactSegment: segInfo freeStart: freeStart ]]. "Final free chunk in segment to fill..." manager addFreeChunkWithBytes: segmentToFill segSize - manager bridgeSize + segmentToFill segStart - freeStart at: freeStart. + "Follow stack zone and caches..." self postForwardingAction ! Item was changed: ----- Method: SpurSelectiveCompactor>>findAndSetSegmentToFill (in category 'freeing') ----- findAndSetSegmentToFill 0 to: manager numSegments - 1 do: [:i| | segInfo firstEntity | segInfo := self addressOf: (manager segmentManager segments at: i). firstEntity := manager objectStartingAt: segInfo segStart. ((manager isFreeObject: firstEntity) and: [(manager objectAfter: firstEntity limit: manager endOfMemory) = (manager segmentManager bridgeFor: segInfo)]) + ifTrue: [segmentToFill := segInfo. ^0]]. - ifTrue: [segmentToFill := segInfo. manager detachFreeObject: firstEntity. ^0]]. ! Item was changed: ----- Method: SpurSelectiveCompactor>>freePastSegmentsAndSetSegmentToFill (in category 'freeing') ----- freePastSegmentsAndSetSegmentToFill + "The first segment being claimed met becomes the segmentToFill. The others are just freed." - "The first segment being claimed met becomes the segmentToFill. The others are just freed" segmentToFill := nil. 0 to: manager numSegments - 1 do: [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). (self isSegmentBeingCompacted: segInfo) ifTrue: + [self freeSegment: segInfo. + segmentToFill ifNil: [segmentToFill := segInfo]]]! - [segmentToFill - ifNil: [segmentToFill := segInfo] - ifNotNil: [self freeSegment: segInfo]]]! Item was changed: ----- Method: SpurSelectiveCompactor>>selectiveCompaction (in category 'compaction') ----- selectiveCompaction "Figures out which segments to compact and compact them into segmentToFill" | atLeastOneSegmentToCompact | self assertNoSegmentBeingCompacted. atLeastOneSegmentToCompact := self computeSegmentsToCompact. "If no compaction we don't pay forwarding cost (stack scan, cache scan, etc.) and we don't allocate segmentToFill if none available." atLeastOneSegmentToCompact ifTrue: [self assert: segmentToFill ~~ nil. + self compactSegmentsToCompact]. - self compactSegmentsToCompact] - ifFalse: - [segmentToFill ifNotNil: [self freeSegment: segmentToFill]]. manager checkFreeSpace: GCModeFull.! Item was added: + ----- Method: SpurSelectiveCompactorSimulator>>selectiveCompaction (in category 'compaction') ----- + selectiveCompaction + super selectiveCompaction. + manager allFreeObjectsDo: [:objOop | manager assertValidFreeObject: objOop]!

3 10

Build completed: opensmalltalk-vm 1.0.1297
by AppVeyor 26 Apr '18

26 Apr '18

1 0

Build failed: opensmalltalk-vm 1.0.1296
by AppVeyor 26 Apr '18

26 Apr '18

1 0

VM Maker: VMMaker.oscog-cb.2376.mcz
by commits＠source.squeak.org 26 Apr '18

26 Apr '18

ClementBera uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2376.mcz ==================== Summary ==================== Name: VMMaker.oscog-cb.2376 Author: cb Time: 26 April 2018, 2:12:30.237019 pm UUID: bf5a9a98-571a-4a0e-985e-9e734854535a Ancestors: VMMaker.oscog-cb.2375 Finally figured out that Slang is not able to do look-ups in compactor classes for some reason, hence only planning compactor was the only one compiling correctly since it defined everything inside... So I duplicated printTheBogon and postSwizzleAction to all subclasses. So I removed inheritance of SpurSelectiveCompactor to SpurSweeper, and duplicated used method. Eliot, if you have any idea why there's no look-up and how to fix it... I'm pretty sure it's just a class-side method somewhere... So I can remove all this duplication. =============== Diff against VMMaker.oscog-cb.2375 =============== Item was changed: CogClass subclass: #SpurCompactor instanceVariableNames: 'manager scavenger coInterpreter' classVariableNames: '' poolDictionaries: 'SpurMemoryManagementConstants VMBasicConstants VMSpurObjectRepresentationConstants' category: 'VMMaker-SpurMemoryManager'! + !SpurCompactor commentStamp: 'cb 4/26/2018 13:53' prior: 0! - !SpurCompactor commentStamp: 'cb 10/3/2017 10:49' prior: 0! Common superclass of all compactors to define apis and simulation variables. + IMPORTANT: This defines APIs only, subclassing is prohibited on auxiliary classes (Slang compiler won't find inherited methods). + The full GC in Spur is split in two, the marking phase and the compactor phase. The subclasses of SpurCompactor are implementations of the second phase, so they are called once the marking phase is finished. SpurCompactor is reponsible for: - freeing unmarked objects - compacting the live old space objects (though each subclass define what it does, some spurCompactor may not compact) - unmarking all objects remaining live - updating oops directly referred by the VM when they are moved (remapObj:/shouldRemapObj: thingy) The main apis are the following: - biasForGC/biasForSnapshot: tells the compactor if the GC is performed for snapshots or not, in general we want to compact more aggressively for snapshots to avoid saving large files with many unused space. - compact: main API, should free the unmarked object, unmark the objects remaining live and potentially compact the heap - remapObj:/shouldRemapObj: => Not really sure what this does, it seems it has to do with updating oops directly referred by the VM when they are moved. Instance Variables coInterpreter: <StackInterpreter> compactedCopySpace: <SpurNewSpaceSpace> manager: <SpurMemoryManager>! Item was changed: ----- Method: SpurCompactor>>postSwizzleAction (in category 'api') ----- postSwizzleAction + self subclassResponsibility! - "Do nothing by default"! Item was changed: ----- Method: SpurCompactor>>printTheBogons: (in category 'debugging') ----- printTheBogons: aBogon + self subclassResponsibility! - <inline: true> - coInterpreter - print: 'bogon '; printHexnp: aBogon; cr! Item was added: + ----- Method: SpurPigCompactor>>postSwizzleAction (in category 'compatibility') ----- + postSwizzleAction + "do nothing" + ! Item was added: + ----- Method: SpurPigCompactor>>printTheBogons: (in category 'debug printing') ----- + printTheBogons: aBogon + <inline: true> + coInterpreter + print: 'bogon '; printHexnp: aBogon; cr! Item was added: + ----- Method: SpurPlanningCompactor>>postSwizzleAction (in category 'compatibility') ----- + postSwizzleAction + "do nothing" + ! Item was changed: + SpurCompactor subclass: #SpurSelectiveCompactor + instanceVariableNames: 'biasForGC segmentToFill' - SpurSweeper subclass: #SpurSelectiveCompactor - instanceVariableNames: 'segmentToFill' classVariableNames: 'MaxOccupationForCompaction' poolDictionaries: '' category: 'VMMaker-SpurMemoryManager'! + !SpurSelectiveCompactor commentStamp: 'cb 4/26/2018 13:59' prior: 0! - !SpurSelectiveCompactor commentStamp: 'cb 4/26/2018 12:53' prior: 0! SpurSelectiveCompactor compacts memory by selecting the memory segments with the most free space and compacting only those, to limit fragmentation while being really quick to perform. The algorithm is fast mostly because it does not update pointers: they are updated lazily during the next marking phase, so there is no need to read the fields of objects in other memory segments that the one compacted. The algorithm works as follow. First, a global sweep pass iterates over the memory linearly, changing unmarked objects to free chunks and concatenating free chunks. During the global sweep phase, the segments of the heap are analysed to determine the percentage of occupation. Second, the least occupied segments are compacted by copying the remaining live objects into an entirely free segment, called regionToFill (we detail later in the paragraph where regionToFill comes from), changing their values to forwarding objects and marking the free chunks as unavailable (removed from free list and marked as data objects). Third, the next marking phase removes all forwarders. Fourth, at the beginning of the next compaction phase the compacted segments from the previous GC can be entirely marked as free space (No need to check anything inside, there were only forwarders and trash data). One of the compacted segment is then selected as the segmentToFill, others are just marked as free chunks. The compaction is effectively partial, compacting only the most critical segments of the heap to limit fragmentation. Compaction time is crazy low, since a low number of objects are moved and pointer updated is lazily done during the next marking phase, while still preventing memory fragmentation. Now this works well when biasForGC is true, but when performing a snapshot, the compactor is just total crap (we need to figure out a solution). + IMPORTANT: I could not figure out to make inheritance work so I copied methods from SpurSweeper here. + segmentToFill <SegInfo> the segment that will be filled through the copying algorithm ! Item was changed: ----- Method: SpurSelectiveCompactor>>assertNoSegmentBeingCompacted (in category 'compaction') ----- assertNoSegmentBeingCompacted - "Assertion only - no segment is being claimed at this point" + | segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> 0 to: manager numSegments - 1 do: + [:i| - [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). self deny: (self isSegmentBeingCompacted: segInfo)]. ! Item was added: + ----- Method: SpurSelectiveCompactor>>biasForGC (in category 'api') ----- + biasForGC + biasForGC := true.! Item was added: + ----- Method: SpurSelectiveCompactor>>biasForSnapshot (in category 'api') ----- + biasForSnapshot + biasForGC := false.! Item was added: + ----- Method: SpurSelectiveCompactor>>bulkFreeChunkFrom: (in category 'sweep phase - copied from SpurSweeper') ----- + bulkFreeChunkFrom: objOop + "ObjOop is either a freeChunk or an object to free, always in old space. The old space entity before objOop is necessarily a marked object. + Attempts to free as many byte from objOop, looking ahead for multiple freechunks / objects to free in a row" + | bytes start next currentObj | + + "Avoids pathological case, not point in dealing with non-mergeable free chunks, we would remove them and re-add them to the free list." + (self isSingleFreeObject: objOop) ifTrue: [^0]. + + "We free unmarked objects and freechunks next to each others and merge them at the same time" + start := manager startOfObject: objOop. + currentObj := objOop. + bytes := 0. + [bytes := bytes + (manager bytesInObject: currentObj). + self freeEntity: currentObj. + next := manager objectStartingAt: start + bytes. + self canUseNextEntityAsFreeSpace: next] + whileTrue: [currentObj := next]. + + manager addFreeChunkWithBytes: bytes at: start. + + ^ next! Item was added: + ----- Method: SpurSelectiveCompactor>>canUseAsFreeSpace: (in category 'sweep phase - copied from SpurSweeper') ----- + canUseAsFreeSpace: objOop + <inline: true> + ^ (manager isFreeObject: objOop) or: [(manager isMarked: objOop) not]! Item was added: + ----- Method: SpurSelectiveCompactor>>canUseNextEntityAsFreeSpace: (in category 'sweep phase - copied from SpurSweeper') ----- + canUseNextEntityAsFreeSpace: next + <inline: true> + ^ (manager oop: next isLessThan: manager endOfMemory) and: [self canUseAsFreeSpace: next]! Item was changed: ----- Method: SpurSelectiveCompactor>>compactSegmentsToCompact (in category 'compaction') ----- compactSegmentsToCompact "Forwards all objects in segments to compact and removes their freechunks" + | segInfo fillStart | + <var: 'segInfo' type: #'SpurSegmentInfo *'> - | fillStart | fillStart := segmentToFill segStart. "Removes initial free chunk in segment to fill... (Segment is entirely free)" manager detachFreeObject: (manager objectStartingAt: fillStart). "Compact each segment to compact..." 0 to: manager numSegments - 1 do: + [:i| - [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). (self isSegmentBeingCompacted: segInfo) ifTrue: [fillStart := self compactSegment: segInfo freeStart: fillStart ]]. "Final free chunk in segment to fill..." manager addFreeChunkWithBytes: segmentToFill segSize - manager bridgeSize + segmentToFill segStart - fillStart at: fillStart. "Follow stack zone and caches..." self postForwardingAction ! Item was changed: ----- Method: SpurSelectiveCompactor>>computeSegmentsToCompact (in category 'compaction') ----- computeSegmentsToCompact "Compute segments to claim: least occupied. Answers true if at least 1 segment is being compacted." - | canStillClaim aboutToClaim aboutToClaimSegment atLeastOneSegmentToCompact | + <var: 'aboutToClaimSegment' type: #'SpurSegmentInfo *'> atLeastOneSegmentToCompact := false. aboutToClaimSegment := self findNextSegmentToCompact. "Segment to fill is one of the segment compacted last GC. If no segment were compacted last GC, and that there is at least one segment to compact, allocate a new one." aboutToClaimSegment ifNil: [^false]. segmentToFill ifNil: [self findOrAllocateSegmentToFill]. canStillClaim := segmentToFill segSize - manager bridgeSize. [aboutToClaimSegment ifNil: [^atLeastOneSegmentToCompact]. aboutToClaim := aboutToClaimSegment segSize - manager bridgeSize * ((self occupationOf: aboutToClaimSegment) + 1) // 255. "+1 to round up, this is approx" aboutToClaim < canStillClaim ] whileTrue: [self markSegmentAsBeingCompacted: aboutToClaimSegment. atLeastOneSegmentToCompact := true. canStillClaim := canStillClaim - aboutToClaim. aboutToClaimSegment := self findNextSegmentToCompact]. ^atLeastOneSegmentToCompact! Item was changed: ----- Method: SpurSelectiveCompactor>>findAndSetSegmentToFill (in category 'freeing') ----- findAndSetSegmentToFill + | segInfo firstEntity | + <var: 'segInfo' type: #'SpurSegmentInfo *'> 0 to: manager numSegments - 1 do: + [:i| - [:i| | segInfo firstEntity | segInfo := self addressOf: (manager segmentManager segments at: i). firstEntity := manager objectStartingAt: segInfo segStart. ((manager isFreeObject: firstEntity) and: [(manager objectAfter: firstEntity limit: manager endOfMemory) = (manager segmentManager bridgeFor: segInfo)]) ifTrue: [segmentToFill := segInfo. ^0]]. ! Item was changed: ----- Method: SpurSelectiveCompactor>>findNextSegmentToCompact (in category 'compaction') ----- findNextSegmentToCompact "Answers the next segment to compact or nil if none. The next segment to compact: - cannot be segment 0 (Segment 0 has specific objects (nil, true, etc.) and special size computed at start-up that we don't want to deal with) - cannot be be a segment already being compacted. - cannot contain pinned object (since we're in a copying GC) - cannot be entirely empty (no need to block that empty segment until next marking phase) - cannot have a high occupation rate (> MaxOccupationForCompaction)" + | leastOccupied leastOccupiedSegment tempOccupied segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> - | leastOccupied leastOccupiedSegment tempOccupied | leastOccupied := 255. 1 to: manager numSegments - 1 do: + [:i| - [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). ((self isSegmentBeingCompacted: segInfo) or: [segInfo containsPinned or: [manager segmentManager isEmptySegment: segInfo] ]) ifFalse: [(tempOccupied := self occupationOf: segInfo) <= leastOccupied ifTrue: [ leastOccupied := tempOccupied. leastOccupiedSegment := segInfo ]]]. leastOccupied > MaxOccupationForCompaction ifTrue: [^nil]. ^ leastOccupiedSegment! Item was added: + ----- Method: SpurSelectiveCompactor>>freeEntity: (in category 'sweep phase - copied from SpurSweeper') ----- + freeEntity: entity + <inline: true> + (manager isFreeObject: entity) + ifFalse: "Freed old space objects are removed from remembered table" + [(manager isRemembered: entity) ifTrue: + [scavenger forgetObject: entity]] + ifTrue: "Merged old space free chunks are removed from free list" + [manager detachFreeObject: entity] + ! Item was changed: ----- Method: SpurSelectiveCompactor>>freePastSegmentsAndSetSegmentToFill (in category 'freeing') ----- freePastSegmentsAndSetSegmentToFill "The first segment being claimed met becomes the segmentToFill. The others are just freed." + | segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> segmentToFill := nil. 0 to: manager numSegments - 1 do: + [:i| - [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). (self isSegmentBeingCompacted: segInfo) ifTrue: [self freeSegment: segInfo. segmentToFill ifNil: [segmentToFill := segInfo]]]! Item was added: + ----- Method: SpurSelectiveCompactor>>isSingleFreeObject: (in category 'sweep phase - copied from SpurSweeper') ----- + isSingleFreeObject: objOop + <inline: true> + | next | + ^ (manager isFreeObject: objOop) and: + [next := manager objectAfter: objOop limit: manager endOfMemory. + (manager oop: next isGreaterThanOrEqualTo: manager endOfMemory) or: [manager isMarked: next]]! Item was changed: ----- Method: SpurSelectiveCompactor>>occupationOf: (in category 'segment access') ----- + occupationOf: segInfo + <var: 'segInfo' type: #'SpurSegmentInfo *'> - occupationOf: segment - <var: 'segment' type: #'SpurSegmentInfo *'> "Swizzle is abused bit 8 isClaimed bits 0-7 occupation" + ^segInfo swizzle bitAnd: 16rFF! - ^segment swizzle bitAnd: 16rFF! Item was changed: ----- Method: SpurSelectiveCompactor>>postSwizzleAction (in category 'api') ----- postSwizzleAction "Since the compact abuses the swizzle field of segment, it needs to be rest after start-up." + | segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> 0 to: manager numSegments - 1 do: + [:i| - [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). segInfo swizzle: 0 ]! Item was added: + ----- Method: SpurSelectiveCompactor>>printTheBogons: (in category 'debugging') ----- + printTheBogons: aBogon + <inline: true> + coInterpreter + print: 'bogon '; printHexnp: aBogon; cr! Item was added: + ----- Method: SpurSelectiveCompactor>>remapObj: (in category 'api') ----- + remapObj: objOop + <api> + <inline: false> + ^manager vanillaRemapObj: objOop! Item was changed: ----- Method: SpurSelectiveCompactor>>setOccupation:used:unused: (in category 'segment access') ----- + setOccupation: segInfo used: used unused: unused + <var: 'segInfo' type: #'SpurSegmentInfo *'> - setOccupation: segment used: used unused: unused - <var: 'segment' type: #'SpurSegmentInfo *'> "Swizzle is abused bit 8 isClaimed bits 0-7 occupation Setting occupation resets the claim bit" | occupation | occupation := used * 255 // (used + unused). + segInfo swizzle: occupation! - segment swizzle: occupation! Item was changed: ----- Method: SpurSelectiveCompactor>>setOccupationAtIndex:used:unused: (in category 'segment access') ----- setOccupationAtIndex: segmentIndex used: used unused: unused "Swizzle is abused bit 8 isClaimed bits 0-7 occupation Setting occupation resets the claim bit" | occupation segInfo | + <var: 'segInfo' type: #'SpurSegmentInfo *'> segInfo := self addressOf: (manager segmentManager segments at: segmentIndex). occupation := used * 255 // (used + unused). segInfo swizzle: occupation! Item was added: + ----- Method: SpurSelectiveCompactor>>shouldRemapObj: (in category 'api') ----- + shouldRemapObj: objOop + <api> + ^manager vanillaShouldRemapObj: objOop! Item was added: + ----- Method: SpurSelectiveCompactor>>unmark: (in category 'sweep phase - copied from SpurSweeper') ----- + unmark: objOop + self assert: ((manager isMarked: objOop) and: [(manager isFreeObject: objOop) not]). + (manager isSegmentBridge: objOop) ifFalse: [manager setIsMarkedOf: objOop to: false]. + (manager isPinned: objOop) ifTrue: [manager segmentManager notePinned: objOop]! Item was added: + ----- Method: SpurSweeper>>postSwizzleAction (in category 'api') ----- + postSwizzleAction + "do nothing" + ! Item was added: + ----- Method: SpurSweeper>>printTheBogons: (in category 'debugging') ----- + printTheBogons: aBogon + <inline: true> + coInterpreter + print: 'bogon '; printHexnp: aBogon; cr!

2 1

VM Maker: VMMaker.oscog-cb.2375.mcz
by commits＠source.squeak.org 26 Apr '18

26 Apr '18

ClementBera uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2375.mcz ==================== Summary ==================== Name: VMMaker.oscog-cb.2375 Author: cb Time: 26 April 2018, 1:30:32.640264 pm UUID: 41eb0797-6cdb-4654-aa71-263c8dd85f53 Ancestors: VMMaker.oscog-cb.2374 Made C compiler happy (addressOf:, macro & shadowing non-sense).. =============== Diff against VMMaker.oscog-cb.2374 =============== Item was changed: ----- Method: SpurSegmentManager>>segments (in category 'accessing') ----- segments + <cmacro: '() GIV(segments)'> ^segments! Item was changed: ----- Method: SpurSelectiveCompactor>>compactSegment:freeStart: (in category 'compaction') ----- compactSegment: segInfo freeStart: initialFreeStart <var: 'segInfo' type: #'SpurSegmentInfo *'> + | currentEntity fillStart bytesToCopy numSlots bridge | - | currentEntity fillStart bytesInObject numSlots bridge | fillStart := initialFreeStart. bridge := manager segmentManager bridgeFor: segInfo. currentEntity := manager objectStartingAt: segInfo segStart. [self oop: currentEntity isLessThan: bridge] whileTrue: [(manager isFreeObject: currentEntity) ifTrue: ["To avoid confusing too much Spur (especially the leak/free checks), we mark the free chunk as a word object." manager detachFreeObject: currentEntity. manager set: currentEntity classIndexTo: manager wordSizeClassIndexPun formatTo: manager wordIndexableFormat] ifFalse: ["Copy the object in segmentToFill and replace it by a forwarder." self assert: (manager isPinned: currentEntity) not. numSlots := manager numSlotsOfAny: currentEntity. + bytesToCopy := manager bytesInObject: currentEntity. - bytesInObject := manager bytesInObject: currentEntity. self assert: (manager objectBytesForSlots: numSlots) = (manager bytesInObject: currentEntity). + manager mem: fillStart asVoidPointer cp: (manager startOfObject: currentEntity) asVoidPointer y: bytesToCopy. - manager mem: fillStart asVoidPointer cp: (manager startOfObject: currentEntity) asVoidPointer y: bytesInObject. self assert: (manager baseHeader: (manager objectStartingAt: fillStart)) = (manager baseHeader: currentEntity). self assert: (manager fetchPointer: numSlots - 1 ofObject: (manager objectStartingAt: fillStart)) = (manager fetchPointer: numSlots - 1 ofObject: currentEntity). manager forward: currentEntity to: (manager objectStartingAt: fillStart). fillStart := fillStart + (manager objectBytesForSlots: numSlots). self assert: (manager isForwarded: currentEntity). self assert: fillStart < (segmentToFill segLimit - manager bridgeSize)]. currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. self assert: currentEntity = bridge. ^ fillStart! Item was changed: ----- Method: SpurSelectiveCompactor>>internalGlobalSweepAndSegmentOccupationAnalysis (in category 'sweep phase') ----- internalGlobalSweepAndSegmentOccupationAnalysis "Iterate over old space, free unmarked objects, annotate each segment with each occupation" | currentEntity nextBridge start segmentIndex currentUsed currentUnused | currentEntity := manager firstObject. nextBridge := manager segmentManager bridgeAt: 0. segmentIndex := currentUnused := currentUsed := 0. [self oop: currentEntity isLessThan: manager endOfMemory] whileTrue: [currentEntity = nextBridge ifTrue: ["End of segment, set occupation" self + setOccupationAtIndex: segmentIndex - setOccupation: (manager segmentManager segments at: segmentIndex) used: currentUsed unused: currentUnused. currentUnused := currentUsed := 0. segmentIndex := segmentIndex + 1. self unmark: currentEntity. nextBridge := manager segmentManager bridgeAt: segmentIndex] ifFalse: ["In-segment, sweep and compute occupation" (self canUseAsFreeSpace: currentEntity) ifTrue: ["bulkFreeChunkFrom: may change a 1 word header object to a double word header object" start := manager startOfObject: currentEntity. self bulkFreeChunkFrom: currentEntity. currentEntity := manager objectStartingAt: start. currentUnused := currentUnused + (manager numSlotsOfAny: currentEntity)] ifFalse: [self unmark: currentEntity. currentUsed := currentUsed + (manager numSlotsOfAny: currentEntity)]]. currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. "set last segment (last bridge = endOfMemory)" self + setOccupationAtIndex: segmentIndex - setOccupation: (manager segmentManager segments at: segmentIndex) used: currentUsed unused: currentUnused.! Item was added: + ----- Method: SpurSelectiveCompactor>>setOccupationAtIndex:used:unused: (in category 'segment access') ----- + setOccupationAtIndex: segmentIndex used: used unused: unused + "Swizzle is abused bit 8 isClaimed bits 0-7 occupation + Setting occupation resets the claim bit" + | occupation segInfo | + segInfo := self addressOf: (manager segmentManager segments at: segmentIndex). + occupation := used * 255 // (used + unused). + segInfo swizzle: occupation!

1 0

VM Maker: VMMaker.oscog-cb.2374.mcz
by commits＠source.squeak.org 26 Apr '18

26 Apr '18

ClementBera uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-cb.2374.mcz ==================== Summary ==================== Name: VMMaker.oscog-cb.2374 Author: cb Time: 26 April 2018, 1:04:16.030075 pm UUID: 62a33f21-e764-413a-b544-289e012d498a Ancestors: VMMaker.oscog-cb.2373 Improved comments & rename variables for Slang to C compilation not to complain (Shadowing globals with inlining cross-classes is a little bit tricky, let's not take any risks) =============== Diff against VMMaker.oscog-cb.2373 =============== Item was changed: SpurSweeper subclass: #SpurSelectiveCompactor instanceVariableNames: 'segmentToFill' classVariableNames: 'MaxOccupationForCompaction' poolDictionaries: '' category: 'VMMaker-SpurMemoryManager'! + !SpurSelectiveCompactor commentStamp: 'cb 4/26/2018 12:53' prior: 0! - !SpurSelectiveCompactor commentStamp: 'cb 4/19/2018 08:49' prior: 0! SpurSelectiveCompactor compacts memory by selecting the memory segments with the most free space and compacting only those, to limit fragmentation while being really quick to perform. The algorithm is fast mostly because it does not update pointers: they are updated lazily during the next marking phase, so there is no need to read the fields of objects in other memory segments that the one compacted. + The algorithm works as follow. First, a global sweep pass iterates over the memory linearly, changing unmarked objects to free chunks and concatenating free chunks. During the global sweep phase, the segments of the heap are analysed to determine the percentage of occupation. Second, the least occupied segments are compacted by copying the remaining live objects into an entirely free segment, called regionToFill (we detail later in the paragraph where regionToFill comes from), changing their values to forwarding objects and marking the free chunks as unavailable (removed from free list and marked as data objects). Third, the next marking phase removes all forwarders. Fourth, at the beginning of the next compaction phase the compacted segments from the previous GC can be entirely marked as free space (No need to check anything inside, there were only forwarders and trash data). One of the compacted segment is then selected as the segmentToFill, others are just marked as free chunks. - The algorithm works as follow. First, a global sweep pass iterates over the memory linearly, changing unmarked objects to free chunks and concatenating free chunks. During the global sweep phase, the segments of the heap are analysed to determine the percentage of occupation. Second, the least occupied segments are compacted by copying the remaining live objects into an entirely free segment, called regionToFill (we detail later in the paragraph where regionToFill comes from) and changing their values to forwarding objects. The rest of each segment being freed is removed from the freeList, it will be entirely freed at the beginning of the next compaction phase. Third, the next marking phase removes all forwarders. Fourth, at the beginning of the next compaction phase the compacted segments from the previous GC can be entirely marked as free space (No need to check anything inside, there were only forwarders that were removed and free chunks not on the free list). One of the freed s egment is then selected as the regionToFill, others are just marked as free space. The compaction is effectively partial, compacting only the most critical regions of the heap to limit fragmentation. + The compaction is effectively partial, compacting only the most critical segments of the heap to limit fragmentation. Compaction time is crazy low, since a low number of objects are moved and pointer updated is lazily done during the next marking phase, while still preventing memory fragmentation. + Now this works well when biasForGC is true, but when performing a snapshot, the compactor is just total crap (we need to figure out a solution). segmentToFill <SegInfo> the segment that will be filled through the copying algorithm ! Item was changed: ----- Method: SpurSelectiveCompactor>>compactSegment:freeStart: (in category 'compaction') ----- compactSegment: segInfo freeStart: initialFreeStart <var: 'segInfo' type: #'SpurSegmentInfo *'> + | currentEntity fillStart bytesInObject numSlots bridge | + fillStart := initialFreeStart. - | currentEntity freeStart bytesInObject numSlots bridge | - freeStart := initialFreeStart. bridge := manager segmentManager bridgeFor: segInfo. currentEntity := manager objectStartingAt: segInfo segStart. [self oop: currentEntity isLessThan: bridge] whileTrue: [(manager isFreeObject: currentEntity) ifTrue: ["To avoid confusing too much Spur (especially the leak/free checks), we mark the free chunk as a word object." manager detachFreeObject: currentEntity. manager set: currentEntity classIndexTo: manager wordSizeClassIndexPun formatTo: manager wordIndexableFormat] ifFalse: ["Copy the object in segmentToFill and replace it by a forwarder." self assert: (manager isPinned: currentEntity) not. numSlots := manager numSlotsOfAny: currentEntity. bytesInObject := manager bytesInObject: currentEntity. self assert: (manager objectBytesForSlots: numSlots) = (manager bytesInObject: currentEntity). + manager mem: fillStart asVoidPointer cp: (manager startOfObject: currentEntity) asVoidPointer y: bytesInObject. + self assert: (manager baseHeader: (manager objectStartingAt: fillStart)) = (manager baseHeader: currentEntity). + self assert: (manager fetchPointer: numSlots - 1 ofObject: (manager objectStartingAt: fillStart)) = (manager fetchPointer: numSlots - 1 ofObject: currentEntity). + manager forward: currentEntity to: (manager objectStartingAt: fillStart). + fillStart := fillStart + (manager objectBytesForSlots: numSlots). - manager mem: freeStart asVoidPointer cp: (manager startOfObject: currentEntity) asVoidPointer y: bytesInObject. - self assert: (manager baseHeader: (manager objectStartingAt: freeStart)) = (manager baseHeader: currentEntity). - self assert: (manager fetchPointer: numSlots - 1 ofObject: (manager objectStartingAt: freeStart)) = (manager fetchPointer: numSlots - 1 ofObject: currentEntity). - manager forward: currentEntity to: (manager objectStartingAt: freeStart). - freeStart := freeStart + (manager objectBytesForSlots: numSlots). self assert: (manager isForwarded: currentEntity). + self assert: fillStart < (segmentToFill segLimit - manager bridgeSize)]. - self assert: freeStart < (segmentToFill segLimit - manager bridgeSize)]. currentEntity := manager objectAfter: currentEntity limit: manager endOfMemory]. self assert: currentEntity = bridge. + ^ fillStart! - ^ freeStart! Item was changed: ----- Method: SpurSelectiveCompactor>>compactSegmentsToCompact (in category 'compaction') ----- compactSegmentsToCompact "Forwards all objects in segments to compact and removes their freechunks" + | fillStart | + fillStart := segmentToFill segStart. - | freeStart | - freeStart := segmentToFill segStart. "Removes initial free chunk in segment to fill... (Segment is entirely free)" + manager detachFreeObject: (manager objectStartingAt: fillStart). - manager detachFreeObject: (manager objectStartingAt: freeStart). "Compact each segment to compact..." 0 to: manager numSegments - 1 do: [:i| | segInfo | segInfo := self addressOf: (manager segmentManager segments at: i). (self isSegmentBeingCompacted: segInfo) + ifTrue: [fillStart := self compactSegment: segInfo freeStart: fillStart ]]. - ifTrue: [freeStart := self compactSegment: segInfo freeStart: freeStart ]]. "Final free chunk in segment to fill..." manager + addFreeChunkWithBytes: segmentToFill segSize - manager bridgeSize + segmentToFill segStart - fillStart + at: fillStart. - addFreeChunkWithBytes: segmentToFill segSize - manager bridgeSize + segmentToFill segStart - freeStart - at: freeStart. "Follow stack zone and caches..." self postForwardingAction !

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev April 2018