Hi,
Thanks for you example, I could reproduce.
It seems when the VM tries to grow the remember set while there is not enough free space in old space to grow it, it does that error.
In your case, the remember set grows in the middle of a GC, and during GC there is not enough free space in old space to allocate a larger remember set. The full GC includes a scavenge, the scavenge tenures objects leading to a growth of the remembered table, and as old space is not reclaimed yet (later in the full GC phase), there is not enough free space for it. I don't think at this point we can do a scavenge for Remembered table shrinkage (we're already in the middle of a scavenge, which is part of the full GC). Hence I think the best bet is to allocate a new old space memory segment, even though that operation can fail, it's still better than crashing. There are other solutions I can think of but I don't like any of them.
In SpurGenerationScavenger>>growRememberedSet, we have:
...
newObj := manager allocatePinnedSlots: numSlots * 2.
newObj ifNil:
[newObj := manager allocatePinnedSlots: numSlots + 1024.
newObj ifNil:
[self error: 'could not grow remembered set']].
...
If I replace:
self error: 'could not grow remembered set'
by:
(manager growOldSpaceByAtLeast: numSlots + 1024) ifNil: [self error: 'could not grow remembered set'].
newObj := manager allocatePinnedSlots: numSlots + 1024. "cannot fail"
Then your example works (in 5min45sec on my machine).
I would like to have Eliot's opinion before integrating as I am not sure if growing old space in the middle of a scavenge performed during a full GC is a good idea, there might be some strange uncommon interactions with the rest of the GC logic I don't see right now.
Eliot what do you think ?