Vm-dev January 2024

vm-dev@lists.squeakfoundation.org

8 participants
11 discussions

[OpenSmalltalk/opensmalltalk-vm] use --whole-archive option instead of -whole-archive for solaris ld (PR #673)
by dcstes 19 Apr '24

19 Apr '24

I believe that the GNU link editor supports both single hyphen and double hyphen options -whole-archive and --whole-archive However the SUN (Solaris and SunOS and Illumos) link editor uses different names like "-z weakextract" Three is a GNU compatibility layer but the SUN link editor insists for GNU compatibility on the double hyphen format -whole-archive and --no-whole-archive David Stes You can view, comment on, or merge this pull request online at: https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/673 -- Commit Summary -- * use --whole-archive option instead of -whole-archive for solaris ld -- File Changes -- M platforms/unix/config/Makefile.in (2) -- Patch Links -- https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/673.patch https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/673.diff -- Reply to this email directly or view it on GitHub: https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/673 You are receiving this because you are subscribed to this thread. Message ID: <OpenSmalltalk/opensmalltalk-vm/pull/673(a)github.com>

2 4

[OpenSmalltalk/opensmalltalk-vm] mmap() sqUnixSpurMemory patch (Issue #665)
by David Stes 03 Mar '24

03 Mar '24

Hi, I have sent in the past email to Eliot Miranda about an issue which I discovered in August 2021 (two years ago), and I thought to raise an issue here. It is a Linux/UNIX patch issue with the behavior of the mmap() system call or library call as used by the OpenSmalltalk COG vm. When way back in 2021 some changes were made for Linux ARM8 to the file saUnixSpurMemory.c I reported that this was breaking the Solaris 11.3 build. Curiously the Solaris 11.4 build was still working fine. I think (I don't know) that perhaps the behavior of mmap() on Solaris 11.4 is slightly diferent than on 11.3, to match (perhaps) the Linux kernel behavior of mmap(). An idea that I have is to ask some UNIX guru's whether there exist a autoconf-archive (configure script) test or script to test the behavior of mmap() for portability. However here I will just report what the issue is : since august 2021 I am building succesfully on Solaris with the following patch ``` --- opensmalltalk-vm-sun-v5.0.34/platforms/unix/vm/sqUnixSpurMemory.c Fri Aug 13 19:54:42 2021 +++ p0/opensmalltalk-vm-sun-v5.0.34/platforms/unix/vm/sqUnixSpurMemory.c Sun Aug 15 14:11:58 2021 @@ -277,16 +277,21 @@ { void *hint = sbrk(0); // a hint of the lowest possible address for mmap void *result; + char *address; + unsigned long alignment; pageSize = getpagesize(); pageMask = ~(pageSize - 1); + alignment = max(pageSize,1024*1024); + address = (char *)(((usqInt)hint + alignment - 1) & ~(alignment - 1)); + #if !defined(MAP_JIT) # define MAP_JIT 0 #endif *desiredSize = roundUpToPage(*desiredSize); - result = mmap(hint, *desiredSize, + result = mmap(address, *desiredSize, #if DUAL_MAPPED_CODE_ZONE PROT_READ | PROT_EXEC, #else ``` I really need Eliot's help on this. Eliot : what I did was try to understand your code and you seem to make some alignment calculations on the desired memory location that you request through mmap(). I believe that the traditional UNIX mmap() or the Solaris 10 and Solaris 11.3 mmap() is reacting slightly differently to the requested memory "hint". By some experimentatoin and debugging I found the above patch, which I wrote based on looking at your code and try to mirror the calculations that you are doing, which are not fully symmetric. By copying some code, I discovered that with the above patch the build for both Solaris 11.3 and Solaris 11.4 are working fine again. I think this is a general UNIX issue that could be of interest to all UNIX software. If an autoconfigure autoconf-archive test would exist to test the behavior of mmap() then that could be dealt in all software that uses mmap() with anonymous memory mapping. As far as I know the history of the mmap() call is that it originated in the world of Sun and Solaris and is since a very long time present in Linux as well, and I think some improvements in Linux where then backported to Solaris 11.4. That would explain why the current OpenSmalltalk-VM code works fine on Solaris 11.4. For reasons of portability to all flavors of UNIX and Linux, I think a configure build test for the code could benefit portability of your Cog VM. Let us know please what you think. I could simply submit my patch as a patch to mainstream upstream OpenSmalltalk but I don't know your opinion about it. And the way I have been using it as a platform specific patch (that I apply prior to the build on Solaris) is fine for me as well. Regards, David Stes -- Reply to this email directly or view it on GitHub: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/665 You are receiving this because you are subscribed to this thread. Message ID: <OpenSmalltalk/opensmalltalk-vm/issues/665(a)github.com>

4 19

[OpenSmalltalk/opensmalltalk-vm] align return value of sbrk() (PR #672)
by dcstes 02 Mar '24

02 Mar '24

Possible fix for issue mmap() sqUnixSpurMemory patch #665 David Stes You can view, comment on, or merge this pull request online at: https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/672 -- Commit Summary -- * align return value of sbrk() -- File Changes -- M platforms/unix/vm/sqUnixSpurMemory.c (7) -- Patch Links -- https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/672.patch https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/672.diff -- Reply to this email directly or view it on GitHub: https://github.com/OpenSmalltalk/opensmalltalk-vm/pull/672 You are receiving this because you are subscribed to this thread. Message ID: <OpenSmalltalk/opensmalltalk-vm/pull/672(a)github.com>

3 8

[OpenSmalltalk/opensmalltalk-vm] b8d838: CogVM source as per VMMaker.oscog-eem.3347
by Eliot Miranda 25 Jan '24

25 Jan '24

Branch: refs/heads/Cog Home: https://github.com/OpenSmalltalk/opensmalltalk-vm Commit: b8d838c2c391f12c1d85c22823daff906e03529c https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/b8d838c2c391f12c1d… Author: Eliot Miranda <eliot.miranda(a)gmail.com> Date: 2024-01-24 (Wed, 24 Jan 2024) Changed paths: M src/plugins/B3DAcceleratorPlugin/B3DAcceleratorPlugin.c M src/spur32.cog.lowcode/cogit.h M src/spur32.cog.lowcode/cogitARMv5.c M src/spur32.cog.lowcode/cogitIA32.c M src/spur32.cog.lowcode/cointerp.c M src/spur32.cog.lowcode/cointerp.h M src/spur32.cog.lowcode/gcc3x-cointerp.c M src/spur32.cog/cogit.h M src/spur32.cog/cogitARMv5.c M src/spur32.cog/cogitIA32.c M src/spur32.cog/cointerp.c M src/spur32.cog/cointerp.h M src/spur32.cog/cointerpmt.c M src/spur32.cog/cointerpmt.h M src/spur32.cog/gcc3x-cointerp.c M src/spur32.cog/gcc3x-cointerpmt.c M src/spur32.sista/cogit.h M src/spur32.sista/cogitARMv5.c M src/spur32.sista/cogitIA32.c M src/spur32.sista/cointerp.c M src/spur32.sista/cointerp.h M src/spur32.sista/gcc3x-cointerp.c M src/spur32.stack.lowcode/gcc3x-interp.c M src/spur32.stack.lowcode/interp.c M src/spur32.stack/gcc3x-interp.c M src/spur32.stack/interp.c M src/spur32.stack/validImage.c M src/spur64.cog.lowcode/cogit.h M src/spur64.cog.lowcode/cogitARMv8.c M src/spur64.cog.lowcode/cogitX64SysV.c M src/spur64.cog.lowcode/cogitX64WIN64.c M src/spur64.cog.lowcode/cointerp.c M src/spur64.cog.lowcode/cointerp.h M src/spur64.cog.lowcode/gcc3x-cointerp.c M src/spur64.cog/cogit.h M src/spur64.cog/cogitARMv8.c M src/spur64.cog/cogitX64SysV.c M src/spur64.cog/cogitX64WIN64.c M src/spur64.cog/cointerp.c M src/spur64.cog/cointerp.h M src/spur64.cog/cointerpmt.c M src/spur64.cog/cointerpmt.h M src/spur64.cog/gcc3x-cointerp.c M src/spur64.cog/gcc3x-cointerpmt.c M src/spur64.sista/cogit.h M src/spur64.sista/cogitARMv8.c M src/spur64.sista/cogitX64SysV.c M src/spur64.sista/cogitX64WIN64.c M src/spur64.sista/cointerp.c M src/spur64.sista/cointerp.h M src/spur64.sista/gcc3x-cointerp.c M src/spur64.stack.lowcode/gcc3x-interp.c M src/spur64.stack.lowcode/interp.c M src/spur64.stack/gcc3x-interp.c M src/spur64.stack/interp.c M src/spur64.stack/validImage.c M src/v3.cog/cointerp.c M src/v3.cog/gcc3x-cointerp.c Log Message: ----------- CogVM source as per VMMaker.oscog-eem.3347 Integrate Jörg Belger's fixes to growing memory by the right ammoun when the first attempt fails at growing the remembered set and/or an obj stack. Inline CogObjectRepresentationForSpur>>#cacheTagIsMarked: always. Fix a typo.

1 0

VM Maker: VMMaker.oscog-eem.3347.mcz
by commits＠source.squeak.org 25 Jan '24

25 Jan '24

Eliot Miranda uploaded a new version of VMMaker to project VM Maker: http://source.squeak.org/VMMaker/VMMaker.oscog-eem.3347.mcz ==================== Summary ==================== Name: VMMaker.oscog-eem.3347 Author: eem Time: 24 January 2024, 8:37:34.392423 pm UUID: 7e4df21b-b561-434b-b2d3-45e8806d7875 Ancestors: VMMaker.oscog-topa.3346 Integrate Jörg Belger's fixes to growing memory by the right ammount when the first attempt fails at growing the remembered set and/or an obj stack. Revert the regression introduced by the non-fix for inlining in VMMaker.oscog-eem.3340. It needs more careful work (and currently affects only the threaded VM). Inline CogObjectRepresentationForSpur>>#cacheTagIsMarked: always. Fix a typo. =============== Diff against VMMaker.oscog-topa.3346 =============== Item was changed: ----- Method: CCodeGenerator>>structClassForType: (in category 'C code generator') ----- structClassForType: structureTypeString + | index structureTypeName | - | index sturctureTypeName | index := structureTypeString findFirst: [:c| c isLetter not]. + structureTypeName := index = 0 - sturctureTypeName := index = 0 ifTrue: [structureTypeString] ifFalse: [structureTypeString first: index - 1]. + ^(Smalltalk classNamed: structureTypeName) ifNil: - ^(Smalltalk classNamed: sturctureTypeName) ifNil: [VMStructType allSubclasses + detect: [:sc| sc structTypeName = structureTypeName] - detect: [:sc| sc structTypeName = sturctureTypeName] ifNone: nil]! Item was changed: ----- Method: CogObjectRepresentationForSpur>>cacheTagIsMarked: (in category 'garbage collection') ----- cacheTagIsMarked: cacheTag "Answer if the cacheTag is not unmarked, i.e. answer true for compact class indices and immediates; only answer false for unmarked objects. In Spur linked send cache tags are class indices so effectively they're always marked." + <inline: #always> ^true! Item was changed: ----- Method: SpurGenerationScavenger>>growRememberedSet (in category 'remembered set') ----- growRememberedSet | obj numSlots newObj base | <inline: false> "Don't ruin locality in remember:" <var: #base type: #'sqInt *'> obj := manager rememberedSetObj. numSlots := manager numSlotsOf: obj. self assert: numSlots >= 1024. newObj := manager allocatePinnedSlots: numSlots * 2. newObj ifNil: [newObj := manager allocatePinnedSlots: numSlots + 1024. newObj ifNil: + [(manager growOldSpaceByAtLeast: (numSlots + 1024) * self bytesPerOop) ifNil: [self error: 'could not grow remembered set']. - [(manager growOldSpaceByAtLeast: numSlots + 1024) ifNil: [self error: 'could not grow remembered set']. newObj := manager allocatePinnedSlots: numSlots + 1024. "cannot fail"]]. manager rememberedSetObj: newObj. base := manager firstIndexableField: newObj. 0 to: rememberedSetSize - 1 do: [:i| base at: i put: (rememberedSet at: i)]. "if growing in the middle of a GC, need to preserve marked status." (manager isMarked: obj) ifTrue: [manager setIsMarkedOf: newObj to: true; setIsMarkedOf: obj to: false]. manager freeObject: obj. rememberedSet := base. rememberedSetLimit := manager numSlotsOf: newObj. self setRememberedSetRedZone! Item was changed: ----- Method: SpurMemoryManager>>ensureRoomOnObjStackAt: (in category 'obj stacks') ----- ensureRoomOnObjStackAt: objStackRootIndex "An obj stack is a stack of objects stored in a hidden root slot, such as the markStack or the ephemeronQueue. It is a linked list of segments, with the hot end at the head of the list. It is a word object. The stack pointer is in ObjStackTopx and 0 means empty. The list goes through ObjStackNextx. We don't want to shrink objStacks, since they're used in GC and its good to keep their memory around. So unused pages created by popping emptying pages are kept on the ObjStackFreex list." | stackOrNil freeOrNewPage | stackOrNil := self fetchPointer: objStackRootIndex ofObject: hiddenRootsObj. (stackOrNil = nilObj or: [(self fetchPointer: ObjStackTopx ofObject: stackOrNil) >= ObjStackLimit]) ifTrue: [freeOrNewPage := stackOrNil = nilObj ifTrue: [0] ifFalse: [self fetchPointer: ObjStackFreex ofObject: stackOrNil]. freeOrNewPage ~= 0 ifTrue: "the free page list is always on the new page." [self storePointer: ObjStackFreex ofObjStack: stackOrNil withValue: 0. self assert: (marking not or: [self isMarked: freeOrNewPage])] ifFalse: [freeOrNewPage := self allocateSlotsInOldSpace: ObjStackPageSlots format: self wordIndexableFormat classIndex: self wordSizeClassIndexPun. freeOrNewPage ifNil: ["Allocate a new segment an retry. This is very uncommon. But it happened to me (Clement)." + self growOldSpaceByAtLeast: ObjStackPageSlots * self bytesPerOop. - self growOldSpaceByAtLeast: ObjStackPageSlots. freeOrNewPage := self allocateSlotsInOldSpace: ObjStackPageSlots format: self wordIndexableFormat classIndex: self wordSizeClassIndexPun. freeOrNewPage ifNil: [self error: 'no memory to allocate or extend obj stack']]. self storePointer: ObjStackFreex ofObjStack: freeOrNewPage withValue: 0. marking ifTrue: [self setIsMarkedOf: freeOrNewPage to: true]]. self storePointer: ObjStackMyx ofObjStack: freeOrNewPage withValue: objStackRootIndex; storePointer: ObjStackNextx ofObjStack: freeOrNewPage withValue: (stackOrNil = nilObj ifTrue: [0] ifFalse: [stackOrNil]); storePointer: ObjStackTopx ofObjStack: freeOrNewPage withValue: 0; storePointer: objStackRootIndex ofObject: hiddenRootsObj withValue: freeOrNewPage. self assert: (self isValidObjStackAt: objStackRootIndex). "Added a new page; now update and answer the relevant cached first page." stackOrNil := self updateRootOfObjStackAt: objStackRootIndex with: freeOrNewPage]. self assert: (self isValidObjStackAt: objStackRootIndex). ^stackOrNil! Item was added: + ----- Method: TMethod>>isStructMethod (in category 'testing') ----- + isStructMethod + ^[definingClass isStructClass] + on: MessageNotUnderstood + do: [:ex| false]! Item was changed: ----- Method: TMethod>>renameVarsForInliningInto:except:in: (in category 'inlining support') ----- renameVarsForInliningInto: destMethod except: doNotRename in: aCodeGen "Rename any variables that would clash with those of the destination method." | destVars usedVars varMap newVarName | destVars := aCodeGen globalsAsSet copy. destVars addAll: destMethod locals. destVars addAll: destMethod args. usedVars := destVars copy. "keeps track of names in use" usedVars addAll: args; addAll: locals. varMap := Dictionary new: 100. locals, args do: [ :v | ((doNotRename includes: v) not and: [destVars includes: v]) ifTrue: [newVarName := self unusedNamePrefixedBy: v avoiding: usedVars. varMap at: v put: newVarName]]. - self protectToBeExcludedReceiversForInlining. self renameVariablesUsing: varMap! Item was changed: ----- Method: TSendNode>>nodeIsThisSendsStructReceiver: (in category 'inlining support') ----- + nodeIsThisSendsStructReceiver: aTParseNode - nodeIsThisSendsStructReceiver: aTParseNode - selector == #'atomic_store:_:' ifTrue: [self halt]. ^aTParseNode == receiver and: [receiver name beginsWith: 'self_in_']!

1 0

[OpenSmalltalk/opensmalltalk-vm] Segmentation fault crashes (Solution included) (Issue #675)
by unique75m 24 Jan '24

24 Jan '24

I reported the issues already to Eliot Miranda, but maybe this here is a better way, so that others can recognise the fixes and integrate them. I made now 3 changes in the current VMMaker.image. Please look at the SegFaultFixes.zip. It contains 3 changes on 3 methods. ### TMethod-protectToBeExcludedReceiversForInlining I needed to comment out the code, coz it is the only place where something is transformed to “self_in_inlined_method”, but there is no other reader/usage. The result was that the C code generator created code where the structure accesses like page->baseFP was corrupted and could not be compiled. Normally code like “page -> baseFP” is created, but with the activated smalltalk code there is code like “baseFP(self_in_inlined_method)” created and it cannot find the declaration of “self_in_inline_method" ### SpurMemoryManager-ensureRoomOnObjStackAt I changed "self growOldSpaceByAtLeast: ObjStackPageSlots” into "self growOldSpaceByAtLeast: ObjStackPageSlots * self bytesPerOop”, so that it grows with the right byte size and not slot size. ### SpurGenerationScavenger-growRememberedSet I changed "manager growOldSpaceByAtLeast: numSlots + 1024” to "manager growOldSpaceByAtLeast: (numSlots + 1024) * self bytesPerOop”, so that it grows with right byte size and not slot size. This was my original bug, where the VM crashes on my Linux system with a segmentation fault. The problem was that it tries multiple times to allocate a new remembered set, but it fails and grows then. But the growing does not use bytesize, it uses the number of slots and then the last "never failing" allocate fails again [SegFaultFixes.zip](https://github.com/OpenSmalltalk/opensmalltalk-vm/files/… There is another segmentation fault bug in the virtual machine, but hard to reproduce and not easy to find. It happens sometimes that an object-slot in future-space contains an object-pointer, where the sixth byte is modified. If I change this sixth byte back matching the other slots, then I can reach a valid object at this address. I don't guess that the probability is high that changing a sixth byte leads to a valid object address. So I assume there is somebody that corrupts this single byte. I have modified my virtual machine sources on C side, so that all memory accesses like longPointerAt() use now validation code, but this validation code was not triggered yet and the virtual machine did not crash again. If I know more I will create another issue. But just for the case, maybe somebody has an idea. -- Reply to this email directly or view it on GitHub: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/675 You are receiving this because you are subscribed to this thread. Message ID: <OpenSmalltalk/opensmalltalk-vm/issues/675(a)github.com>

1 0

Are applicative order evaluation and normal order evaluation relevant concepts in the Cog VM ?
by gettimothy 21 Jan '24

21 Jan '24

Thanks for the replies on how to begin to think about a Context. vs Environments. Next up in the SICP study are the terms. applicative order evaluation where the operands are evaluated first and normal order evaluation, the operands are not evaluated until their values are needed. I vaguely remember Eliot relying on Squeak strict Right to Left (or is it Left to Right) evaluation rule in an email from some years ago. Thank you in advance. t

3 3

[ANN] OpenSmalltalk VM 2023.12 released (tag 202312181441)
by Taeumel, Marcel 18 Jan '24

18 Jan '24

Hi all! We just released the next version of the OpenSmalltalk VM. Please find the binaries here: https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/202312181441 (see VMMaker.oscog-eem.3339 and update.oscog-mt.8.mcm) That version will be used in the updated bundles for Squeak 6.0 and 5.3. And probably in upcoming Cuis releases. :-) Here is an attempt of a change log (since 2022.06): - Adds error codes for ImageSegment storage primitive fail - Adds error codes for primitiveFileClose on Linux, macOS, and Windows - Adds machine-code primitive for #primitiveHighResClock on 64-bit platforms - Adds machine-code primitive for #perform:withArguments: (84) - Adds VM parameter 50 for maxLiteralCountForCompile - Adds primitive 96 (primitiveObjectsAccessibleFromRoots) to be used on an array of objects; internally part of primitive 98 (primitiveStoreImageSegment) - Adds support for multi-byte data to ClipboardExtendedPlugin - Adds support for various data types to ClipboardExtendedPlugin on Windows - Adds support for UTF8 Unicode to ClipboardExtendedPlugin on Windows - Adds flag #upscaleDisplayIfHighDPI to be set via command line or stored in image header; true by default - Adds support for -eventtrace on Linux and Windows - Fixes memory leak related to scale-factor lookup on Linux (X11) - Fixes warning about applicationSupportsSecureRestorableState on macOS 12.0+ - Fixes potential GDI leak in getDpiSystem on Windows - Fixes failure of Spur image segment loading due to not having enough memory - Fixes path lookup for "." (i.e., current directory) on Windows - Fixes JIT issue around frameless block activation - Fixes JIT issue around cache flushing on ARMv8 - Fixes bug in primitive 105 (primitiveSpurStringReplace) - Fixes bug in SpurSegmentManager - Fixes bug in BitBlt rgbMul combination - Fixes structure argument passing for ARMv8 FFI call-outs - Fixes float conversions for FFI call-outs - Fixes float arguments for FFI call-outs on RiscV64 - Fixes #become: on compiled methods for Spur flavors - Fixes jitted #perform: primitive by adding argument-count check - Fixes MoveCqR on ARMv8 for certain bit patterns (in the JIT) - Fixes crash on ARMv8 when toggling fail/succeed primitives for mixed mode arithmetic - Fixes accessor-depth calculation in SocketPlugin and other places - Fixes type-declaration problem in DeflatePlugin for 64-bit RiscV - Fixes other issues around SmartSyntaxInterpreterPlugins (e.g., JPEGReadWriter2Plugin, BitBltSimulation, SocketPlugin) - Updates support for compilation on RiscV64 (including FFI plugin) - Updates VectorEnginePlugin to 2022-10-28 (89cc00e86...) from Cuis - Updates BitBltPlugin to be faster for noSource/noHalftone setup - Updates joystick support on Linux I am sure that I forgot something especially in plugin code. Please expand on this. BIG THANKS to everybody who has worked on this release! Personally, I would like to thank Eliot, who is a great software architect who keeps on making the OSVM faster with every commit. Thank you! Best, Marcel (on behalf of the OSVM core dev team)

1 0

Fwd: [OpenSmalltalk/opensmalltalk-vm] Many hard page faults during full GC (Issue #674)
by Eliot Miranda 17 Jan '24

17 Jan '24

1 0

[OpenSmalltalk/opensmalltalk-vm] Many hard page faults during full GC (Issue #674)
by Christoph Thiede 17 Jan '24

17 Jan '24

When working with images that (necessarily) have a size of a few GBs, I frequently note interruptions during full GCs that can take 30 seconds or longer. resmon confirms that there is a high number of hard page faults during these GCs, as my overall RAM consumption is often >90% and most of the time >50% of memory of Squeak.exe is swapped out to my SSD. If somebody felt like investing efforts in making the GC more swap-friendly, this would be highly welcome from my side. For example: <https://people.cs.umass.edu/~emery/pubs/f034-hertz.pdf> (I do not pretend to have read this paper.) -- Reply to this email directly or view it on GitHub: https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/674 You are receiving this because you are subscribed to this thread. Message ID: <OpenSmalltalk/opensmalltalk-vm/issues/674(a)github.com>

4 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev January 2024