I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image. I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.
Which platform? How are you measuring it? I'm not seeing anything that makes me suspect this on pi/linux.
On 2022-06-19, at 2:11 PM, smalltalking notifications@github.com wrote:
I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image. I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful Latin Phrases:- Noli me vocare, ego te vocabo = Don't call me, I'll call you.
Ubuntu 20.04.4 LTS. Start the image and watch as the resident set size grows indefinitely:
$ watch ps -C squeak -o rss
You may prefer htop over watch+ps.
Levente
On Sun, 19 Jun 2022, tim Rowledge wrote:
Which platform? How are you measuring it? I'm not seeing anything that makes me suspect this on pi/linux.
On 2022-06-19, at 2:11 PM, smalltalking notifications@github.com wrote:
I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image. I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful Latin Phrases:- Noli me vocare, ego te vocabo = Don't call me, I'll call you.
Eeek. That's nasty.
On Raspberry Pi OS the number does go up, but at worst very slowly, and seems to go back down as well.
On an x64 Ubuntu 20.04.1 machine of my acquaintance :-
uname -a Linux thing 5.13.0-44-generic #49~20.04.1-Ubuntu SMP Wed May 18 18:44:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Ah, but an older VM, so providing a backstop version :-
Virtual Machine --------------- /usr/bin/sageteacloudsq/lib/squeak Open Smalltalk Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2936] Unix built on Mar 31 2021 18:28:22 Compiler: 4.2.1 Compatible Clang 6.0.0 (tags/RELEASE_600/final) platform sources revision VM: 202101191919 tim@gravious:DizietFS/Documents/Squeak/opensmalltalk-vm Date: Tue Jan 19 11:19:12 2021 CommitHash: 253cbfc14 Plugins: 202101191919 tim@gravious:DizietFS/Documents/Squeak/opensmalltalk-vm CoInterpreter VMMaker.oscog-eem.2936 uuid: 69233536-ece4-4ea1-b523-c28c4aa7d4c2 Mar 31 2021 StackToRegisterMappingCogit VMMaker.oscog-eem.2936 uuid: 69233536-ece4-4ea1-b523-c28c4aa7d4c2 Mar 31 2021
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- MERCI RIEN - Thanks for nothin'.
On Sun, Jun 19, 2022 at 2:11 PM smalltalking ***@***.***> wrote:
I noticed that recent VMs leak a few hundred kilobytes of memory every 3-6 seconds. It happens even if you just start VM and do nothing with the image.
Can someone bisect and identify the commit that introduced this?
I tested it with various 64-bit Linux VMs. There seems to be no upper limit on the leaked memory. The leak happens even with the -memory option set. The size of the object memory is not affected by the leak, saved images remain small and the image does not see the increase in memory usage. Affected VMs include: 5.0-202206021410, 5.0-202205110711, 5.0-202204190959 Unaffected VMs include: 5.0-202112022203 and 5.0-202112201228 The 5.0-202206021410 assert and debug VMs leak the memory as well.
_,,,^..^,,,_ best, Eliot
I cannot reproduce this issue on
OS: Win32 10.0 X64 VM: 202206021410 (c9fd365) SQ: Squeak6.0beta #22056 (64 bit)
So I can trigger it so I'll try to bisect it.
Squeak6.0beta latest update: #22053
platform sources revision VM: 202206181934 XXX@YYY/opensmalltalk-vm Date: Sat Jun 18 12:34:41 2022 CommitHash: 27657fe6a Plugins: 202206181934 XXX@YYY:tmp/opensmalltalk-vm CoInterpreter VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022 StackToRegisterMappingCogit VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022
Ubuntu 22.04 - x86-64, does it, see below
Raspberry PI ARMv8 64 bit based on Debian 11.3, ie, April 4 2022, does not.
Raspberry PI ARMv7 32 bit based on Debian 11.3, ie, April 4 2022 does not. Of course this one is the 32 bit VM and the 32 bit image but the identical image/vm combo as above.
Ubuntu 18.04 ARMv8 64 bit does it, also see below.
========================== Ubuntu 22.04 - x86-64.
$ while true; do ps -C squeak -o rss; sleep 5; done RSS 71668 RSS 71928 RSS 72984 RSS 73248 RSS 73248 RSS 73512 RSS 73776 RSS 73776 RSS 74040 RSS 74040 RSS 74304 RSS 74304 RSS 74568 RSS 74568 RSS 74832 RSS 75096 RSS 75096 RSS 75360 RSS 75360 RSS 75624 RSS 75624 RSS 75888 RSS 76152 RSS 76152
===================================== Ubuntu 18.04, ARMv8 64 bit
76764 RSS 76764 RSS 77028 RSS 77292 RSS 77292 RSS 77556 RSS 77556 RSS 77820 RSS 77820 RSS 78084 RSS 78348 RSS 78348 RSS 78612 RSS 78612 RSS 78876 RSS 78876 RSS 79140 RSS 79404 RSS 79404 RSS 79668 RSS 79668 RSS 79932 RSS 79932 RSS 80196 RSS 80460
So I spent some time looking into this and I think I have tracked down when it happens. In all cases my test case was Ubuntu 22.04 on x86-64 but I also see it on Ubuntu 18.04 on ARMv8.
Some recent change to the combo of the VM and the image will do this, but, only on a local X11 display when you are displaying. Using -vm-display-X11 -headless prevents this from happening, so, good news, servers will be ok. It also does not happen when you use a remote X11 display. It only happens when you use a local graphics card.
It does not show up in the squeak memory usage, just the RSS size. I guess that something is not getting freed somewhere in C, and it looks to be somewhere in the C code for the X11 driver.
Wayland vs X11 does not change anything.
This was not in Squeak 5.3 and the Squeak 53 VM running almost the newest 6.0 image is ok too.
e908c9c22 20220621T1716 ** Squeak6.0rc1 22076 ** Bad 202206021410 ** Squeak6.0rc1 22076 ** Bad e908c9c22 20220621T1716 ** Squeak5.3-19481 ** Good 6a0bc96 202003021730 ** Squeak6.0beta-21757 ** Good
This can be closed here because it is not a VM issue. It is an image side issue between image update 20981, which is good, and image update 21617 which has this problem. Updating a working 20981 image with all updates causes it to start increasing RSS.
Closed relative to https://github.com/squeak-smalltalk/squeak-object-memory/issues/51
Closed #642 as completed.
Reopened #642.
I think this might be better solved on the VM side.
So - Sometime this spring between change 20981 and change 21617 there were changes made to the screen resolution code that changed how often, it seems, we call display_ioScreenScaleFactor. Before it must have been rarely called like on startup and resize or whatever and now it is called often, sometimes many times per second.
The X11 code, being X11 code for Linux, has multiple ways of getting the screen scale factor. Which path was followed depended on some things but on assorted Ubuntu systems the path followed called scale_Xftdpi_usable. This called XrmGetStringDatabase every time it entered but if Xft.dpi was there then it returned without calling XrmDestroyDatabase. Ergo the memory leaked.
The patch is trivial and attached
[PatchX11Scale.patch.txt](https://github.com/OpenSmalltalk/opensmalltalk-vm/files/8977689/PatchX11Scal...) ..
Closed #642 as completed.
Oh, how embarrasing. I'm really sorry.
This can be closed here because it is not a VM issue. It is an image side issue between image update 20981, which is good, and image update 21617 which has this problem. Updating a working 20981 image with all updates causes it to start increasing RSS.
Just a minor remark on this intermediate conclusion. At that point, we already knew that the .image file was not affected. It did not get bigger because of the memory leak. While an image-side workaround might have been possible, it clearly pointed to VM-side code not being correct. From the top of my hat, I cannot think of a scenario where non-FFI, image-only code leaks memory without affecting the .image size. (Maybe thousands of open file handles? But those have image-side counterparts as well, thus affecting the .image size.)
Thanks for the thorough investigation, Bruce!!! ❤️
Documented here: - https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/202206021410
Squeak-app bundles use workaround for now: - https://github.com/squeak-smalltalk/squeak-app/commit/1c8490a9cf8ff60fc58308...
Hi all,
So I can trigger it so I'll try to bisect it.
cheers
bruce
Squeak6.0beta
latest update: #22053
platform sources revision VM: 202206181934 ***@***.***/opensmalltalk-vm Date: Sat Jun 18 12:34:41 2022 CommitHash: 27657fe6a Plugins: 202206181934 ***@***.***:tmp/opensmalltalk-vm
CoInterpreter VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022
StackToRegisterMappingCogit VMMaker.oscog-eem.3188 uuid: 010bab73-d814-4c05-b0f1-f2da78c1568f Jun 20 2022
Ubuntu 22.04 - x86-64, does it, see below
Raspberry PI ARMv8 64 bit based on Debian 11.3, ie, April 4 2022, does not.
Raspberry PI ARMv7 32 bit based on Debian 11.3, ie, April 4 2022 does not. Of course this one is the 32 bit VM and the 32 bit image but the identical image/vm combo as above.
Ubuntu 18.04 ARMv8 64 bit does it, also see below.
==========================
Ubuntu 22.04 - x86-64.
$ while true; do ps -C squeak -o rss; sleep 5; done
RSS
71668
RSS
71928
RSS
72984
RSS
73248
RSS
73248
RSS
73512
RSS
73776
RSS
73776
RSS
74040
RSS
74040
RSS
74304
RSS
74304
RSS
74568
RSS
74568
RSS
74832
RSS
75096
RSS
75096
RSS
75360
RSS
75360
RSS
75624
RSS
75624
RSS
75888
RSS
76152
RSS
76152
=====================================
Ubuntu 18.04, ARMv8 64 bit
76764
RSS
76764
RSS
77028
RSS
77292
RSS
77292
RSS
77556
RSS
77556
RSS
77820
RSS
77820
RSS
78084
RSS
78348
RSS
78348
RSS
78612
RSS
78612
RSS
78876
RSS
78876
RSS
79140
RSS
79404
RSS
79404
RSS
79668
RSS
79668
RSS
79932
RSS
79932
RSS
80196
RSS
80460
On 2022-06-22T08:18:17.000+02:00, Marcel Taeumel ***@***.***> wrote:
I cannot reproduce this issue on OS: Win32 10.0 X64 VM: 202206021410 (c9fd365) SQ: Squeak6.0beta #22056 (64 bit) — Reply to this email directly, view it on GitHub [https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/642#issuecomment-11...], or unsubscribe [https://github.com/notifications/unsubscribe-auth/ACEXJISYDDVRRBJHGY6M6XTVQK...]. You are receiving this because you are subscribed to this thread.Message ID:
***@***.***>
vm-dev@lists.squeakfoundation.org
-
edoneel -
Eliot Miranda
-
Levente Uzonyi -
Marcel Taeumel
-
smalltalking
-
tim Rowledge -
Tobias Pape