Hi All,
I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally mywork image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18. changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18. image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18. changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18. image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
My behavior was different. Also, my image is a bit smaller (300mb).
I ran you check for large Bitmaps, and found 3 (all length 1924440). #chasePointers on each of them from the inspect chased them back to the inspecter. So I ran:
(Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]) anyOne chasePointers
This goes away for a long time - I got bored, interupted it, and inspected the large bitmaps again. This time, I only found 1.
After saving the image, the image size did not drop (still 326mb).
Image and VM is undoubttedly behind yours, but here it is:
Image ----- C:\Squeak\Viz Landscape 6.0 64\hmm-64bit.1.image Squeak6.0alpha latest update: #17647 Image format 68021 (64 bit)
Virtual Machine --------------- C:\Squeak\Viz Landscape 6.0 64\Squeak.exe Croquet Closure Stack VM [StackInterpreterPrimitives VMMaker.oscog-eem.2188] Win32 built on Apr 12 2017 09:46:07 GMT Compiler: 4.2.1 Compatible Clang 3.9.1 (tags/RELEASE_391/final) platform sources revision VM: 201704120850 https://github.com/OpenSmalltalk/opensmalltalk-vm.git $ Date: Wed Apr 12 10:50:48 2017 +0200 $ Plugins: 201704120850 https://github.com/OpenSmalltalk/opensmalltalk-vm.git $ StackInterpreter VMMaker.oscog-eem.2188 uuid: ff4ca601-cd05-4792-ab0d-dcdf19975239 Apr 12 2017
-cbc
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
It seems extra-odd that the FlexibleVocabularies package seems to be empty these days.
I see a tenuous connection from using the pointerchaser to #isLiteral (Object>>#shouldFollowOutboundPointers is sent from PointerFinder>>#follow:from:) but quite why that would lead to FlexibleVocabularies I have no idea. Maybe proxy things should over-ride #shouldFollowOutboundPointers ?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Objects are closer than they appear.
Hi Eliot,
It sounds like you selected 'flush cached versions and ancestry' from the Repository menu of the Monticello Working Copy browser.
That's something that you would only want to do for a production deployment image where you aren't planning to do any more development in. It saves memory on ancestry while providing a dynamic (albeit, slow) means of loading it back on the rare case that one would need to do that in the production image ever again. The PointerFinder is not sensitive enough for proxies, so using it ends up agitating them into reification, forcing the dynamic reload of ancestry like you experienced.
What you want is simply, "flush cached versions" and then you'll never have that issue.
- Chris
On Thu, Jan 18, 2018 at 5:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally my workimage reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Displaybits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
Hi Chris,
On Thu, Jan 18, 2018 at 7:20 PM, Chris Muller asqueaker@gmail.com wrote:
Hi Eliot,
It sounds like you selected 'flush cached versions and ancestry' from the Repository menu of the Monticello Working Copy browser.
That's something that you would only want to do for a production deployment image where you aren't planning to do any more development in. It saves memory on ancestry while providing a dynamic (albeit, slow) means of loading it back on the rare case that one would need to do that in the production image ever again. The PointerFinder is not sensitive enough for proxies, so using it ends up agitating them into reification, forcing the dynamic reload of ancestry like you experienced.
What you want is simply, "flush cached versions" and then you'll never have that issue.
Can you explain? How does this create the issue? Why have an item like "flush cached versions and ancestry" if its dangerous. The implication of :cached" is that the information can be rebuilt when required. The ancestry is out there in the .mcz files. I don't understand what's going on here. feels like a bad design decision.
- Chris
On Thu, Jan 18, 2018 at 5:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally mywork
image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top
two,
so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~Display
bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object
had
not understood isLiteral and from there what looked like an attempt to
turn
this stub into a real object caused FlexibleVocabularies-who.NN to start
to
download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can
we
please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
It wasn't clear to me that your bloated image issue was related to MCInfoProxy at all -- you said it showed ByteArray's and Bitmaps as the top, but the Proxy wasn't agitated until you ran the pointer-finder.
As I mentioned, the intention of the design is to allow purging the MC ancestry to recover that memory for building a smaller, tighter production image, without losing the ability to access it, transparently, if needed. Because no one would want to accidentally save a version with the ancestry gone (corrupt), right? This design requires no extra thinking, just a little extra waiting. Conceptually, that feels like a classic cache, to me. If there's a glitch in how it interacts with MC that could be improved, we should fix it. I'm curious what else was on that stack..
I agree, though, it isn't needed on the UI menu, so we should remove it. It's something one would normally only call from a deployment/build script.
Best, Chris
On Thu, Jan 18, 2018 at 9:43 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Chris,
On Thu, Jan 18, 2018 at 7:20 PM, Chris Muller asqueaker@gmail.com wrote:
Hi Eliot,
It sounds like you selected 'flush cached versions and ancestry' from the Repository menu of the Monticello Working Copy browser.
That's something that you would only want to do for a production deployment image where you aren't planning to do any more development in. It saves memory on ancestry while providing a dynamic (albeit, slow) means of loading it back on the rare case that one would need to do that in the production image ever again. The PointerFinder is not sensitive enough for proxies, so using it ends up agitating them into reification, forcing the dynamic reload of ancestry like you experienced.
What you want is simply, "flush cached versions" and then you'll never have that issue.
Can you explain? How does this create the issue? Why have an item like "flush cached versions and ancestry" if its dangerous. The implication of :cached" is that the information can be rebuilt when required. The ancestry is out there in the .mcz files. I don't understand what's going on here. feels like a bad design decision.
- Chris
On Thu, Jan 18, 2018 at 5:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally mywork image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
On 19 January 2018 at 05:30, Chris Muller ma.chris.m@gmail.com wrote:
It wasn't clear to me that your bloated image issue was related to MCInfoProxy at all -- you said it showed ByteArray's and Bitmaps as the top, but the Proxy wasn't agitated until you ran the pointer-finder.
As I mentioned, the intention of the design is to allow purging the MC ancestry to recover that memory for building a smaller, tighter production image, without losing the ability to access it, transparently, if needed. Because no one would want to accidentally save a version with the ancestry gone (corrupt), right? This design requires no extra thinking, just a little extra waiting. Conceptually, that feels like a classic cache, to me. If there's a glitch in how it interacts with MC that could be improved, we should fix it. I'm curious what else was on that stack..
I've voiced my criticism of that design before, so I'll just point out that it's exactly this unpredictable nature of proxy interactions that makes it fragile.
I agree, though, it isn't needed on the UI menu, so we should remove
it. It's something one would normally only call from a deployment/build script.
+1
If even Master Eliot is confused, we really should not expose it with one tempting click ;)
- Bert -
Best, Chris
On Thu, Jan 18, 2018 at 9:43 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Chris,
On Thu, Jan 18, 2018 at 7:20 PM, Chris Muller asqueaker@gmail.com
wrote:
Hi Eliot,
It sounds like you selected 'flush cached versions and ancestry' from the Repository menu of the Monticello Working Copy browser.
That's something that you would only want to do for a production deployment image where you aren't planning to do any more development in. It saves memory on ancestry while providing a dynamic (albeit, slow) means of loading it back on the rare case that one would need to do that in the production image ever again. The PointerFinder is not sensitive enough for proxies, so using it ends up agitating them into reification, forcing the dynamic reload of ancestry like you experienced.
What you want is simply, "flush cached versions" and then you'll never have that issue.
Can you explain? How does this create the issue? Why have an item like "flush cached versions and ancestry" if its dangerous. The implication
of
:cached" is that the information can be rebuilt when required. The
ancestry
is out there in the .mcz files. I don't understand what's going on here. feels like a bad design decision.
- Chris
On Thu, Jan 18, 2018 at 5:24 PM, Eliot Miranda <eliot.miranda@gmail.com
wrote:
Hi All,
I've been experiencing image save slowdowns recently and finallymy
work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the
top
two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~Display bits]]
I inspected the three and did a chase pointers on one of them. As I
did
that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN
appeared
I interrupted this and did a very cursory stack examination. Some
object
had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to
start
to download.
I threw away the debugger, ran the GC and suddenly all my free space
was
back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak.
Can
we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
+1 :-)
Best, Marcel Am 19.01.2018 12:51:26 schrieb Bert Freudenberg bert@freudenbergs.de: On 19 January 2018 at 05:30, Chris Muller <ma.chris.m@gmail.com [mailto:ma.chris.m@gmail.com]> wrote:
It wasn't clear to me that your bloated image issue was related to MCInfoProxy at all -- you said it showed ByteArray's and Bitmaps as the top, but the Proxy wasn't agitated until you ran the pointer-finder.
As I mentioned, the intention of the design is to allow purging the MC ancestry to recover that memory for building a smaller, tighter production image, without losing the ability to access it, transparently, if needed. Because no one would want to accidentally save a version with the ancestry gone (corrupt), right? This design requires no extra thinking, just a little extra waiting. Conceptually, that feels like a classic cache, to me. If there's a glitch in how it interacts with MC that could be improved, we should fix it. I'm curious what else was on that stack..
I've voiced my criticism of that design before, so I'll just point out that it's exactly this unpredictable nature of proxy interactions that makes it fragile.
I agree, though, it isn't needed on the UI menu, so we should remove it. It's something one would normally only call from a deployment/build script.
+1
If even Master Eliot is confused, we really should not expose it with one tempting click ;)
- Bert -
Best, Chris
On Thu, Jan 18, 2018 at 9:43 PM, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> wrote:
Hi Chris,
On Thu, Jan 18, 2018 at 7:20 PM, Chris Muller <asqueaker@gmail.com [mailto:asqueaker@gmail.com]> wrote:
Hi Eliot,
It sounds like you selected 'flush cached versions and ancestry' from the Repository menu of the Monticello Working Copy browser.
That's something that you would only want to do for a production deployment image where you aren't planning to do any more development in. It saves memory on ancestry while providing a dynamic (albeit, slow) means of loading it back on the rare case that one would need to do that in the production image ever again. The PointerFinder is not sensitive enough for proxies, so using it ends up agitating them into reification, forcing the dynamic reload of ancestry like you experienced.
What you want is simply, "flush cached versions" and then you'll never have that issue.
Can you explain? How does this create the issue? Why have an item like "flush cached versions and ancestry" if its dangerous. The implication of :cached" is that the information can be rebuilt when required. The ancestry is out there in the .mcz files. I don't understand what's going on here. feels like a bad design decision.
- Chris
On Thu, Jan 18, 2018 at 5:24 PM, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
To what unpredictability and fragility do you refer? If you're referring to the override of DNU, then you appear to be singling out this one use of it without proposing a better alternative or complaining about other uses of it as "unpredictable". Why?
Given your own record advocating for the most insidious and dangerous types of bugs that silently corrupt data (notably, the deeply flawed ideas about the Auto-Timezone). I suppose you would prefer something that simply nil's out the ancestry so that when someone saves it, they're saving a corrupt package?
That's would be a lot worse than an "unpredictable" delay.
On Fri, Jan 19, 2018 at 5:50 AM, Bert Freudenberg bert@freudenbergs.de wrote:
On 19 January 2018 at 05:30, Chris Muller ma.chris.m@gmail.com wrote:
It wasn't clear to me that your bloated image issue was related to MCInfoProxy at all -- you said it showed ByteArray's and Bitmaps as the top, but the Proxy wasn't agitated until you ran the pointer-finder.
As I mentioned, the intention of the design is to allow purging the MC ancestry to recover that memory for building a smaller, tighter production image, without losing the ability to access it, transparently, if needed. Because no one would want to accidentally save a version with the ancestry gone (corrupt), right? This design requires no extra thinking, just a little extra waiting. Conceptually, that feels like a classic cache, to me. If there's a glitch in how it interacts with MC that could be improved, we should fix it. I'm curious what else was on that stack..
I've voiced my criticism of that design before, so I'll just point out that it's exactly this unpredictable nature of proxy interactions that makes it fragile.
I agree, though, it isn't needed on the UI menu, so we should remove it. It's something one would normally only call from a deployment/build script.
+1
If even Master Eliot is confused, we really should not expose it with one tempting click ;)
- Bert -
Best, Chris
On Thu, Jan 18, 2018 at 9:43 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Chris,
On Thu, Jan 18, 2018 at 7:20 PM, Chris Muller asqueaker@gmail.com wrote:
Hi Eliot,
It sounds like you selected 'flush cached versions and ancestry' from the Repository menu of the Monticello Working Copy browser.
That's something that you would only want to do for a production deployment image where you aren't planning to do any more development in. It saves memory on ancestry while providing a dynamic (albeit, slow) means of loading it back on the rare case that one would need to do that in the production image ever again. The PointerFinder is not sensitive enough for proxies, so using it ends up agitating them into reification, forcing the dynamic reload of ancestry like you experienced.
What you want is simply, "flush cached versions" and then you'll never have that issue.
Can you explain? How does this create the issue? Why have an item like "flush cached versions and ancestry" if its dangerous. The implication of :cached" is that the information can be rebuilt when required. The ancestry is out there in the .mcz files. I don't understand what's going on here. feels like a bad design decision.
- Chris
On Thu, Jan 18, 2018 at 5:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently and finallymy work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
The problem I have with proxies *as a user* is that when you’re doing something where they help, they're great and when you do something where they surprise you and trigger some time-consuming thing they’re terrible. Finding a middle ground may be difficult.
In a case where something triggers a load of packages - which is almost always going to be pretty time-consuming, even when you have gigabit internet and the package server is in a really good mood - it would be nice to have some opportunity to stop it. In fact I think it would be nice to have a reasonably clean ‘stop this right now!’ button on the progress morph for almost any kind of long-running activity.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: SLTMDL: Shift Left, Test Mask and Dim the Lights
I'm not beholden to the "proxy" implementation, just the design-properties that it provides: 1) achieves the goal of purging / recovering the memory consumed by the ancestry objects while, 2) doing so safely and transparently, with never any intention of mutilating or dis-membering the ancestry -- because the stub is as good as the original when called upon.
Those are the _design_ properties anyway, if there is some other implementation which could achieve these properties, I would support it.
As for the time-consuming reification, I like Tim's idea of a cancel button on the Progress bar (note that Eliot DID simply use Command+dot) rather than a confirmation prompt because of the headless server situation -- a delay would be painful enough to notice, but not enough to bring the server down like a confirmation prompt sitting there...
I'll at least get it removed from the menu for now.
On Fri, Jan 19, 2018 at 12:47 PM, tim Rowledge tim@rowledge.org wrote:
The problem I have with proxies *as a user* is that when you’re doing something where they help, they're great and when you do something where they surprise you and trigger some time-consuming thing they’re terrible. Finding a middle ground may be difficult.
In a case where something triggers a load of packages - which is almost always going to be pretty time-consuming, even when you have gigabit internet and the package server is in a really good mood - it would be nice to have some opportunity to stop it. In fact I think it would be nice to have a reasonably clean ‘stop this right now!’ button on the progress morph for almost any kind of long-running activity.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: SLTMDL: Shift Left, Test Mask and Dim the Lights
Hi Chris,
On Fri, Jan 19, 2018 at 10:30 AM, Chris Muller asqueaker@gmail.com wrote:
To what unpredictability and fragility do you refer? If you're referring to the override of DNU, then you appear to be singling out this one use of it without proposing a better alternative or complaining about other uses of it as "unpredictable". Why?
I have no objections to the proxies. It was the presence of something for deployment that is on a user menu that freaked me out. In fact, if I understand it correctly flush ancestry replaces the ancestry with proxies that will auto-load the ancestry on first access? If that's so I actually have no objection. Dat is not being used. What I had misunderstood from the earlier messages was that flush cached versions and ancestry destroyed ancestry. As long as that's not true I'm happy.
I'm still alarmed that my now 54Mb image ballooned to 1.5Gb until it appears I somehow kicked off a cleanup by touching the ancestry proxies. But that's a different issue.
Given your own record advocating for the most insidious and dangerous types of bugs that silently corrupt data (notably, the deeply flawed ideas about the Auto-Timezone). I suppose you would prefer something that simply nil's out the ancestry so that when someone saves it, they're saving a corrupt package?
I've upset you. I'm sorry. The latter is what I had misunderstood the situation to be. Forgive me; when I'm debugging the VM I find myself at the limit of my capacities and when, in the course of bug fixing, I have to interact with some unfamiliar piece of the system I get frustrated.
That's would be a lot worse than an "unpredictable" delay.
I stand corrected.
On Fri, Jan 19, 2018 at 5:50 AM, Bert Freudenberg bert@freudenbergs.de wrote:
On 19 January 2018 at 05:30, Chris Muller ma.chris.m@gmail.com wrote:
It wasn't clear to me that your bloated image issue was related to MCInfoProxy at all -- you said it showed ByteArray's and Bitmaps as the top, but the Proxy wasn't agitated until you ran the pointer-finder.
As I mentioned, the intention of the design is to allow purging the MC ancestry to recover that memory for building a smaller, tighter production image, without losing the ability to access it, transparently, if needed. Because no one would want to accidentally save a version with the ancestry gone (corrupt), right? This design requires no extra thinking, just a little extra waiting. Conceptually, that feels like a classic cache, to me. If there's a glitch in how it interacts with MC that could be improved, we should fix it. I'm curious what else was on that stack..
I've voiced my criticism of that design before, so I'll just point out
that
it's exactly this unpredictable nature of proxy interactions that makes
it
fragile.
I agree, though, it isn't needed on the UI menu, so we should remove it. It's something one would normally only call from a deployment/build script.
+1
If even Master Eliot is confused, we really should not expose it with one tempting click ;)
- Bert -
Best, Chris
On Thu, Jan 18, 2018 at 9:43 PM, Eliot Miranda <eliot.miranda@gmail.com
wrote:
Hi Chris,
On Thu, Jan 18, 2018 at 7:20 PM, Chris Muller asqueaker@gmail.com wrote:
Hi Eliot,
It sounds like you selected 'flush cached versions and ancestry' from the Repository menu of the Monticello Working Copy browser.
That's something that you would only want to do for a production deployment image where you aren't planning to do any more development in. It saves memory on ancestry while providing a dynamic (albeit, slow) means of loading it back on the rare case that one would need
to
do that in the production image ever again. The PointerFinder is not sensitive enough for proxies, so using it ends up agitating them into reification, forcing the dynamic reload of ancestry like you experienced.
What you want is simply, "flush cached versions" and then you'll
never
have that issue.
Can you explain? How does this create the issue? Why have an item
like
"flush cached versions and ancestry" if its dangerous. The
implication
of :cached" is that the information can be rebuilt when required. The ancestry is out there in the .mcz files. I don't understand what's going on here. feels like a bad design decision.
- Chris
On Thu, Jan 18, 2018 at 5:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently andfinally
my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm~~
Display bits]]
I inspected the three and did a chase pointers on one of them. As
I
did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free
space
was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at
all
:-)
_,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
On Fri, Jan 19, 2018 at 3:48 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Chris,
On Fri, Jan 19, 2018 at 10:30 AM, Chris Muller asqueaker@gmail.com wrote:
To what unpredictability and fragility do you refer? If you're referring to the override of DNU, then you appear to be singling out this one use of it without proposing a better alternative or complaining about other uses of it as "unpredictable". Why?
I have no objections to the proxies. It was the presence of something for deployment that is on a user menu that freaked me out.
I just committed it's removal from the menu. I agree that it should not be on there.
In fact, if I understand it correctly flush ancestry replaces the ancestry with proxies that will auto-load the ancestry on first access?
Yes.
If that's so I actually have no objection. Dat is not being used. What I had misunderstood from the earlier messages was that flush cached versions and ancestry destroyed ancestry. As long as that's not true I'm happy.
Good. If you're happy, I'm happy. :)
I'm still alarmed that my now 54Mb image ballooned to 1.5Gb until it appears I somehow kicked off a cleanup by touching the ancestry proxies. But that's a different issue.
That does sound strange. The next time I use this, I'll keep my eye on that.
I've upset you. I'm sorry. The latter is what I had misunderstood the situation to be. Forgive me; when I'm debugging the VM I find myself at the limit of my capacities and when, in the course of bug fixing, I have to interact with some unfamiliar piece of the system I get frustrated.
No no, I was not upset, above I was being cheeky to Bert as a way to say I prefer to discuss solutions, not opinions. I'm sorry for coming across like a brillo pad. I know exactly where you're coming from about running into something unexpected when one is deep in a zone. Getting yanked out of that mental "deep stack" that took a lot to get there is.. very frustrating. Sorry, Eliot.
Best, Chris
Hi Marcel,
I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions). Would it be possible to arrange to flush all drop shadows in the current project when: - exiting a project - snapshotting ?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally mywork image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18. changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18. image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18. changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18. image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda eliot.miranda@gmail.com: Hi Marcel,
I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions). Would it be possible to arrange to flush all drop shadows in the current project when: - exiting a project - snapshotting ?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
Probably good enough might be
World allMorphsDo: [:ea | ea removeProperty: #dropShadow]
takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel
Am 20.02.2018 02:51:06 schrieb Eliot Miranda eliot.miranda@gmail.com:
Hi Marcel,
I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions). Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda <eliot.miranda@gmail.com mailto:eliot.miranda@gmail.com> wrote:
Hi All, I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look: Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion: Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]] I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download. I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-) _,,,^..^,,,_ best, Eliot-- _,,,^..^,,,_ best, Eliot
We have to consider all open projects:
Project allMorphicProjects do: [:project | project world allMorphsDo: [:morph | morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | project world submorphsDo: [:morph | morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning arning315@comcast.net: Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]: Hi Marcel,
I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions). Would it be possible to arrange to flush all drop shadows in the current project when: - exiting a project - snapshotting ?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
Hi Marcel,
On Feb 20, 2018, at 2:26 AM, Marcel Taeumel marcel.taeumel@hpi.de wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | project world allMorphsDo: [:morph | morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | project world submorphsDo: [:morph | morph removeProperty: #dropShadow]].
Surely we only need to scan all morphs in the current world. There is no point keeping drop shadows for other than the action be world so if should be easy to establish the invariant that there are only drop shadows in the current world and hence we can use Bob's approach of only visiting the morphs in the active world.
Remember that the drop shadows are all destroyed (or at least appear to be) when one does a save and quit. Bloating happens when, as I do, one repeatedly saves an image and one does not quit. Starting an image results in a system that only has drop shadows in the active world. So there is no need to establish the invariant by e.g. adding a package script, but if one wanted one could put the slow trawl through all morphs in a package script.
But you can speed up the trawl through all morphs by using
MorphExtension allInstancesDo: [:morphExtension| ...
since drop shadows are only held in MorphExtensions, and that should speed up the initial scan to void drop shadows to establish the invariant that there are no drop shadows in other than the current world.
Cheers!
Best, Marcel
Am 20.02.2018 10:57:08 schrieb Bob Arning arning315@comcast.net:
Probably good enough might be
World allMorphsDo: [:ea | ea removeProperty: #dropShadow]
takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote: Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea | ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel
Am 20.02.2018 02:51:06 schrieb Eliot Miranda eliot.miranda@gmail.com:
Hi Marcel,
I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions). Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi All,
I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot staff 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps. I found three that fit this criterion:
Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]I inspected the three and did a chase pointers on one of them. As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back. So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot staff 28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot staff 57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot staff 28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot staff 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here? There seems to be a very bad storage leak. Can we please discuss this? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
To handle all of the morphic projects, it would be better to do the cleanup when exiting the projects:
MorphicProject>>finalExitActions: enteringProject world allMorphsDo: [:ea | ea removeProperty: #dropShadow]. ...
That will leave only the current project to worry about when saving the image. Of course, you do not want to to cleam up the morphs if the current world is a ControlManager, so it might look something like this:
| world | (world := Project current world) isMorph ifTrue: [world allMorphsDo: [:ea | ea removeProperty: #dropShadow]]
Dave
On Tue, Feb 20, 2018 at 11:26:19AM +0100, Marcel Taeumel wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | ?? ??project world allMorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | ?? ??project world submorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning arning315@comcast.net: Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]: Hi Marcel,
?? ?? I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions).?? Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> wrote:
Hi All,
?? ?? I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps.?? I found three that fit this criterion:
?? ?? Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them.?? As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back.?? So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? ??57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here??? There seems to be a very bad storage leak.?? Can we please discuss this??? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best,??Eliot
--
_,,,^..^,,,_
best,??Eliot
This might also justify adding one new method to Project:
Project>>cleanUpForExit
Default implementation would be do nothing, and for MorphicProject it would clean up the drop morphs. This would be called from #finalExitActions: for ongoing cleanup, and when saving the image you would do this:
Project current cleanupForExit
Dave
On Tue, Feb 20, 2018 at 07:50:57AM -0500, David T. Lewis wrote:
To handle all of the morphic projects, it would be better to do the cleanup when exiting the projects:
MorphicProject>>finalExitActions: enteringProject world allMorphsDo: [:ea | ea removeProperty: #dropShadow]. ...
That will leave only the current project to worry about when saving the image. Of course, you do not want to to cleam up the morphs if the current world is a ControlManager, so it might look something like this:
| world | (world := Project current world) isMorph ifTrue: [world allMorphsDo: [:ea | ea removeProperty: #dropShadow]]
Dave
On Tue, Feb 20, 2018 at 11:26:19AM +0100, Marcel Taeumel wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | ?? ??project world allMorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | ?? ??project world submorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning arning315@comcast.net: Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]: Hi Marcel,
?? ?? I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions).?? Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> wrote:
Hi All,
?? ?? I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps.?? I found three that fit this criterion:
?? ?? Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them.?? As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back.?? So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? ??57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here??? There seems to be a very bad storage leak.?? Can we please discuss this??? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best,??Eliot
--
_,,,^..^,,,_
best,??Eliot
Hmm... I do not see value in cleaning up projects (to free memory) when the user just switches between them back and forth. Especially in the light of worlds-in-worlds or background projects, this seems like an unreasonable path to follow. We should not make too much assumptions about application code or user needs at this point. There could always be a "Oh, please render that background project into this file. Oh, why is it so slow at the moment..."
Cleaning up resources for save-and-quit, however, seems reasonable because it is at the environment's boundary and not some arbitrary in-image boundary.
Best, Marcel Am 20.02.2018 13:58:38 schrieb David T. Lewis lewis@mail.msen.com: This might also justify adding one new method to Project:
Project>>cleanUpForExit
Default implementation would be do nothing, and for MorphicProject it would clean up the drop morphs. This would be called from #finalExitActions: for ongoing cleanup, and when saving the image you would do this:
Project current cleanupForExit
Dave
On Tue, Feb 20, 2018 at 07:50:57AM -0500, David T. Lewis wrote:
To handle all of the morphic projects, it would be better to do the cleanup when exiting the projects:
MorphicProject>>finalExitActions: enteringProject world allMorphsDo: [:ea | ea removeProperty: #dropShadow]. ...
That will leave only the current project to worry about when saving the image. Of course, you do not want to to cleam up the morphs if the current world is a ControlManager, so it might look something like this:
| world | (world := Project current world) isMorph ifTrue: [world allMorphsDo: [:ea | ea removeProperty: #dropShadow]]
Dave
On Tue, Feb 20, 2018 at 11:26:19AM +0100, Marcel Taeumel wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | ?? ??project world allMorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | ?? ??project world submorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning : Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda [mailto:eliot.miranda@gmail.com]: Hi Marcel,
?? ?? I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions).?? Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda wrote:
Hi All,
?? ?? I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps.?? I found three that fit this criterion:
?? ?? Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them.?? As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back.?? So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? ??57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here??? There seems to be a very bad storage leak.?? Can we please discuss this??? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best,??Eliot
--
_,,,^..^,,,_
best,??Eliot
Hi Marcel,
On Feb 20, 2018, at 5:00 AM, Marcel Taeumel marcel.taeumel@hpi.de wrote:
Hmm... I do not see value in cleaning up projects (to free memory) when the user just switches between them back and forth. Especially in the light of worlds-in-worlds or background projects, this seems like an unreasonable path to follow. We should not make too much assumptions about application code or user needs at this point. There could always be a "Oh, please render that background project into this file. Oh, why is it so slow at the moment..."
Well one might want to make the drop shadow caches responsive to some overall memory budget because, as I've observed, the overhead of the drop shadows can be many megabytes. I saw an overhead that approached 1Gb!!
Cleaning up resources for save-and-quit, however, seems reasonable because it is at the environment's boundary and not some arbitrary in-image boundary.
Best, Marcel
Am 20.02.2018 13:58:38 schrieb David T. Lewis lewis@mail.msen.com:
This might also justify adding one new method to Project:
Project>>cleanUpForExit
Default implementation would be do nothing, and for MorphicProject it would clean up the drop morphs. This would be called from #finalExitActions: for ongoing cleanup, and when saving the image you would do this:
Project current cleanupForExit
Dave
On Tue, Feb 20, 2018 at 07:50:57AM -0500, David T. Lewis wrote:
To handle all of the morphic projects, it would be better to do the cleanup when exiting the projects:
MorphicProject>>finalExitActions: enteringProject world allMorphsDo: [:ea | ea removeProperty: #dropShadow]. ...
That will leave only the current project to worry about when saving the image. Of course, you do not want to to cleam up the morphs if the current world is a ControlManager, so it might look something like this:
| world | (world := Project current world) isMorph ifTrue: [world allMorphsDo: [:ea | ea removeProperty: #dropShadow]]
Dave
On Tue, Feb 20, 2018 at 11:26:19AM +0100, Marcel Taeumel wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | ?? ??project world allMorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | ?? ??project world submorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning : Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda [mailto:eliot.miranda@gmail.com]: Hi Marcel,
?? ?? I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions).?? Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda wrote:
Hi All,
?? ?? I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps.?? I found three that fit this criterion:
?? ?? Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them.?? As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back.?? So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? ??57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here??? There seems to be a very bad storage leak.?? Can we please discuss this??? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best,??Eliot
--
_,,,^..^,,,_
best,??Eliot
Hi Eliot,
okay. I added the clean-up to project switching and save-and/or-quit. All changes are in MorphicProject. See Morphic-mt.1397.
Best, Marcel
Am 20.02.2018 14:14:01 schrieb Eliot Miranda eliot.miranda@gmail.com: Hi Marcel,
On Feb 20, 2018, at 5:00 AM, Marcel Taeumel <marcel.taeumel@hpi.de [mailto:marcel.taeumel@hpi.de]> wrote:
Hmm... I do not see value in cleaning up projects (to free memory) when the user just switches between them back and forth. Especially in the light of worlds-in-worlds or background projects, this seems like an unreasonable path to follow. We should not make too much assumptions about application code or user needs at this point. There could always be a "Oh, please render that background project into this file. Oh, why is it so slow at the moment..."
Well one might want to make the drop shadow caches responsive to some overall memory budget because, as I've observed, the overhead of the drop shadows can be many megabytes. I saw an overhead that approached 1Gb!!
Cleaning up resources for save-and-quit, however, seems reasonable because it is at the environment's boundary and not some arbitrary in-image boundary.
Best, Marcel Am 20.02.2018 13:58:38 schrieb David T. Lewis <lewis@mail.msen.com [mailto:lewis@mail.msen.com]>: This might also justify adding one new method to Project:
Project>>cleanUpForExit
Default implementation would be do nothing, and for MorphicProject it would clean up the drop morphs. This would be called from #finalExitActions: for ongoing cleanup, and when saving the image you would do this:
Project current cleanupForExit
Dave
On Tue, Feb 20, 2018 at 07:50:57AM -0500, David T. Lewis wrote:
To handle all of the morphic projects, it would be better to do the cleanup when exiting the projects:
MorphicProject>>finalExitActions: enteringProject world allMorphsDo: [:ea | ea removeProperty: #dropShadow]. ...
That will leave only the current project to worry about when saving the image. Of course, you do not want to to cleam up the morphs if the current world is a ControlManager, so it might look something like this:
| world | (world := Project current world) isMorph ifTrue: [world allMorphsDo: [:ea | ea removeProperty: #dropShadow]]
Dave
On Tue, Feb 20, 2018 at 11:26:19AM +0100, Marcel Taeumel wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | ?? ??project world allMorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | ?? ??project world submorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning : Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda [mailto:eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]]: Hi Marcel,
?? ?? I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions).?? Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda wrote:
Hi All,
?? ?? I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps.?? I found three that fit this criterion:
?? ?? Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them.?? As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back.?? So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? ??57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here??? There seems to be a very bad storage leak.?? Can we please discuss this??? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best,??Eliot
--
_,,,^..^,,,_
best,??Eliot
If...
- SystemWindows are holding lots of drop shadows
- the shadow is only shown for the frontmost window
- recreating the shadow only takes around 20ms
Then SystemWindow>>#passivate seems like a nice place to keep things tidy. Works fine for me.
On 2/20/18 11:21 AM, Marcel Taeumel wrote:
Hi Eliot,
okay. I added the clean-up to project switching and save-and/or-quit. All changes are in MorphicProject. See Morphic-mt.1397.
Best, Marcel
Am 20.02.2018 14:14:01 schrieb Eliot Miranda eliot.miranda@gmail.com:
Hi Marcel,
On Feb 20, 2018, at 5:00 AM, Marcel Taeumel <marcel.taeumel@hpi.de mailto:marcel.taeumel@hpi.de> wrote:
Hmm... I do not see value in cleaning up projects (to free memory) when the user just switches between them back and forth. Especially in the light of worlds-in-worlds or background projects, this seems like an unreasonable path to follow. We should not make too much assumptions about application code or user needs at this point. There could always be a "Oh, please render that background project into this file. Oh, why is it so slow at the moment..."
Well one might want to make the drop shadow caches responsive to some overall memory budget because, as I've observed, the overhead of the drop shadows can be many megabytes. I saw an overhead that approached 1Gb!!
Cleaning up resources for save-and-quit, however, seems reasonable because it is at the environment's boundary and not some arbitrary in-image boundary.
Best, Marcel
Am 20.02.2018 13:58:38 schrieb David T. Lewis <lewis@mail.msen.com mailto:lewis@mail.msen.com>:
This might also justify adding one new method to Project:
Project>>cleanUpForExit
Default implementation would be do nothing, and for MorphicProject it would clean up the drop morphs. This would be called from #finalExitActions: for ongoing cleanup, and when saving the image you would do this:
Project current cleanupForExit
Dave
On Tue, Feb 20, 2018 at 07:50:57AM -0500, David T. Lewis wrote:
To handle all of the morphic projects, it would be better to do the cleanup when exiting the projects:
MorphicProject>>finalExitActions: enteringProject world allMorphsDo: [:ea | ea removeProperty: #dropShadow]. ...
That will leave only the current project to worry about when saving the image. Of course, you do not want to to cleam up the morphs if the current world is a ControlManager, so it might look something like this:
| world | (world := Project current world) isMorph ifTrue: [world allMorphsDo: [:ea | ea removeProperty: #dropShadow]]
Dave
On Tue, Feb 20, 2018 at 11:26:19AM +0100, Marcel Taeumel wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | ?? ??project world allMorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we
can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | ?? ??project world submorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning : Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs
takes time:
Morph allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea |??ea removeProperty:
#dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda
[mailto:eliot.miranda@gmail.com]:
Hi Marcel,
?? ?? I've finally worked out that the space overhead here
comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions).?? Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda wrote:
Hi All,
?? ?? I've been experiencing image save slowdowns recently and
finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:47
SpurWork64.changes
-rw-r--r--@ 1 eliot ??staff ?? 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03
save/SpurWork64-2018-01-18.changes
-rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03
save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were
the top two, so I looked for large Bitmaps.?? I found three that fit this criterion:
?? ?? Bitmap allInstances select: [:bm| bm size >= 1000000 and:
[bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of
them.?? As I did that suddenly
a) the inspector on the Array became empty (still an array but
zero elements)
b) the progress bar for Downloading FlexibleVocabularies-who.NN
appeared
I interrupted this and did a very cursory stack examination.
Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free
space was back.?? So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 15:17
SpurWork64.changes
-rw-r--r--@ 1 eliot ??staff ?? ??57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03
save/SpurWork64-2018-01-18.changes
-rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03
save/SpurWork64-2018-01-18.image
What is going on here??? There seems to be a very bad storage
leak.?? Can we please discuss this??? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best,??Eliot
--
_,,,^..^,,,_
best,??Eliot
If you would put it in #passivate or #lookUnfocused, then the cache would not work for moving and re-focusing windows anymore. :-) I do not have more data to support my claims here. :-(
Updating the shadow for a 2516@1378 window takes 350 milliseconds on my, rather fast, machine.
Best, Marcel Am 20.02.2018 17:59:13 schrieb Bob Arning arning315@comcast.net: If... - SystemWindows are holding lots of drop shadows - the shadow is only shown for the frontmost window - recreating the shadow only takes around 20ms Then SystemWindow>>#passivate seems like a nice place to keep things tidy. Works fine for me.
On 2/20/18 11:21 AM, Marcel Taeumel wrote:
Hi Eliot,
okay. I added the clean-up to project switching and save-and/or-quit. All changes are in MorphicProject. See Morphic-mt.1397.
Best, Marcel
Am 20.02.2018 14:14:01 schrieb Eliot Miranda eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]: Hi Marcel,
On Feb 20, 2018, at 5:00 AM, Marcel Taeumel <marcel.taeumel@hpi.de [mailto:marcel.taeumel@hpi.de]> wrote:
Hmm... I do not see value in cleaning up projects (to free memory) when the user just switches between them back and forth. Especially in the light of worlds-in-worlds or background projects, this seems like an unreasonable path to follow. We should not make too much assumptions about application code or user needs at this point. There could always be a "Oh, please render that background project into this file. Oh, why is it so slow at the moment..."
Well one might want to make the drop shadow caches responsive to some overall memory budget because, as I've observed, the overhead of the drop shadows can be many megabytes. I saw an overhead that approached 1Gb!!
Cleaning up resources for save-and-quit, however, seems reasonable because it is at the environment's boundary and not some arbitrary in-image boundary.
Best, Marcel Am 20.02.2018 13:58:38 schrieb David T. Lewis <lewis@mail.msen.com [mailto:lewis@mail.msen.com]>: This might also justify adding one new method to Project:
Project>>cleanUpForExit
Default implementation would be do nothing, and for MorphicProject it would clean up the drop morphs. This would be called from #finalExitActions: for ongoing cleanup, and when saving the image you would do this:
Project current cleanupForExit
Dave
On Tue, Feb 20, 2018 at 07:50:57AM -0500, David T. Lewis wrote:
To handle all of the morphic projects, it would be better to do the cleanup when exiting the projects:
MorphicProject>>finalExitActions: enteringProject world allMorphsDo: [:ea | ea removeProperty: #dropShadow]. ...
That will leave only the current project to worry about when saving the image. Of course, you do not want to to cleam up the morphs if the current world is a ControlManager, so it might look something like this:
| world | (world := Project current world) isMorph ifTrue: [world allMorphsDo: [:ea | ea removeProperty: #dropShadow]]
Dave
On Tue, Feb 20, 2018 at 11:26:19AM +0100, Marcel Taeumel wrote:
We have to consider all open projects:
Project allMorphicProjects do: [:project | ?? ??project world allMorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Yet, there can be hidden morphs. If we want to ignore those, we can stick with only cleaning up top-level morphs, which are usually the system windows:
Project allMorphicProjects do: [:project | ?? ??project world submorphsDo: [:morph | ?? ?? ?? morph removeProperty: #dropShadow]].
Best, Marcel Am 20.02.2018 10:57:08 schrieb Bob Arning : Probably good enough might be World allMorphsDo: [:ea | ea removeProperty: #dropShadow] takes 0 ms for me
On 2/20/18 2:46 AM, Marcel Taeumel wrote:
Hi Eliot,
sure. Removing the potential drop shadow of all kinds of morphs takes time:
Morph allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
About 3 seconds here in a quite clean image.
SystemWindow allSubInstancesDo: [:ea |??ea removeProperty: #dropShadow].
Works at 100 milliseconds.
What has the biggest effect in your image?
Best, Marcel Am 20.02.2018 02:51:06 schrieb Eliot Miranda [mailto:eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]]: Hi Marcel,
?? ?? I've finally worked out that the space overhead here comes from the dropShadow cache system in Morphs (in otherProperties in MorphExtensions).?? Would it be possible to arrange to flush all drop shadows in the current project when:
- exiting a project
- snapshotting
?
On Thu, Jan 18, 2018 at 3:24 PM, Eliot Miranda wrote:
Hi All,
?? ?? I've been experiencing image save slowdowns recently and finally my work image reached 1.%Gb and I thought I better take a look:
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:47 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? 1.6G Jan 18 12:48 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
I ran a space analysis and found that Bitmap and ByteArray were the top two, so I looked for large Bitmaps.?? I found three that fit this criterion:
?? ?? Bitmap allInstances select: [:bm| bm size >= 1000000 and: [bm ~~ Display bits]]
I inspected the three and did a chase pointers on one of them.?? As I did that suddenly a) the inspector on the Array became empty (still an array but zero elements) b) the progress bar for Downloading FlexibleVocabularies-who.NN appeared
I interrupted this and did a very cursory stack examination. Some object had not understood isLiteral and from there what looked like an attempt to turn this stub into a real object caused FlexibleVocabularies-who.NN to start to download.
I threw away the debugger, ran the GC and suddenly all my free space was back.?? So now on disc I have
Sisyphus.Cog$ ls -lh SpurWork64.* save/SpurWork64-* -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 15:17 SpurWork64.changes -rw-r--r--@ 1 eliot ??staff ?? ??57M Jan 18 15:17 SpurWork64.image -rw-r--r--@ 1 eliot ??staff ?? ??28M Jan 18 12:03 save/SpurWork64-2018-01-18.changes -rw-r--r--@ 1 eliot ??staff ?? 1.5G Jan 18 12:03 save/SpurWork64-2018-01-18.image
What is going on here??? There seems to be a very bad storage leak.?? Can we please discuss this??? This doesn't seem like healthy behaviour at all :-)
_,,,^..^,,,_
best,??Eliot
--
_,,,^..^,,,_
best,??Eliot
On 20-02-2018, at 9:06 AM, Marcel Taeumel marcel.taeumel@hpi.de wrote:
If you would put it in #passivate or #lookUnfocused, then the cache would not work for moving and re-focusing windows anymore. :-) I do not have more data to support my claims here. :-(
Updating the shadow for a 2516@1378 window takes 350 milliseconds on my, rather fast, machine.
Well therein lies the crux of the problem; why does it take so long and use so much memory? So far as I can see (on a 17650 image) the dropshadow form is the size of the entire window as expanded by the shadow offsets. That seems a bit excessive in size, build-time and display time. Surely the typical case would need 2 skinny Forms?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful random insult:- He has a tenuous grip on the obvious.
With some playing around with shadow related preferences and theme options it looks like there are some rather oddinteractions buried in there.
Turn on soft shadows from Extras-'Themes & Colors' and then pick a window. In its halo menu pick 'drop shadow'-'shadow offset' and move the indicator around. If I move it to the ~20 pixels range I see a strange mix of soft and hard shadow not to mention some 'interesting' visuals if I try to get shadows to the left of the window.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- MERCI RIEN - Thanks for nothin'.
Just for the record: Hibernate/Unhibernate takes roughly the same time as to recompute a drop show for 2546@1392. It reduces the form's bits size to about 0.004 % because most of the shadow form are zeros. Thus, the would be only a small gain to choose "hibernate" over "discard".
Not sure about all the BitBlt magic, but we could split the shadow form into multiple forms to reduce memory load. Yet, last time I checked, it took much longer so blit many forms for the shadow instead of one.
Best, Marcel Am 20.02.2018 19:19:05 schrieb tim Rowledge tim@rowledge.org: With some playing around with shadow related preferences and theme options it looks like there are some rather oddinteractions buried in there.
Turn on soft shadows from Extras-'Themes & Colors' and then pick a window. In its halo menu pick 'drop shadow'-'shadow offset' and move the indicator around. If I move it to the ~20 pixels range I see a strange mix of soft and hard shadow not to mention some 'interesting' visuals if I try to get shadows to the left of the window.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Fractured Idiom:- MERCI RIEN - Thanks for nothin'.
squeak-dev@lists.squeakfoundation.org
-
Bert Freudenberg -
Bob Arning -
Chris Cunningham -
Chris Muller -
Chris Muller -
David T. Lewis -
Eliot Miranda -
Marcel Taeumel -
tim Rowledge