Hi folks,
Having a Spur VM, I wonder how to measure heap allocations made by a process executing a particular block, from within the image. For example, let's have following block:
[ | a |
100 timesRepeat: [ a := Array new: 100 ] ].
When run, this block allocates - if I'm correct - some 100 * (<object header size> + 4*100) bytes.
Note that: * Ideally, I'd like to exclude allocations made by other threads but having allocations made by all threads would be OK too. * Block can span multiple scavenges even oldspace collects. * As shown in the example above, I need to take garbage into an account.
How to best measure this?
Thanks, Jan
Hi Jan,
On Jun 1, 2017, at 1:17 AM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Hi folks,
Having a Spur VM, I wonder how to measure heap allocations made by a process executing a particular block, from within the image. For example, let's have following block:
[ | a |
100 timesRepeat: [ a := Array new: 100 ] ].
When run, this block allocates - if I'm correct - some 100 * (<object header size> + 4*100) bytes.
Plus the block arg to timesRepeat:. And the size of an Array in 32-bits is
(slots >= 255 ifTrue: [16] ifFalse: [8]) + (slots = 0 ifTrue: [8] ifFalse: [slots + 1 * 2 // 2 * 8])
i.e. 32-bit objects are rounded up to a 64-bit boundary. Spur objects have at least one slot. Slot sizes greater than 254 require an overflow header (255 marks an overflow header).
Hence in 64-bits (slots >= 255 ifTrue: [16] ifFalse: [8]) + (slots = 0 ifTrue: [8] ifFalse: [slots * 8])
There's a primitive in Behavior that answers the byte size of an instance so you don't have to know the formula. The primitive is used in one of the failingBasicNew: methods to compute how much to grow memory by when a huge allocation fails.
Note that:
- Ideally, I'd like to exclude allocations made by other threads
Then run at high priority.
but having allocations made by all threads would be OK too.
- Block can span multiple scavenges even oldspace collects.
- As shown in the example above, I need to take garbage into an account.
How to best measure this?
What exactly do you want to measure? There is no allocation count, as that would slow down allocation. Time taken to allocate would be derived by sending timeTiRun to the above and subtracting the time for a null loop to run, and the time taken in GC.
The time taken in GC is maintained by the vm. See vmParameterAt: and surrounding. If you open the About Squeak dialog in a Squeak image this is nicely displayed in the VM Stats tab (IIRC).
Thanks, Jan
Cheers! Eliot
On Thu, 2017-06-01 at 07:52 -0700, Eliot Miranda wrote:
Hi Jan,
On Jun 1, 2017, at 1:17 AM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Hi folks,
Having a Spur VM, I wonder how to measure heap allocations made by a process executing a particular block, from within the image. For example, let's have following block:
[ | a |
100 timesRepeat: [ a := Array new: 100 ] ].
When run, this block allocates - if I'm correct - some 100 * (<object header size> + 4*100) bytes.
Plus the block arg to timesRepeat:. And the size of an Array in 32- bits is
Sure, sure. The exact number is not important for me right now.
Note that:
- Ideally, I'd like to exclude allocations made by other threads
Then run at high priority.
Good point, thanks. I always forget about that.
but having allocations made by all threads would be OK too.
- Block can span multiple scavenges even oldspace collects.
- As shown in the example above, I need to take garbage into
an account.
How to best measure this?
What exactly do you want to measure?
I want to know how much given code allocates. For example that given piece of code in total allocated 2.5GB. Say.
There is no allocation count, as that would slow down allocation.
Fair enough.
Time taken to allocate would be derived by sending timeTiRun to the above and subtracting the time for a null loop to run, and the time taken in GC.
The time taken in GC is maintained by the vm. See vmParameterAt: and surrounding. If you open the About Squeak dialog in a Squeak image this is nicely displayed in the VM Stats tab (IIRC).
It is indeed. But in this case I'm not interested in time spent in GC :-)
Thanks, Jan
You should be able do that in your image by replacing #basicNew, #basicNew: and #shallowCopy with method wrappers which would sum up the size of the allocated objects in ProcessLocalVariables before returning the new objects.
Levente
On Thu, 1 Jun 2017, Jan Vrany wrote:
On Thu, 2017-06-01 at 07:52 -0700, Eliot Miranda wrote:
Hi Jan,
On Jun 1, 2017, at 1:17 AM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Hi folks,
Having a Spur VM, I wonder how to measure heap allocations made by a process executing a particular block, from within the image. For example, let's have following block:
[ | a |
100 timesRepeat: [ a := Array new: 100 ] ].
When run, this block allocates - if I'm correct - some 100 * (<object header size> + 4*100) bytes.
Plus the block arg to timesRepeat:. And the size of an Array in 32- bits is
Sure, sure. The exact number is not important for me right now.
Note that:
- Ideally, I'd like to exclude allocations made by other threads
Then run at high priority.
Good point, thanks. I always forget about that.
but having allocations made by all threads would be OK too.
- Block can span multiple scavenges even oldspace collects.
- As shown in the example above, I need to take garbage into
an account.
How to best measure this?
What exactly do you want to measure?
I want to know how much given code allocates. For example that given piece of code in total allocated 2.5GB. Say.
There is no allocation count, as that would slow down allocation.
Fair enough.
Time taken to allocate would be derived by sending timeTiRun to the above and subtracting the time for a null loop to run, and the time taken in GC.
The time taken in GC is maintained by the vm. See vmParameterAt: and surrounding. If you open the About Squeak dialog in a Squeak image this is nicely displayed in the VM Stats tab (IIRC).
It is indeed. But in this case I'm not interested in time spent in GC :-)
Thanks, Jan
Hi Levente,
On Jun 2, 2017, at 3:59 AM, Levente Uzonyi leves@caesar.elte.hu wrote:
You should be able do that in your image by replacing #basicNew, #basicNew: and #shallowCopy with method wrappers which would sum up the size of the allocated objects in ProcessLocalVariables before returning the new objects.
This only catches explicit allocations. It misses allocations of blocks, indirection vectors, contexts MNU messages, and boxed floats and integers that the vm allocates as side effects of execution or the results of arithmetic primitives. The VM could easily account for the bytes allocated (at each scavenge measure size of eden; growth of old space is already accounted for).
Levente
On Thu, 1 Jun 2017, Jan Vrany wrote:
On Thu, 2017-06-01 at 07:52 -0700, Eliot Miranda wrote:
Hi Jan,
On Jun 1, 2017, at 1:17 AM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Hi folks,
Having a Spur VM, I wonder how to measure heap allocations made
by a process executing a particular block, from within the image. For example, let's have following block:
[
| a |
100 timesRepeat: [
a := Array new: 100] ].
When run, this block allocates - if I'm correct - some 100 *
(<object header size> + 4*100) bytes.
Plus the block arg to timesRepeat:. And the size of an Array in 32- bits is
Sure, sure. The exact number is not important for me right now.
Note that:
- Ideally, I'd like to exclude allocations made by other threads
Then run at high priority.
Good point, thanks. I always forget about that.
but having allocations made by all threads would be OK too.
- Block can span multiple scavenges even oldspace collects.
- As shown in the example above, I need to take garbage into an account.
How to best measure this?
What exactly do you want to measure?
I want to know how much given code allocates. For example that given piece of code in total allocated 2.5GB. Say.
There is no allocation count, as that would slow down allocation.
Fair enough.
Time taken to allocate would be derived by sending timeTiRun to the above and subtracting the time for a null loop to run, and the time taken in GC. The time taken in GC is maintained by the vm. See vmParameterAt: and surrounding. If you open the About Squeak dialog in a Squeak image this is nicely displayed in the VM Stats tab (IIRC).
It is indeed. But in this case I'm not interested in time spent in GC :-) Thanks, Jan
Hi Jan,
On Thu, Jun 1, 2017 at 1:17 AM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Hi folks,
Having a Spur VM, I wonder how to measure heap allocations made by a process executing a particular block, from within the image. For example, let's have following block:
[ | a |
100 timesRepeat: [ a := Array new: 100 ] ].
When run, this block allocates - if I'm correct - some 100 * (<object header size> + 4*100) bytes.
Note that:
- Ideally, I'd like to exclude allocations made by other threads but having allocations made by all threads would be OK too.
- Block can span multiple scavenges even oldspace collects.
- As shown in the example above, I need to take garbage into an account.
How to best measure this?
I just committed the code for Spur, so as soon as the CI has built VMs you should be able to do
initialAllocation := Smalltalk vmParameterAt: 34. self doMyThing. bytesAllocated := (Smalltalk vmParameterAt: 34) - initialAllocation
and/or
Smalltalk vmParameterAt: 34 put: 0. self doMyThing. bytesAllocated := Smalltalk vmParameterAt: 34
It needs testing. Note that it measures bytes allocated at the lowest level I can manage so it should include everything, including hidden allocations in the GC's marked stack and remembered tables, and the class tables. I *think* the code is not confused by heap growth and shrinkage. I'd love for anyone interested to review the code (see the VMMaker.oscog-eem.2237 commit).
Thanks, Jan
_,,,^..^,,,_ best, Eliot
Hi Eliot,
On Wed, 2017-06-07 at 18:17 -0700, Eliot Miranda wrote:
Hi Jan,
On Thu, Jun 1, 2017 at 1:17 AM, Jan Vrany jan.vrany@fit.cvut.cz wrote:
Hi folks,
Having a Spur VM, I wonder how to measure heap allocations made by a process executing a particular block, from within the image. For example, let's have following block:
[ | a |
100 timesRepeat: [ a := Array new: 100 ] ].
When run, this block allocates - if I'm correct - some 100 * (<object header size> + 4*100) bytes.
Note that: * Ideally, I'd like to exclude allocations made by other threads but having allocations made by all threads would be OK too. * Block can span multiple scavenges even oldspace collects. * As shown in the example above, I need to take garbage into an account.
How to best measure this?
I just committed the code for Spur, so as soon as the CI has built VMs you should be able to do
initialAllocation := Smalltalk vmParameterAt: 34. self doMyThing. bytesAllocated := (Smalltalk vmParameterAt: 34) - initialAllocation
and/or
Smalltalk vmParameterAt: 34 put: 0. self doMyThing. bytesAllocated := Smalltalk vmParameterAt: 34
Nice, thanks! I just tried and it seems to work fine - it does return plausible numbers for some quick and trivial tests.
It needs testing. Note that it measures bytes allocated at the lowest level I can manage so it should include everything, including hidden allocations in the GC's marked stack and remembered tables, and the class tables. I *think* the code is not confused by heap growth and shrinkage. I'd love for anyone interested to review the code (see the VMMaker.oscog-eem.2237 commit).
I'm sorry I don't feel experienced enough / familiar with VM internals to review the code. The general idea is OK - or at least it worked well enough for me in the past. But the devil is in the detail, as always.
Anyways, thanks!
Jan
Thanks, Jan
_,,,^..^,,,_ best, Eliot
vm-dev@lists.squeakfoundation.org
-
Eliot Miranda -
Jan Vrany -
Levente Uzonyi