I have a need for an image larger than 2GB. Eliot, I thought I remember this subject coming up recently that, for Linux, the limit was increased to 3GB or 4GB or was I just dreaming? I can't seem to find the thread and my own experiements with -mmap seem to cap at 2000m..
Maybe it was a new Spur-specific capability?
Hi Chris,
it depends...
First of all, 32-bit linux only provides 3Gb of address space to user processes, the upper quarter being used to map the kernel into the user's address space. Look at the stack pointer and you'll see it starts at 0xbfffxxxx, not 0xffffxxxx. Apparently there are ways around this but I can't find good references for this. e.g. this is an unhelpful page, talking about total system address space, not per-process address space: http://en.wikipedia.org/wiki/3_GB_barrier.
Second of all, the VM shares the address space with other components in the program, especially X11 which uses a lot of address space. Running headless should allow you to gro a little bit more.
In my recent experience on linux one can indeed start Cog V3 with about 1.9Gb. Spur can grow to about 2.9Gb. The V3 memory manager requires a contiguous heap, which forces preallocation. This is a serious problem, but one that Spur solves. Spur has a segmented memory model, and so can grow the heap dynamically. But the fundamental problem is that linux only provides 3Gb per process. So until one has a 64-bit Spur heap size will be limited to 2.9Gb or there abouts on linux.
Is this clear enough?
For more, see my message announcing the availability of the 3018 VMs. From that message I wrote:
Here's my test script: | them | them := OrderedCollection new. [[them addLast: (ByteArray new: 16000000). Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush] repeat] on: OutOfMemory do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk garbageCollect]]. Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush. them := nil. Smalltalk garbageCollect. Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush
On Thu, Sep 25, 2014 at 12:55 PM, Chris Muller asqueaker@gmail.com wrote:
I have a need for an image larger than 2GB. Eliot, I thought I remember this subject coming up recently that, for Linux, the limit was increased to 3GB or 4GB or was I just dreaming? I can't seem to find the thread and my own experiements with -mmap seem to cap at 2000m..
Maybe it was a new Spur-specific capability?
On Thu, Sep 25, 2014 at 4:17 PM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Chris,
it depends...First of all, 32-bit linux only provides 3Gb of address space to user processes, the upper quarter being used to map the kernel into the user's address space. Look at the stack pointer and you'll see it starts at 0xbfffxxxx, not 0xffffxxxx. Apparently there are ways around this but I can't find good references for this. e.g. this is an unhelpful page, talking about total system address space, not per-process address space: http://en.wikipedia.org/wiki/3_GB_barrier.
Second of all, the VM shares the address space with other components in the program, especially X11 which uses a lot of address space. Running headless should allow you to gro a little bit more.
In my recent experience on linux one can indeed start Cog V3 with about 1.9Gb. Spur can grow to about 2.9Gb. The V3 memory manager requires a contiguous heap, which forces preallocation. This is a serious problem, but one that Spur solves. Spur has a segmented memory model, and so can grow the heap dynamically. But the fundamental problem is that linux only provides 3Gb per process. So until one has a 64-bit Spur heap size will be limited to 2.9Gb or there abouts on linux.
Is this clear enough?
Totally. Thanks.
For more, see my message announcing the availability of the 3018 VMs. From that message I wrote:
Here's my test script: | them | them := OrderedCollection new. [[them addLast: (ByteArray new: 16000000). Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush] repeat] on: OutOfMemory do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk garbageCollect]]. Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush. them := nil. Smalltalk garbageCollect. Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush
I have had to put memory-checking into my code and terminate "stale" resources when it hits the ceiling. Part of my memory-checking logic relies on (Smalltalk vmParameterAt: 3) to estimate available memory, but since it is returning -868260916 (that's right, --negative--) for that value, my assumptions that it would always be positive were wrong and that explains why the memory-checking is not working correctly.
Is something possibly wrong that this is negative? Am I supposed to take the absolute value of it?
On Thu, Sep 25, 2014 at 11:28:04PM -0500, Chris Muller wrote:
I have had to put memory-checking into my code and terminate "stale" resources when it hits the ceiling. Part of my memory-checking logic relies on (Smalltalk vmParameterAt: 3) to estimate available memory, but since it is returning -868260916 (that's right, --negative--) for that value, my assumptions that it would always be positive were wrong and that explains why the memory-checking is not working correctly.
Is something possibly wrong that this is negative? Am I supposed to take the absolute value of it?
It's probably a VM buglet, an unsigned address is being returned as a signed integer.
To illustrate, load package TwosComplement from SqueakMap, then:
-868260916 asRegister asUnsignedInteger ==> 3426706380
Dave
On Fri, Sep 26, 2014 at 7:08 AM, David T. Lewis lewis@mail.msen.com wrote:
On Thu, Sep 25, 2014 at 11:28:04PM -0500, Chris Muller wrote:
I have had to put memory-checking into my code and terminate "stale" resources when it hits the ceiling. Part of my memory-checking logic relies on (Smalltalk vmParameterAt: 3) to estimate available memory, but since it is returning -868260916 (that's right, --negative--) for that value, my assumptions that it would always be positive were wrong and that explains why the memory-checking is not working correctly.
Is something possibly wrong that this is negative? Am I supposed to take the absolute value of it?
It's probably a VM buglet, an unsigned address is being returned as a signed integer.
Indeed, I had installed the 3072 VM but forgot to refresh my scripts which were still pointing to a older VM. Oops!
Since then, I have not seen a negative value for that in 3072.
Thanks.
squeak-dev@lists.squeakfoundation.org
-
Chris Muller -
David T. Lewis -
Eliot Miranda