On Wed, 17 Dec 2014, David T. Lewis wrote:
On Wed, Dec 17, 2014 at 04:30:10PM -0800, tim Rowledge wrote:
Because I need speed and efficiency more than precision. I'll be loading _billions_ of 64-bit Squeak Floats into a indexing system
I?m more than a little curious about a system that needs to handle billions of anything; that?s one honkin? great image we could be talking about!
It does sound like an interesting problem. Chris seems to be focused on exporting to an external indexing system, but I think that we should be able to manage data sets of that scale directly in a Squeak memory.
The 64-bit Spur object memory, with partitioning of object memory spaces and with improved garbage collection, is likely to make this a real practical option that can be implemented on very low cost hardware platforms.
Even now, the simple 68002 object format on an intepreter VM enables huge object memories on low cost personal computers, so to me the potential of a more efficient object memory and garbage collector seems clear. I'd like to try loading some of Chris' billions of Floats into a 68002 memory just to see how it runs (slowly I presume, but still an interesting experiment - Chris, can I try it?). I expect that if object memories of that scale work at all on the old 68002 format, then they are likely to work very well indeed on a 64-bit Spur object memory.
I think it depends on the way data is stored. One can easily avoid GC slowdowns by using byte and word objects to store data, because the garbage collector doesn't have to check their contents. I implemented an in-image cache a few years ago, which stores rendered webpages. It's 1000 times faster to access or store a webpage using this cache, than using memcached from the image. It has its limits of course, but it works great for what it's designed for.
Levente
Dave