Hello Squeakers.
After studying distributed systems, distributed algorithms, parallel programming, and the associated practical work at uni, I think I'm almost ready to tackle an ambitious project:
Distributed Squeak.
Okay, it's been done before. But I'm going to do it again, and better :-). This will be my master's project (although it still has to be formally confirmed), and I've got 6 months to do it (until November). At the moment, all my ideas are a bit hand-wavy, but I'll be digging in and designing lots of things once I've finished hacking the Minix kernel (another practical at Uni).
See the homepage for the basic idea - http://dpon.sourceforge.net. More info will be added when I've managed to write it up.
Basically what I want to do is make a distributed, multi-user Squeak "image" (although it wouldn't really be an image anymore). So instead of starting up Squeak locally, you'd start up a "client" and log into a network of Squeak servers on the Internet. Objects could be migratable, remotely invocable or replicated, depending on the implementation of each distributed object. The advantages should be obvious - collaborative development for one. SqueakMap wouldn't be needed because everybody is using the same "image" concurrently. I'll quietly not mention the disadvantages :-).
I think it can be done. It's just difficult, not impossible.
If anybody knows of any related existing systems or technology not mentioned on dpon.sourceforge.net, I'd like to know. Also, feedback on whether this is actually a sane idea is greatly appreciated before I waste 6 months of my life :-).
Michael.
Hi Michael!
Michael van der Gulik mikevdg@hetnet.nl wrote: [SNIP]
Basically what I want to do is make a distributed, multi-user Squeak "image" (although it wouldn't really be an image anymore). So instead of starting up Squeak locally, you'd start up a "client" and log into a network of Squeak servers on the Internet. Objects could be migratable, remotely invocable or replicated, depending on the implementation of each distributed object. The advantages should be obvious - collaborative development for one. SqueakMap wouldn't be needed because everybody is using the same "image" concurrently. I'll quietly not mention the disadvantages :-).
I think it can be done. It's just difficult, not impossible.
If anybody knows of any related existing systems or technology not mentioned on dpon.sourceforge.net, I'd like to know. Also, feedback on
Well, if I were you I would do two things:
1. Take a hard look at Magma. 2. Talk with Stephen Pair and Craig Latta.
Magma is a GemStone kindof OODB for Squeak and does give multiple running Squeaks a common transactional object memory. It doesn't have support AFAIK for changing the "locus" (location of execution I believe it means) by using remote message sends but there are a bunch of Squeak frameworks for that.
Stephen is also building stuff in this vein and has demonstrated "remote object memories" at OOPSLA etc. He has also done modifications to the VM to better facilitate these systems (like dirty marking etc). And a bunch of other stuff too. :-)
And Craig has just recently looked through all the "remote message send"-frameworks in Squeak in his Squat project so he should be able to give you a quick rundown on what is available and how they relate to his own new networking framework he promptly decided to build. :-)
whether this is actually a sane idea is greatly appreciated before I waste 6 months of my life :-).
Well, please do look carefully at the stuff that is out there before duplicating work. But I think that combining some form of "remote message sending" with Magma would get you very far. Or perhaps something using Stephen's work.
Michael.
regards, Göran
Thanks for the feedback, all. I'll be away for Easter, but I'll be reading up and linking things to the web-site after then.
goran.hultgren@bluefish.se wrote:
Well, if I were you I would do two things:
- Take a hard look at Magma.
Will do.
- Talk with Stephen Pair and Craig Latta.
Well, if they're on this list I'm sure they'll gladly answer any call for help.
Magma is a GemStone kindof OODB for Squeak and does give multiple running Squeaks a common transactional object memory.
I haven't looked at Magma.. yet.. but if it's an OODB then it's more or less just for storage, right? That's an important part of what I want to achieve, but the more important part is getting remote messaging and object replication working. That's tricky.
Stephen is also building stuff in this vein and has demonstrated "remote object memories" at OOPSLA etc. He has also done modifications to the VM to better facilitate these systems (like dirty marking etc). And a bunch of other stuff too. :-)
Does anybody have a link for this? It's a bit hard to find.
Well, please do look carefully at the stuff that is out there before duplicating work. But I think that combining some form of "remote message sending" with Magma would get you very far. Or perhaps something using Stephen's work.
I shall.
Michael.
Michael van der Gulik mikevdg@hetnet.nl wrote: [SNIP]
- Talk with Stephen Pair and Craig Latta.
Well, if they're on this list I'm sure they'll gladly answer any call for help.
They most definitely are - but drop them a personal email just in case! :-)
Magma is a GemStone kindof OODB for Squeak and does give multiple running Squeaks a common transactional object memory.
I haven't looked at Magma.. yet.. but if it's an OODB then it's more or less just for storage, right? That's an important part of what I want to achieve, but the more important part is getting remote messaging and object replication working. That's tricky.
I think you are perhaps not really seeing what Magma is. It isn't "just for storage" - at least not IMHO.
Magma (as is GemStone) is a "shared object memory". Sure, it is also persistent - but don't get blinded by that.
What it does is that it allows multiple VMs to connect, share and modify one single object space. The objects in this object space are serialized over into the VMs on demand (using proxies etc) when they are sent messages (called "faulting"). BUT... and this is important - Magma still maintains the object identities so from the programmers point of view these objects aren't copies - they are the originals.
So what happens then when multiple VMs fault the same objects into their RAM and start modifying them? Well - you need transactions to be able to coordinate the VMs.
You wrote "object replication" above. That is *exactly* what Magma does. Anyway, just look at it and it will probably get clearer.
regards, Göran
Ahh.. Magma isn't doing it for me. The first step fails:
MagmaRepositoryController create: '/home/mikevdg/sq/myRepository.magma' root: Dictionary new.
I don't know how to cut&paste a stack dump, but it fails first with an error: "MessageNotUnderstood: binary", and the offending message send seems to be in "MaLargeCollectionOfIntegers>>create:" which looks like this:
create: fileNameString file _ (FileStream concreteStream new open: fileNameString forWrite: true) binary. self writeHeader
(FileStream concreteStream new) is a StandardFileStream, and the "binary" method seems to exist in that class.
I've got.. let me see.. a highly modified 3.6 alpha image, updated, with all sorts of things installed (Zurgle is just plain cool :-) ), running on a Debian unstable system. I've also tried it on the 3.5 image downloaded a few minutes ago from the website; same behaviour. If I keep trying to get Magma going, then I just keep getting other errors.
What is the current status of Magma (on SqueakMap)? Does it work for other people?
Michael.
goran.hultgren@bluefish.se wrote:
Well, if I were you I would do two things:
- Take a hard look at Magma.
Michael van der Gulik mikevdg@hetnet.nl wrote:
Ahh.. Magma isn't doing it for me. The first step fails:
MagmaRepositoryController create: '/home/mikevdg/sq/myRepository.magma' root: Dictionary new.
I don't know how to cut&paste a stack dump, but it fails first with an error: "MessageNotUnderstood: binary", and the offending message send seems to be in "MaLargeCollectionOfIntegers>>create:" which looks like this:
create: fileNameString file _ (FileStream concreteStream new open: fileNameString forWrite: true) binary. self writeHeader
(FileStream concreteStream new) is a StandardFileStream, and the "binary" method seems to exist in that class.
The #open:forWrite: method is probably returning nil which in turn is sent #binary - which it doesn't understand. An MNU or DNU (Message not understood, do not understand) is the common symptom when a nil value is erroneusly used. Chris could add some Exception handling code to the above but anyway:
So the problem is that the file with the given filename can not be opened for writing, possibly something with an oncorrect path or a permission problem?
I haven't tested Magma under Linux myself (but I think Chris has) but the error above has most surely nothing to do with Magma.
regards, Göran
Le Vendredi, 25 avri 2003, à 05:57 America/Montreal, goran.hultgren@bluefish.se a écrit :
Michael van der Gulik mikevdg@hetnet.nl wrote:
Ahh.. Magma isn't doing it for me. The first step fails:
MagmaRepositoryController create: '/home/mikevdg/sq/myRepository.magma' root: Dictionary new.
I don't know how to cut&paste a stack dump, but it fails first with an error: "MessageNotUnderstood: binary", and the offending message send seems to be in "MaLargeCollectionOfIntegers>>create:" which looks like this:
create: fileNameString file _ (FileStream concreteStream new open: fileNameString forWrite: true) binary. self writeHeader
(FileStream concreteStream new) is a StandardFileStream, and the "binary" method seems to exist in that class.
snip-------
I haven't tested Magma under Linux myself (but I think Chris has) but the error above has most surely nothing to do with Magma.
regards, Göran
Hi Göran, I don't know what system Michael van der Gulik use but I'm using MasOs X and I got exactly the same error than Michael.
Raymond
I ran into this as well - but I think it was because the parent directory didn't exist. Once I created the parent directory it created the file. I then launched the server and saved the image. After relaunching the image (this time from the command line so I could get multiple executables going), the image's UI was frozen. This was two weeks ago - I haven't tried since.
On Friday, April 25, 2003, at 02:50 PM, Raymond wrote:
Le Vendredi, 25 avri 2003, à 05:57 America/Montreal, goran.hultgren@bluefish.se a écrit :
Michael van der Gulik mikevdg@hetnet.nl wrote:
Ahh.. Magma isn't doing it for me. The first step fails:
MagmaRepositoryController create: '/home/mikevdg/sq/myRepository.magma' root: Dictionary new.
I don't know how to cut&paste a stack dump, but it fails first with an error: "MessageNotUnderstood: binary", and the offending message send seems to be in "MaLargeCollectionOfIntegers>>create:" which looks like this:
create: fileNameString file _ (FileStream concreteStream new open: fileNameString forWrite: true) binary. self writeHeader
(FileStream concreteStream new) is a StandardFileStream, and the "binary" method seems to exist in that class.
snip-------
I haven't tested Magma under Linux myself (but I think Chris has) but the error above has most surely nothing to do with Magma.
regards, Göran
Hi Göran, I don't know what system Michael van der Gulik use but I'm using MasOs X and I got exactly the same error than Michael.
Raymond
I found and fixed my first bug!! :-). Squeak is kind of addictive, isn't it. I'll work out how to submit things properly later.
"MaFilename>>upTo:" appends the pathDelimiter (see the source) too early. If you only specify the filename without a path, it automatically prefixes a "/" to the filename, which is wrong.
I feel good. More.. more!!!
Michael.
goran.hultgren@bluefish.se wrote:
So the problem is that the file with the given filename can not be opened for writing, possibly something with an oncorrect path or a permission problem?
Michael van der Gulik wrote:
I found and fixed my first bug!! :-). Squeak is kind of addictive, isn't it. I'll work out how to submit things properly later.
"MaFilename>>upTo:" appends the pathDelimiter (see the source) too early. If you only specify the filename without a path, it automatically prefixes a "/" to the filename, which is wrong.
Next error: a failed primitive.
Magma extends the ByteArray class a bit - maUint:at: and maUint:at:put: fail quite readily. This is because they call ByteArray>>integerAt:put:size:signed: which is a <primitive: 'primitiveFFIIntegerAtPut' module:'SqueakFFIPrims'>. And I don't have a working FFI on my system.
Luckily, the problem is easy to fix: ByteArray>>maUint:at and maUint:at:put both have handy (big-endian?) alternative implementations at the end of the function. Commenting out the first part of these methods works quite nicely.. I think. Untested, but it doesn't crash and burn this time.
A question remains: something like ByteArray>>integerAt:.... seems like fairly basic functionality. Why aren't these just "normal" primitives, or at least compiled into core Squeak instead of in the FFI (which I assume to be the Foreign Function Interface). FFI is platform dependant.
Michael.
On Tue, Jan 01, 2002 at 08:49:45PM +0100, Michael van der Gulik wrote:
Michael van der Gulik wrote:
Next error: a failed primitive.
Magma extends the ByteArray class a bit - maUint:at: and maUint:at:put: fail quite readily. This is because they call ByteArray>>integerAt:put:size:signed: which is a <primitive: 'primitiveFFIIntegerAtPut' module:'SqueakFFIPrims'>. And I don't have a working FFI on my system.
Luckily, the problem is easy to fix: ByteArray>>maUint:at and maUint:at:put both have handy (big-endian?) alternative implementations at the end of the function. Commenting out the first part of these methods works quite nicely.. I think. Untested, but it doesn't crash and burn this time.
A question remains: something like ByteArray>>integerAt:.... seems like fairly basic functionality. Why aren't these just "normal" primitives, or at least compiled into core Squeak instead of in the FFI (which I assume to be the Foreign Function Interface). FFI is platform dependant.
Good question. The ByteArray>>integerAt:put:signed: method is one of several methods in category "external access". These methods are all dependent on FFI (they are actually part of the FFI support code). This is probably clear enough to anyone who is writing something in Squeak that uses these methods, but it's not so clear to a user of the code when it does not work due to a missing FFI plugin. I don't know the "right" way to address this but perhaps changing the method category to something like "external access - FFI" would help to communicate the intention of these methods.
I suspect that better-informed Squeakers may be able to suggest solutions involving exception handling or Squeak modularization.
Dave
[closed] post didn't contain an attached [fix] and not for the image... (was about magma).
< I'm a bug-fixing machine! >
This post brought to you by the BugFixArchiveViewer, a handy tool that makes it easy to comment on proposed fixes and enhancements for Squeak. For more information, check out the Web page for the BugFixArchiveViewer project: http://minnow.cc.gatech.edu/squeak/3214
< I'm a bug-fixing machine! >
[closed] no fix for the image.
< I'm a bug-fixing machine! >
This post brought to you by the BugFixArchiveViewer, a handy tool that makes it easy to comment on proposed fixes and enhancements for Squeak. For more information, check out the Web page for the BugFixArchiveViewer project: http://minnow.cc.gatech.edu/squeak/3214
< I'm a bug-fixing machine! >
squeak-dev@lists.squeakfoundation.org
-
David T. Lewis -
goran.hultgren＠bluefish.se -
marcus＠ira.uka.de -
Michael van der Gulik -
Raymond -
tblanchard＠mac.com