Peter,
I also want to thank you for this link: http://www.greenteapress.com/semaphores/downey05semaphores.pdf I started to read it after David's comment about it and it is entertaining and I am learning lots.
I also plan on using it in phase 3 of my multi-threaded vm project.
Phase 1, my current phase, is to get all msg sends to be interceptable by the SqueakElib promise framework. This includes things that have been macro transformed by the Compiler, like #ifTrue:, #ifNil:, #whileTrue:, and so on. It also includes bytecodeMethods like #class and #==.
Phase 2 is to allow all primitives and bytecode methods to have a promise as an argument. Here, my plan is to stop short the primitive call and send the excapsulated primitive call to the promise(s) as part of a whenMoreResolved call. When the promise resolves, the primitive call will be made. QoS can be satisfied by joining the promise with a timer, such that if the promise does not resolve in xxx milliseconds, it will become broken and the primitive call will "fail".
Phase 3 is to make the Interpreter multithreaded, while protecting ObjectMemory with Semaphores. I have a quad-core chip and so I want 4 Interpreter threads (Vats). Only one of them can be inside of ObjectMemory at a time and that could be for purposes of allocation, mutation, or GC. It's possible that a simple mutex semaphore would suffice, initially. In this model, references to objects in other Vats will be ThreadRefs (a form of a FarRef) and msgs will be serialized to the other Vat (reassigned the VatID in the same shared ObjectMemory, or copied to a different but co-located ObjectMemory).
I don't think having a single ObjectMemory will scale to 10's of "processors", but will probably also need to be multithreaded with one per Vat. It's good from the standpoint of no shared memory. One challenge then is what if refs from 2 Vats are involved in the same primitive call. Well, memory reads don't have to be protected, unless memory can be relocated, that is. One thing at a time, I tell myself.
I have 0 experience in this area (Interpreter+ObjectMemory), but I thought it would be fun. Your link will help tremendously.
Cheers, Rob