Almost; the test case is started from the full image by sending a

message to the appropriate remote object in the minimal image, and the minimal image records the transfers with a module (somewhat in the manner of a changeset).

   Module objects from different images know how to synchronize their

respective systems by comparing their contents, so, for example, when a test case is finished, a minimal image could immediately transfer that new behavior to another minimal image (which uses another new, empty module).

Is the module system new? I didn't see anything in the December version of Spoon like that. Maybe I just don't recognize it.

It is a very good idea. A module is defined as all the code that is needed to run a test suite. To have a module, you must have tests. I like it!

-Ralph

Hi Nicolas--

Spoon is promising, but won't solve everything.

I don't think anyone said it would... yet. :) I think you're broadening the discussion a bit from what Ralph and I were talking about. We were just dealing with how to disentangle what is in the Squeak image right now. Solving this problem is useful all on its own. Clearly there are other problems to solve as well (like how to support multiple conflicting applications at the same time).

But since you raise them...

Previously, we had no good solutions to the problems you mentioned. I do think Spoon will put us in a better position to solve them. Specifically, it will help a lot to have completely unconstrained class names, to be able to transfer behavior between systems without using source code, compilation, or files, and to have multiple methods associated with any given method name.

-C

Hi Andreas--

...

I do think Spoon will put us in a better position to solve [the problem of multiple co-resident applications clashing]. Specifically, it will help a lot to have completely unconstrained class names, to be able to transfer behavior between systems without using source code, compilation, or files, and to have multiple methods associated with any given method name.

I'm curious but how exactly does this help for modularity? Since I haven't used Spoon so far I don't really know what to look for here...

(If you haven't read them already, you might find the previous Spoon progress reports[1] interesting.)

...but it'd be great if you were willing to share a few success stories, or even if there aren't any yet just your reasoning about why you think these abilities are an important step.

The main success story I have with Spoon is in reducing the size of a complex, unmodularized application (OpenAugment), without having to analyze how it was put together or what exactly its prerequisites were. I did this by using imprinting between the large development object memory where the application originated and a small deployment memory. My goal was simply to make the application small, but the resulting artifact provided a useful basis for actually understanding the architecture of the application and its dependencies.

By imprinting methods directly, immediately as they were invoked, I was assured of installing each method and its literals (particularly classes) in the proper order, and that I wouldn't waste time or space with things which were already present. You don't get this when using source code and/or static files for transferring behavior.

As for the three aspects I mentioned:

- Having unconstrained class names allows each author to name classes however they please, and makes it easier for code from different sources to coexist. (We would probably want the development tools to visually distinguish a class name appearing in source code when the name is shared by multiple classes, and provide a means for getting more information about the particular class associated with a particular occurrence of the name.)

- Being able to transfer behavior directly between systems increases delivery accuracy. A module isn't much good if, when applied to a target system, the behavior it is meant to convey is not properly installed. When the transfer medium is static files, the author effectively has to guess at what the state of the receiving system will be at installation time.

If we use direct negotiation instead, we can be assured of actually installing the correct behavior, and in such a way that the receiving system can then act as a provider to any other system. It's amazing how many current packages only work in one particular release of Squeak (and usually an unmodified one at that). With this approach, I think we're much more likely to produce modules that are usable in multiple Squeak releases, in various states of end-user modification.

The remote-messaging infrastructure surrounding the use of live module objects also enables other useful abilities, like browsing and even running the behavior associated with a module in the providing system (by opening remote browsers and other tools directly on the providing system), before ever deciding to install it locally.

- Being able to associate multiple methods with the same method signature avoids another major source of collisions (and places similar demands on the tools that unconstrained class names do).

thanks,

-C

[1] http://lists.squeakfoundation.org/pipermail/spoon

Hi Andreas--

Modularity is one of those areas where theory goes only so far and having a practical example is really important to study real-life implications of the system. For example, personally I was (well, still am ;-) somewhat hesitant whether imprinting will be good enough to capture all the corner cases; the cases that happen "one in a thousand" (like a file not being there) and may require a set of methods that's not available at runtime (e.g., after imprinting). I'm curious - how do you deal with that issue in practice?

I liken that risk to the risk one takes in any application deployment. If you don't have sufficient test case coverage, bugs will slip through to users. Spoon can help automate the creation of the more common test cases, but the burden of being "complete enough to ship" is still upon the human developer, as it always has been. People have written tools to try to deduce what *might* happen when a system runs, but so far Spoon doesn't attempt anything along those lines.

...what does Spoon do to make [unconstrained class names] work?

(Please see my response to Chris.)

Without the files at the beginning aren't we risking that we basically arrive at some "fixed point" for each system that we can't leave without breaking things horribly?

Hmm, I don't think so. Can you give a concrete example?

How do you deal with this [overloaded method signatures]?

(Please see my response to Colin.)

This is very interesting, thanks a lot!

Sure thing! Thanks for the questions.

-C

Hi Colin--

...

Being able to associate multiple methods with the same method signature avoids another major source of collisions (and places similar demands on the tools that unconstrained class names do).

Oh? That's an aspect of Spoon I was unaware of. Can you point me toward a more thorough description?

Sadly, no, not yet; I'll just have to write one here. :)

The next version of Spoon has a subclass of Symbol, called Selector. (I use a new Selector class for this so as not to disturb traditional notions of Symbol identity.) The Selector class implements multiple selector tables (similar in concept to a normal symbol table). An author can specify via the tools that a method's selector is not identical to any existing selector, and so should be in a new table.

When an author attempts to compile method source that uses a selector that appears in multiple selector tables, the system asks for disambiguation (e.g., by presenting the lead comment from the corresponding method sources). The number of selector tables the system maintains at any given moment is equal to the number of meanings held by the most-overloaded method signature in the system, and the system does reclamation as necessary (via weak references). Method dictionaries use Selectors as keys, instead of Symbols.

Again, since behavior is transferred from one system to another without relying solely on source code, this scheme is feasible.

I'm not sure how much this feature would actually get used, though. At this point it's more of a marketing-checklist thing. :)

thanks,

-C

Hi Colin--

How does Spoon maintain Selector identity across object memories? When a method is transferred from one object memory to another, how are the selectors in its literal frame interned when it arrives? I don't imagine the selectors have UUIDs.... perhaps the selector tables do?

There's no need to maintain Selector identity across object memories. The only important thing is that a Selector used in the literal frame of a sending method be the same one used as the method dictionary key for the method to which the sending method refers. The behavior for transferring methods ensures this.

thanks,

-C

Hi Colin--

Let's say I've got some GIS code, and I overload #at:put:. So 1-at:put: refers to the usual meaning for Arrays and Dictionaries, while 2-at:put: is my new meaning for plotting locations on maps.

I'll just add here that if you're actually coming up with a new conceptual meaning, from a sender's point of view, it's better (by virtue of intelligibility) to just use a differently-named selector. I'd anticipate overloaded selectors being useful for having multiple implementations of the same conceptual action (e.g., for performance reasons).

My GIS module uses both. Let's say that a class implementing 2-at:put: gets transferred...

"A" class? Don't you mean *the* class?

...and then later, a method that sends 2-at:put:. So we've got 2-at:put: going across the wire in 3 places: as a key in a method dictionary...

Note that, normally, during imprinting, when a class is defined remotely, no additions are made to the new method dictionary at that time. Methods are defined only later, as they are needed. An exception to this is when the class contains overrides; then method placeholders[1] are added.

If an involved Selector is overloaded (not interned in the default selector table), the providing system tells the requesting system this as part of installing the placeholder. The receiving system can then ensure that the Selector it uses for a method dictionary key is not interned in the default Selector table.

...in the literal frame of a compiled method, and during [passive] imprinting, a request for a missing method implementation.

Somehow, Spoon has to disambiguate between 1-at:put: and 2-at:put: as they move across the wire, so that selectors in literal frames correspond to the selectors in method dictionaries the same way they did in the development image. This is what I mean by "maintain Selector identity." How does Spoon accomplish it?

When a providing system transfers a method with an overloaded Selector in the literal frame, it uses a special literal marker to transfer the Selector when transferring the method's literals. See the class comment for the MethodLiteralTransmissionMarker class in Spoon for more about literal markers. Basically, they're used to transfer "special" method literals, like class pool associations.

When a system requests a method with an overloaded Selector, it communicates the sending method's signature as part of the request. The providing system can then find the appropriate Selector in its own memory (from the literal frame of its own copy of the sending method), and thus look up the correct method to define.

If the sending method's signature is itself overloaded, the requesting system provides a sending chain of method signatures, ending with a non-overloaded method signature. The providing system can then proceed as above. This scheme won't work if the only non-overloaded method signature in the sender graph is an unbound method, but that never happens in an imprinting situation. There is other metadata associated with methods (by modules) that I could use for resolution in the providing system, like author, but I don't think I need to.

Spoon's remote messaging protocol supports sending remote messages in the middle of sending other messages, nested arbitrarily deeply, amongst an arbitrary number of systems. For example, for system A to answer a message sent by system B, it might in turn need to send remote messages to B and C, which in turn might need to send still more messages to A. And each parameter of each message can be a proxy on a different arbitrary system.

So, if necessary, there can be a quite complex back-and-forth conversation between multiple systems in order to accomplish a particular message-send. This functionality is needed for remote debugging (it's why I wrote it), but I don't think it's needed for this overloaded-selector problem. But it's available in case we come across some pathological case I haven't thought of yet. :)

Also note that each Selector has its own version sequence (recall that I store a 15-bit version number in the trailer of each compiled method). I.e., if the 1-at:put: method has version number 1, with no 2-at:put: in existence, and I create a 2-at:put: method, the new 2-at:put: method will have version number 1, not 2.

Phew. :)

thanks,

-C

[1]

http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-May/103418.html

Hi Stef--

I have the impression that your Selector is a bit like selectorNamespace of SmallScript? David Simmons introduced a different Symbol equality.

Probably; I haven't seen the SmallScript implementation.

-C

Hi Chris--

...

...it will help a lot to have completely unconstrained class names...

I don't understand this. What do hard-references to classes in code bind to then?

That machinery is unchanged. The class objects themselves are mentioned in the literal frames of the methods in which they appear. The execution machinery doesn't care what their names are, just their object identities ("Name And Identity Are Distinct"). Any tool that displays source code can be smart about methods with literal frames that refer to classes with names shared by other classes (phew).

Such a tool can render the *source code* of such a method to cue the reader that such a class does not have a unique name. For example, the tool could use a hyperlink, which the reader could follow to find out more about the particular class object used by a method.

For example, two totally different Person classes are loaded and one code says:

myPerson := Person new

and other:

someOtherKindOfPerson := Person new

Ah, is there a conflict-checking operation that dynamically renames one of them?

No, they both get to keep their identical names. That is to say, both class objects have #Foo in their "name" slot (said slot currently defined by class Class). So, we've got two class objects; for the sake of discussion, let's say that one of them has name #Foo and identity hash 7777, and the other has name #Foo and identity hash 8888. (Note that Spoon actually uses metaclass UUID slots to track the identity of each class/metaclass pair.)

Now, for the first expression you mentioned, there is a corresponding method literal that refers to the class with identity hash 7777. For the second expression, there is a corresponding method literal that refers to the class with identity hash 8888. There is no effect on the source code (both expressions simply refer to "Person"), which makes sense since we can transfer methods between systems without using source code or compiling anything. When transferring behavior, source code is merely an annotation for the benefit of human readers (particularly given the fact that we can decompile methods at will).

The only compilation that need occur is when authors accept methods after editing. The browser can easily detect when an author attempts to compile some source which invokes a name used by multiple classes, and prompt for disambiguation. The system could, for example, provide a list of the class categories in which the "Person" classes appear.

In effect, each class in the system has its own namespace, since all class names are completely unconstrained all the time in all locations (i.e., "universally", which really just recapitulates the "universal" in "UUID" :). It will still be useful to put classes into named groups, for the purposes of organizing them, but I don't consider such groups to be namespaces. For one thing, within any such group you can have any number of classes with the same name. You could give every class in the system the same name if you really wanted to. :)

This scheme doesn't really work if behavior moves between systems by means of source code in files (fileouts), as we've done previously.

I hope that helps, thanks for asking!

-C

Hi Ralph--

I'm just getting into Spoon (hmm, I'm just learning to use Spoon, I'm getting stirred up with Spoon, ... there ought to be a good line here)...

Yeah, I like "getting a handle on Spoon". :)

Spoon currently still has the old system dictionary, but I'd like to remove it. I think we should just rewrite users of "Smalltalk at:" (probably in some automated fashion, both in situ and during method transfer). Instead of a system dictionary, I'd rather have some class take responsibility for each former non-class "global" variable, and use messages for access. Or just use shared pools for such variables. (Heh, shared pools is another topic that gets some people riled up. :)

The classes we can put in some cache collection that is known to the compiler, but it need not be keyed. Then each class has complete responsibility for its name, and we needn't worry about other objects having out-of-sync notions of any class names.

I don't think we need class or module namespaces in Spoon.

-C

Hi Ralph--

...

Spoon currently still has the old system dictionary, but I'd like to remove it. I think we should just rewrite users of "Smalltalk at:" (probably in some automated fashion, both in situ and during method transfer).

And how do you propose to do this?

It would proceed similarly to what happens when an author attempts to compile some source that uses an overloaded class name. When we encounter an occurrence of "Smalltalk at:", we can find all the classes in the system which have the given name. We don't have to use

...

allSubclasses since we'll have a cache of all classes in the system

(recall my proposed replacement for the current system dictionary).

If there is more than one class with the given name, we can ask the human to disambiguate, showing further information about the matching classes (author, module, etc.). If one of the choices is in a "kernel" module (as opposed to a third-party module), that could be the default.

I'll note here that I think "Smalltalk at:" is poor style in the first place.

...

The classes we can put in some cache collection that is known to the compiler, but it need not be keyed. Then each class has complete responsibility for its name, and we needn't worry about other objects having out-of-sync notions of any class names.

So, you want to find a class by sequential search, asking each class for its name?

Yes, because class names are dynamic. I don't think the time hit is a big deal, either. I assume you do?

Do you want to search the subclasses of Object, or ProtoObject? Or would you rather have a separate object to be this cache?

The latter (as I mentioned previously). And now that you mention ProtoObject... :) Spoon has no need for ProtoObject; in particular, Spoon's proxy class works with VM assistance and its superclass can be anything. I intend to remove ProtoObject.

If you have a separate object, how do you add objects to it when you load a module?

When you load a module, what you're actually doing is asking a new, empty, local module to synchronize itself with a remote module that has what you want (see Module>>synchronizeWith:). This effectively synchronizes the two systems which host the modules, with regard to the desired module content. During this synchronization, new classes which are defined in the local system would be added to the class cache by the local module.

thanks,

-C

Class names are not dynamic if you want source code to work. If you change the name of a class then you have to find code that refers to the class and patch it. If you only care about compiled code, and not source code, then class names are dynamic.

-Ralph

This brings up a point that has been bouncing around for a while. While Smalltalk sees all things as objects, the one thing that is not currently an object is source code. We store the original string from the user as the master copy, and derive objects from that. This is the source of the whole Class name thing, and the need for refactoring them. If we only kept the objects (with things like comments intact) and regenerated the source on need this would not be an issue. It would also allow richer literals and a few other things. It could also support auto-formatting of all source code if the user chose to do that. It would require that variable names and comments be retained in the image, but with modern machines that should not be a big deal. Done properly it would also resolve the case where different intended classes have the same name, as their references would be different. When editing the source the system could either prompt for desired class, use the pre-existing reference as a guide, or treat non-unique class names differently when generating source.

Michael

Hi Ralph--

...

When we encounter an occurrence of "Smalltalk at:", we can find all the classes in the system which have the given name. We don't have to use >>allSubclasses since we'll have a cache of all classes in the system (recall my proposed replacement for the current system dictionary).

I don't understand your replacement for the current system dictionary. It is just an unkeyed collection of classes. Why bother, when you can already ask Class for its instances, or Object for its subclasses? Why duplicate functionality?

Oh, I just assumed some uses might want faster access to that bunch of objects than by trolling the hierarchy; but if there aren't any, then great, let's discard it. :)

Class names are not dynamic if you want source code to work. If you change the name of a class then you have to find code that refers to the class and patch it.

Right, so you do that. I don't accept that as an argument for class names not being dynamic. It's like saying that objects with dependents aren't dynamic. Perhaps I'm missing something.

thanks,

-C

Hi Chris--

...

I intend to remove ProtoObject.

What does this mean...? Remove it from what?

Remove it from the minimal object memory.

Will Object still inherit from ProtoObject in a Spoon system?

No, in the minimal Spoon object memory, Object's superclass slot will refer to nil, and there will be no ProtoObject class.

If not, how will other proxies (of the non-Spoon type) work?

If they need ProtoObject, they will depend on some module that installs ProtoObject.

thanks,

-C

...

If they need ProtoObject, they will depend on some module that installs ProtoObject.

So, if they started from your minimal object memory that would be a recompile of the entire class hierarchy, right?

Strictly speaking, no, since ProtoObject doesn't add any variables. And that's convenient, since it means you don't have to load the ClassBuilder module first. :) (ClassBuilder is also absent from the minimal object memory.)

For easy-entry into the Spoon and backward compatibility, I hope that a "public" minimal object memory would include ProtoObject...

Actually, I think very few applications will really need ProtoObject. I myself will never have a need for it. It's there to support persistence, which I support in a different way with very small, live object memories which send remote messages to each other. But nothing will stop people from making their own starting-point object memories available with various modules included.

That's the beauty of taking on the creation of a minimal system: the decisions about what to leave and what to take out are pretty simple (take out everything that isn't needed for the system to start and extend itself). I get to leave it to others how urgent it is to reload the removed things. Some people will think it's very important to get Morphic reloaded, others won't, etc.

-C

Hi Craig

Le Mercredi 24 Mai 2006 06:08, Craig Latta a écrit :

Hi Nicolas--

...

Spoon is promising, but won't solve everything.

I don't think anyone said it would... yet. :) I think you're broadening the discussion a bit from what Ralph and I were talking about.

Yes, i like to push things a little further.

We were just dealing with how to disentangle what is in the Squeak image right now. Solving this problem is useful all on its own.

I agree, that is already great, and i call it promising when thinking of tools that we can build above it. Of course, all my best encouragements for it.

Clearly there are other problems to solve as well (like how to support multiple conflicting applications at the same time).

But since you raise them...

Yes, i was thinking with a programmer's point of view in a monolithic image: how to identify and resolve these conflicts?

Previously, we had no good solutions to the problems you mentioned. I do think Spoon will put us in a better position to solve them. Specifically, it will help a lot to have completely unconstrained class names, to be able to transfer behavior between systems without using source code, compilation, or files, and to have multiple methods associated with any given method name.

-C

This is a completely different and pragmatic approach: do not resolve source code conflicts, let them co-exist peacefully...

Something close to typical OS job: loading different binary executable into separated space processes, maybe with .dll/.so dynamic loading/sharing. But then we have displaced the problem at interprocess communication level... How does spoon compare to this model?

Nicolas

Hi Ralph--

Is the module system new? I didn't see anything in the December version of Spoon like that. Maybe I just don't recognize it.

The beginnings of it are there. The classes to look at are Module, Author, MethodDescription, and the others in the 'Spoon-Modules*' categories. You can browse them in either the control or working memories. The main difference between that and the latest unreleased stuff is the way new classes are defined (it no longer uses ClassBuilder).

Of course, I should just be able to cite the UUID of a module that refers to everything you should look at, but we're still bootstrapping. :)

It is a very good idea. A module is defined as all the code that is needed to run a test suite. To have a module, you must have tests. I like it!

Cool! I expect that there will be some fiddling with that delineation (to deal with conceptual overlap, etc.), but that's more or less the idea.

thanks,

-C