Hi

A while ago, Andreas and I spent quite some time fixing the Squeak kernel (resp. the class builder). Unfortunately, it looks like it is broken again: I downloaded the latest Squeak 3.4 gamma image and tried to add an instance variable to ClassDescription, which resulted in an error during compilation.

I didn't have the time for any debugging yet, so I don't know what is broken...

Nathanael

Nathanael,

I didn't have the time for any debugging yet, so I don't know what is broken...

I actually mentioned the problem a while ago and it's not trivial. Here's the relevant part of that message (quoted in full below):

There's a single problem left at this point which is that you can't recompile the entire class hierarchy - it will break at Metaclass. This for two reasons. For one, Metaclass class is an instance of Metaclass (e.g., "Metaclass class isMemberOf: Metaclass" yields true) which means that when we convert the _instances_ of Metaclass we also convert the _class_ of Metaclass with possibly fatal consequences (and I have _no_ idea why this has ever worked...) I think that just excluding Metaclass' class from the update process should be enough here but I'm not certain. The second problem with Metaclass is that when you reshape Metaclass (e.g., creating NewMetaclass) the meta class created for it (NewMetaclass class) needs to be an instance of NewMetaclass for the same reason. Not doing so will leave Metaclass with a broken instance layout since after reshaping Metaclass its class (Metaclass class) will still have the shape of OldMetaclass (so to speak) while having the class pointer to NewMetaclass (due to #become). I think that some creative use of a "multiple new" (e.g., create NewMetaclass class <instance of: OldMetaclass>, then create NewMetaclass <instanceOf: NewMetaclass class>, then create NewMetaclass2 class<instanceOf: NewMetaclass>, then create <NewMetaclass2>) should do the trick since the only thing that matters is that the shape of both NewMetaclass and NewMetaclass class is correct (everything else will fixed by #become later). Alternatively, a well-placed #primitiveChangeClassTo: might work miracles.

If you want to twist your mind around this problem, please do!

Cheers, - Andreas

-----Original Message----- From: Andreas Raab [mailto:Andreas.Raab@gmx.de] Sent: Saturday, September 21, 2002 2:19 AM To: 'Nathanael Schärli'; 'schaerli@iam.unibe.ch'; 'Dan@SqueakLand.org'; 'ian.piumarta@inria.fr'; 'John.Maloney@SqueakLand.org'; 'Ted@SqueakLand.org'; 'Bill Cole' Subject: RE: More ClassBuilder trouble

Hi Guys,

I actually got into thinking mode with the problem and it turns out we've all been overlooking a couple of things for way too long. First of all, some observations:

• Class and Metaclass reshapes: I think that not updating the

meta-classes when reshaping non-meta classes was a real bug. When we create a new non-meta class we already create a meta-class for it, and not updating the meta-class where we update the non-meta class just brings trouble. The worst thing is that it partly breaks the class/meta class relationship since the becomeForward: of the old into the new class will make the old meta class point to the new non-meta class, and the old non-meta class an instance of the new meta class. Clearly bad stuff, but in the grand scheme of things actually just a minor quibble ;-)

• Obsolete subclasses: What are they?! They are classes that

still have instances when somebody wanted to remove a class from the system. Since they may have instances we need to record them just in case their superclass gets reshaped at some point. However, from the point of ClassBuilder they are just subclasses. There is nothing special about them they have _exactly_ the same semantics as regular subclasses have. So far so good.

Those of you who have been around ClassBuilder will know quite a number of places where the obsolete subclasses were managed. When I got into thinking mode one of my first observations was that this is a pretty pointless exercise. After all, ClassBuilder is *reshaping* classes, not removing them, so from the point of updating an old into a new class (using become) it simply _cannot_ introduce an obsolete subclass. So why bother recording it?!

• Obsolete instances: We go to endless length in order to

make sure that there are no references to any old instances during the reshape process. E.g., we run stuff unpreemptively, use become on the instances, and, just at the point where we can be _absolutely_ certain there are no more references to it, we start remapping those (known to have no references) instances into temp instances, use #primitiveChangeClassTo: and dance all around the fact that a nice fat GC would solve the entire problem.

• 'Super send' pointers: We 'fix' all all of the methods

while recompiling a class which may send to super in order to get super-association right, we dance up and down with this stuff, although we will later do a #become: of the old into the new class which means that all of this 'fixing' is just a plain pointless exercise. Since after the #becomeForward: there can be no references to the old class whatsoever, and any of those 'fixed' association will point to the new class anyways. Which, of course, they would do equally well if they would've pointed to the old class - hey that's why we use #become, don't we?! ;-)

After I got the above straight down, I started to rip out code and try to think what it is we *really* need to do. When we mutate a class from old to new then we first mutate all of the old class' subclasses (obsolete or not) into subclasses of the new one. Fine. Let's assume we're going to do this in a way that the old ones simply no longer exists. Then we take the old class and want to update the class, its instances and possibly its metaclass into the new class, instances and meta class. So we need to do an #updateInstancesFrom: which converts the instances and will guarantee that there's no lasting pointer to any of the old instances (this is a critical invariant). At this point, all we need to do is to take the old class and meta class and just become them into the new class and meta class.

That's it. That's all we ever need. After we've done the final #become there are neither any pointers to the old instances (known from #updateInstancesFrom:) nor can there be any to the old class or meta class (since we just forward-became them). So if we do a nice fat GC after converting everything we are just done. Period. No need to do any of that complex stuff we're doing right now. Neither in reshaping nor in preparation of the new classes. Since we know we're going to #become them later on we can just create new classes, recompile them, update them.

So I thought, and I tried and instantly broke my system ;-) Then I spent a day figuring out "what is wrong with #become" (since I was absolutely certain that the above _must_ work) ... only to find that I had stupidly broken the superclass/subclass invariant which we work so hard to preserve even for obsolete subclasses. Sigh. But as I said, it _must_ work and it does, if you preserve the critical invariants.

Then I went on to remove the GC overhead since although the above _greatly_ simplifies the entire reshape process (removing four and heavily simplifying three more methods) it is somewhat slower due to the extra GC after the #become. I found two more possibilities to solve that problem besides just that fat GC after each become: We could run the entire update of the class hierarchy within a non-preemptive block. Since ClassBuilder will not do any "bad stuff" it is known that noone else can get a pointer to anything obsolete. But still, I didn't like this very much since reshaping the entire class hierarchy may run for minutes (even hours on slower machines) and not being able to interact, even when you absolutely have to, is kinda bad. So I went for the third solution which is to assume that generally no other process will do "bad stuff at high priority". The basic assumption is that no process running above normal priority will use reflection techniques to pick up, for example, #allInstances or somesuch. Then, we can just put a single well-placed GC at the end of reshaping the entire class hierarchy and everything's done.

[Note: If you think that the above 'heuristics' is not good enough, we can still switch to any of the other two solution. Or use a method that would avoid breaking the class pointers

• see below]

There's one exception of course. If we reshape the entire class hierarchy we may have obsolete Metaclasses when we come across Metaclass (since both may be in the tree, the metaclasses as well as Metaclass itself). This problem is easy to solve by adding "two extra GCs" for the single case of reshaping Metaclass itself. That was a reasonable solution for me, since how often do we reshape Metaclass in practice?! ;-)

Okay, I have attached the changes for you to play with. These changes will work for everything you might try (one exception below) IFF your system is in a consistent state to begin with (this is important - we've done so many mistakes that there's a good chance it may not). The changes simply must work - the code is trivial (compared to what was there before almost non-existent) and there's nothing that even eventually could go wrong if you "play by the heuristics".

There's a single problem left at this point which is that you can't recompile the entire class hierarchy - it will break at Metaclass. This for two reasons. For one, Metaclass class is an instance of Metaclass (e.g., "Metaclass class isMemberOf: Metaclass" yields true) which means that when we convert the _instances_ of Metaclass we also convert the _class_ of Metaclass with possibly fatal consequences (and I have _no_ idea why this has ever worked...) I think that just excluding Metaclass' class from the update process should be enough here but I'm not certain. The second problem with Metaclass is that when you reshape Metaclass (e.g., creating NewMetaclass) the meta class created for it (NewMetaclass class) needs to be an instance of NewMetaclass for the same reason. Not doing so will leave Metaclass with a broken instance layout since after reshaping Metaclass its class (Metaclass class) will still have the shape of OldMetaclass (so to speak) while having the class pointer to NewMetaclass (due to #become). I think that some creative use of a "multiple new" (e.g., create NewMetaclass class <instance of: OldMetaclass>, then create NewMetaclass <instanceOf: NewMetaclass class>, then create NewMetaclass2 class<instanceOf: NewMetaclass>, then create <NewMetaclass2>) should do the trick since the only thing that matters is that the shape of both NewMetaclass and NewMetaclass class is correct (everything else will fixed by #become later). Alternatively, a well-placed #primitiveChangeClassTo: might work miracles.

If you want to twist your mind around this problem, please do!

Ah, yes, and two more related notes: The issue with MethodChangeRecord is almost non-existent after these changes. It came mostly from not updating the meta-classes. E.g., CompiledMethods may refer to any classes but will now always refer to the "current" class if that class exists. So only truly obsolete classes (e.g., those that were removed at some point) could be held on by MCR's compiled method. Which is good, since it means that the obsolete subclasses now _really_ record only truly obsolete subclasses again.

The other note is about #become: changing class pointers. It would still be _helpful_ if #become wouldn't change class pointers since it would allow us not to worry about the need for GC at all (the reason why we need the GC is that all of the old class pointers will be broken after become). But it is not strictly required. And, it would be _equally_ helpful to make #becomeForward: automatically free the the forwarded objects, e.g., free the chunks that they used. This might be a simpler thing to do, equally effective (no broken class-pointers; and we could use one-way become on instances, too) and logically, after you're done with a becomeForward: the forwarded object is garbage and could be "implicitly collected" here.

Cheers,

• Andreas

On Tuesday 04 February 2003 05:37 am, Nathanael Schärli wrote:

http://www.iam.unibe.ch/~schaerli/smalltalk/traits/traitsPrototype. htm

There, you can also read more about our future plans and why there is no changeset available yet.

Nathanael, thanks for making this available!

You say on your page: --- Another reason is that filing in traits requires recompilation of the whole system. This is because we need to add some instance variables to ClassDescription and because we need to generate some data to fill the caches for the incremental reasoning. Unfortunately, the kernel (resp. the ClassBuilder) of Squeak is broken for at least half of the (inofficial) versions since Squeak 3.2, and it is also broken for the latest version of Squeak 3.4 (state January 31). Thus, it would not be possible to load a traits changeset into such an image even if it would exist. ---

Can you describe the ClassBuilder bug in such a way that we could fix it? Or maybe I've missed the bug report...

Thanks,

In section 11 of the Tutorial you talk about displaying a tree structure for deeply nested sub-traits. Since traits can be "flattened" conceptually, couldn't you avoid the tree structure?

Well, it is true that traits can always be flattened and therefore you don't need such a tree view in order to understand the code. However, sometimes it is very useful to look at the code in a "compositional view"

Let's say you are browsing the class ColoredCircle. By default, the browser shows it in a flat way and you don't need to worry whether it is built by using traits at all. However, the browser should indicate, that this class is actually built from traits and it should allow you to see more details by clicking on something like the (+) icon of the tree widget. Then, you can see all the traits (e.g., TCircle, TColor) the class uses. In order to be consistent, the browser should now also indicate, that the trait TCircle is built as a composition of multiple traits. Then, the user can, if he wants, again click on the (+) icon of the widget and see the details (i.e., traits TMagnitude and TGeometry). Etc.

The current browser only shows one nesting level. This means that for the class ColoredCircle, you see the traits TCircle and TColor. If you want to see more, you need to go to the definition of these subtraits.

This means that there is a "compositional view", but its depth is limited to 2. I think it should be unlimited...

Nathanael

-----Original Message----- From: squeak-dev-bounces@lists.squeakfoundation.org [mailto:squeak-dev-bounces@lists.squeakfoundation.org] On Behalf Of Doug Mair Sent: Dienstag, 4. Februar 2003 19:04 To: The general-purpose Squeak developers list Subject: Re: Traits prototype image

Very impressive work.

In section 11 of the Tutorial you talk about displaying a tree structure for deeply nested sub-traits. Since traits can be "flattened" conceptually, couldn't you avoid the tree structure?

----- Original Message ----- From: "Nathanael Schärli" n.schaerli@gmx.net To: squeak-dev@lists.squeakfoundation.org Sent: Tuesday, February 04, 2003 8:37 AM Subject: Traits prototype image

...

Hi

I finally released a first version of our trait prototype

image. More

...

information can be found at:

http://www.iam.unibe.ch/%3E

~schaerli/smalltalk/traits/traitsPrototype.htm

...

There, you can also read more about our future plans and

why there is

...

no changeset available yet.

Cheers, Nathanael

Hi,

I totally agree with you Stephane. I faced the same problems when making MetaclassTalk. The classbuilder is not clean at all. And in many kernel classes, one can find methods with hundreds of lines and hardwirings. This make any experiments with core Squeak classes heavy...

Note: My intent is not to blame people who wrote the code, but to make Squeak become better...

Noury

Stephane Ducasse a écrit:

Hi all,

I'm one of the lucky guys working with nathanael here and I can tell you what is really needed for traits :

- a clean classBuilder (we may do that)
- a real clean ***from scratch*** browser that we could use (and not 

patching again this BAD browser) - a real scanner/parser/AST that is not buggy.

Don't think that I'm too strong, I'm just right. I tried to build a pretty printer with Squeak, nathanael patched everywhere Squeak and now we are at the limits.

We are about to decide in which dialect we will develop the next generation of Traits/Modules/....This would be great to build that on top of Squeak but sometimes the quality of certain parts is really getting into our way/nerves.... So the way to go is to use Smacc and RB parseTree.

May be we should look at the ClassBuilder form VW and replace the one in Squeak. And we should definitively throw away the browser.

Stef

So if you are interested to participate contact nathanael.

On Tuesday, February 4, 2003, at 07:22 PM, Nathanael Schärli wrote:

...

...

In section 11 of the Tutorial you talk about displaying a tree structure for deeply nested sub-traits. Since traits can be "flattened" conceptually, couldn't you avoid the tree structure?

Well, it is true that traits can always be flattened and therefore you don't need such a tree view in order to understand the code. However, sometimes it is very useful to look at the code in a "compositional view"

Let's say you are browsing the class ColoredCircle. By default, the browser shows it in a flat way and you don't need to worry whether it is built by using traits at all. However, the browser should indicate, that this class is actually built from traits and it should allow you to see more details by clicking on something like the (+) icon of the tree widget. Then, you can see all the traits (e.g., TCircle, TColor) the class uses. In order to be consistent, the browser should now also indicate, that the trait TCircle is built as a composition of multiple traits. Then, the user can, if he wants, again click on the (+) icon of the widget and see the details (i.e., traits TMagnitude and TGeometry). Etc.

The current browser only shows one nesting level. This means that for the class ColoredCircle, you see the traits TCircle and TColor. If you want to see more, you need to go to the definition of these subtraits.

This means that there is a "compositional view", but its depth is limited to 2. I think it should be unlimited...

Nathanael

...

-----Original Message----- From: squeak-dev-bounces@lists.squeakfoundation.org [mailto:squeak-dev-bounces@lists.squeakfoundation.org] On Behalf Of Doug Mair Sent: Dienstag, 4. Februar 2003 19:04 To: The general-purpose Squeak developers list Subject: Re: Traits prototype image

Very impressive work.

In section 11 of the Tutorial you talk about displaying a tree structure for deeply nested sub-traits. Since traits can be "flattened" conceptually, couldn't you avoid the tree structure?

----- Original Message ----- From: "Nathanael Schärli" n.schaerli@gmx.net To: squeak-dev@lists.squeakfoundation.org Sent: Tuesday, February 04, 2003 8:37 AM Subject: Traits prototype image

...

Hi

I finally released a first version of our trait prototype

image. More

...

information can be found at:

http://www.iam.unibe.ch/%3E

~schaerli/smalltalk/traits/traitsPrototype.htm

...

...

There, you can also read more about our future plans and

why there is

...

no changeset available yet.

Cheers, Nathanael

Prof. Dr. Stéphane DUCASSE (ducasse@iam.unibe.ch) http://www.iam.unibe.ch/~ducasse/ "if you knew today was your last day on earth, what would you do different? ... especially if, by doing something different, today might not be your last day on earth" Calvin&Hobbes

Daniel Joyce wrote: [...]

Squeak is good, but I find the cruft really gets in the way of a lot of things. I've started using PHP lately, and I really like it for webpage scripting. It's clean, it works, and it's fast for a scripting host. Plus conceptually, it's very smalltalky in it's object behaviour.

I'm curious about the ways in which you think PHP's objects are smalltalky. PHP's object system is one of the more painful object implementations I've used. I'm hesitating to say it, but I'm pretty sure I would rather use C++ objects than PHP objects.

Julian

Stephane Ducasse wrote:

Hi all,

I'm one of the lucky guys working with nathanael here and I can tell you what is really needed for traits :

- a clean classBuilder (we may do that)
- a real clean ***from scratch*** browser that we could use (and 

not patching again this BAD browser) - a real scanner/parser/AST that is not buggy.

Did you try the RB parser and AST? I found it very usable on several occasions.

Cheers, Hans-Martin

On Friday 28 November 2003 08:26 pm, Eric F. Fagan wrote:

Thanks for the attachment, but I can't open it, even with the latest versions of WinZip. Whata would you suggest?

It's not an archive, so WinZip won't open it.

Anyway, you can just load it into Squeak. Select it in the FileList, choose "decompress to file", and the .cs will be created. Then choose that and choose "install".