Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the deliverables with 3dot9 final should be a 3.10 with all changes from 3.0.
As one who looks into squeak code for understanding I have found that the full version history really speeds tracking down bugs. Often it is the only good way to discover the programers intent. It is also important to know how many have mucked in a particular piece of code. A large number of cooks is one of the smell tests for troubled code.
So I am very concerned that 3.9 gamma might lose all this information when the sources get created anew.
It seems to me poor timing to do it at the end of the development cycle.
A better plan IMHO would be to bring out 3.9 without compressing the sources.
Then compress sources as the first (and possibly last step) of a 3.10.
If you choose to do only source compression for 3.10 you would have an adoptable squeak; and a fall back to 3.9 with the full history around for maintainence sake.
With both 3.9 and 3.10 representing the same point of development you would not have to do 3.9 full. You would do 3.10 full. And if you ran into problems you could check it against 3.9 full and get your insights that way.
That makes more sense. The numbering even suggest a major point. (Or you could call the compressed source version 4.0 and let the spoon version be 5.0).
I believe a great deal of code will rot and squeak will become a lot more fragile than it already is if you compress sources now before 3.9 is finalized.
Thank you for your attention to these concerns.
Yours in service, -- Jerome Peace
"Version numbers are cheap. It is patience that is precious."
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
On 24.07.2006, at 08:46, Peace Jerome wrote:
Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the problems is that the .changes is limited to 32 MB, and 3.9 7048 is already 25MB, which means people would very soon see the limit in normal use (e.g. when loading a bunch of packages).
Another problem is that if we remove the linefeeds in those couple of hundrets of methods and do a bit of more _ conversion (which is intended), then we even hit the 32MB limit while in 3.9alpha.
One of the deliverables with 3dot9 final should be a 3.10 with all changes from 3.0.
This is not possible because of two reasons: - the MC based process is broken, we can't load from 3.0 to get 3.9beta. Yes, bad. I am taking full responsibility. (We have the complete code history, though. Just not easily loadable) - The changes file would hit the 32MB limite before you would get a complete 39beta even if it would work in theory.
Besides keeping the source, I'm sure you would want to keep the changeset information, too. Cost: 5MB of image size for 3.9 alone.
So just for a 3.9 with all changes, all in all we are talking about 30MB that nobody needs but system developers. Do we *really* want to ship that to all the users?
The history mechanism of Squeak does not scale anyway. Why only ship the history from 3.0? why not the *complete* history? What do you do in 10 years? 7MB left in the changeset is kind of small...
The whole history mechanism of Squeak does not scale: There is no need at all to have the old versions on the local disc at all. What we need is a server that has the complete history that then can be queried from the clients, everthing else makes no sense.
For now, we should keep of course the latest beta version with the history of 3.9 available. Just that it's not called "3.9", but "3.9 dev" or something like that.
And we should get a student to work on that code history project...
Marcus
2006/7/24, Marcus Denker denker@iam.unibe.ch:
On 24.07.2006, at 08:46, Peace Jerome wrote:
Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the problems is that the .changes is limited to 32 MB, and 3.9 7048 is already 25MB, which means people would very soon see the limit in normal use (e.g. when loading a bunch of packages).
And can we extend the limit?
On 24.07.2006, at 09:41, Mathieu SUEN wrote:
2006/7/24, Marcus Denker denker@iam.unibe.ch:
On 24.07.2006, at 08:46, Peace Jerome wrote:
Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the problems is that the .changes is limited to 32 MB, and 3.9 7048 is already 25MB, which means people would very soon see the limit in normal use (e.g. when loading a bunch of packages).
And can we extend the limit?
1) Yes, but it is some work. 2) Unrealistic to do in 3.9beta 3) Even if it is extended, do we then want to ship a 50MB changes file? or a 500MB one in 20 years?
For 1): We have properties for CompiledMethods in 3.9 that could be used to store the sourcePointer as a real Integer, as a start. but then it would be nice to have compiledMethod be "real" objects (Tims vm patch from 1997...). And then the whole thing should be re-thougt taking into account the types of systems we have now. The design is from pre 1980... there sure are somet things that can be designed cleaner on todays systems.
E.g. if compressed sources are 5MB. Do we really need to store them on Disk? If yes, why not provide a general mechanism for this that could be used for other kinds of data that is not often accessed (e.g. the Unicode tables, in-image font and graphics)?
Marcus
2006/7/24, Marcus Denker denker@iam.unibe.ch:
On 24.07.2006, at 09:41, Mathieu SUEN wrote:
2006/7/24, Marcus Denker denker@iam.unibe.ch:
On 24.07.2006, at 08:46, Peace Jerome wrote:
Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the problems is that the .changes is limited to 32 MB, and 3.9 7048 is already 25MB, which means people would very soon see the limit in normal use (e.g. when loading a bunch of packages).
And can we extend the limit?
- Yes, but it is some work.
- Unrealistic to do in 3.9beta
- Even if it is extended, do we then want to ship a 50MB changes file? or a 500MB one in 20 years?
O yes I see
Hi Marcus,
since every other project (package) depends on the integrity of the source code mechanism, I hereby volunteer to get a new source code subsystem (incl. code history) done during this summer. We have no time to wait for a new (reliable, scalable, available) source code mechanism, so I intend to make resources available for this project.
Here's a draft for a new source code subsystem:
0] (digit for Wolfgang ;-) there are at least three preferences:
1] use the traditional subsystem as is
2] use Dan Ingalls Source Compression as suggested earlier
3] use a service based (network based) mechanism
I've studied the recent additions which where made to CompiledMethod w.r.t. MethodProperties, this has to be cleaned up before anything else can be done. No, I'm not a friend of storing any MetaData *in* the meta data's object. I propose to put the MethodProperties into the MethodDictionary where they belong (same method can *and will* have different properties per class = one MethodProperties in each class' method dictionary). As a consequence, today's MethodProperties would all be garbage collected (except for the few pragmas), because all they store is the selector. And CompiledMethod>>#selector is resolved by #methodClass and #keyAtValue: so #who can disappear as is intended.
After that, the *new* source code information can be put into the new MethodProperties, which allows an easy migration because the old source code pointer won't be touched until after migration.
Comments, suggestions, critique, all appreciated.
/Klaus
On Mon, 24 Jul 2006 09:16:58 +0200, Marcus wrote:
On 24.07.2006, at 08:46, Peace Jerome wrote:
Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the problems is that the .changes is limited to 32 MB, and 3.9 7048 is already 25MB, which means people would very soon see the limit in normal use (e.g. when loading a bunch of packages).
Another problem is that if we remove the linefeeds in those couple of hundrets of methods and do a bit of more _ conversion (which is intended), then we even hit the 32MB limit while in 3.9alpha.
One of the deliverables with 3dot9 final should be a 3.10 with all changes from 3.0.
This is not possible because of two reasons:
- the MC based process is broken, we can't load from 3.0 to get 3.9beta. Yes, bad. I am taking full responsibility. (We have the complete code history, though. Just not
easily loadable)
- The changes file would hit the 32MB limite before you would get a complete 39beta even if it would work in theory.
Besides keeping the source, I'm sure you would want to keep the changeset information, too. Cost: 5MB of image size for 3.9 alone.
So just for a 3.9 with all changes, all in all we are talking about 30MB that nobody needs but system developers. Do we *really* want to ship that to all the users?
The history mechanism of Squeak does not scale anyway. Why only ship the history from 3.0? why not the *complete* history? What do you do in 10 years? 7MB left in the changeset is kind of small...
The whole history mechanism of Squeak does not scale: There is no need at all to have the old versions on the local disc at all. What we need is a server that has the complete history that then can be queried from the clients, everthing else makes no sense.
For now, we should keep of course the latest beta version with the history of 3.9 available. Just that it's not called "3.9", but "3.9 dev" or something like that.
And we should get a student to work on that code history project...
Marcus
On 24.07.2006, at 11:54, Klaus D. Witzel wrote:
Hi Marcus,
since every other project (package) depends on the integrity of the source code mechanism, I hereby volunteer to get a new source code subsystem (incl. code history) done during this summer. We have no time to wait for a new (reliable, scalable, available) source code mechanism, so I intend to make resources available for this project.
Good! But we should not wait with 3.9 for this.
Here's a draft for a new source code subsystem:
0] (digit for Wolfgang ;-) there are at least three preferences:
1] use the traditional subsystem as is
2] use Dan Ingalls Source Compression as suggested earlier
3] use a service based (network based) mechanism
I've studied the recent additions which where made to CompiledMethod w.r.t. MethodProperties, this has to be cleaned up before anything else can be done. No, I'm not a friend of storing any MetaData *in* the meta data's object.
By the selector of a Method is not "meta"! Every method should have a name, and if you want a method with another name, make a new method. Even the very idea of different names for the same method is not a good one, as the name is hard-coded in the sourcecode, too. It's a direct, non-meta property of the method object.
I propose to put the MethodProperties into the MethodDictionary where they belong (same method can *and will* have different properties per class
Methods should be only installed in one class. yes... Traits, sigh... the copying of Traits is evil... I think a better model for traits would have been to share Bytecode, not Methods. But then in Squeak these two concepts have been mangled into one for space reasons, which was a bad idea in general. (Or: It was a bad idea not to harvest Tims fix in 1998...).
= one MethodProperties in each class' method dictionary). As a consequence, today's MethodProperties would all be garbage collected (except for the few pragmas), because all they store is the selector. And CompiledMethod>>#selector is resolved by #methodClass and #keyAtValue: so #who can disappear as is intended.
After that, the *new* source code information can be put into the new MethodProperties, which allows an easy migration because the old source code pointer won't be touched until after migration.
You can do that even in the current structure. The sourcePointer is now per Method, too.
Marcus
Hi Marcus,
on Mon, 24 Jul 2006 12:11:31 +0200, you wrote:
On 24.07.2006, at 11:54, Klaus D. Witzel wrote:
Hi Marcus,
since every other project (package) depends on the integrity of the source code mechanism, I hereby volunteer to get a new source code subsystem (incl. code history) done during this summer. We have no time to wait for a new (reliable, scalable, available) source code mechanism, so I intend to make resources available for this project.
Good! But we should not wait with 3.9 for this.
O.K.
Here's a draft for a new source code subsystem:
0] (digit for Wolfgang ;-) there are at least three preferences:
1] use the traditional subsystem as is
2] use Dan Ingalls Source Compression as suggested earlier
3] use a service based (network based) mechanism
I've studied the recent additions which where made to CompiledMethod w.r.t. MethodProperties, this has to be cleaned up before anything else can be done. No, I'm not a friend of storing any MetaData *in* the meta data's object.
By the selector of a Method is not "meta"!
I'm sorry it is "meta" by tradition. Otherwise it would have been stored in the method (literals) since the 80's of the previous century. And if you cannot agree with that view:
only the VM knows about the relationship between the selector used for method lookup and the found and executed method. This *is* what meta is about, that's what I think.
Every method should have a name, and if you want a method with another name, make a new method.
No clash here: if I throw the MethodProperties oop out of CompiledMethod's literals then I can still fulfill this your requirement (and you owe me a beer ;-).
Even the very idea of different names for the same method is not a good one, as the name is hard-coded in the sourcecode, too. It's a direct, non-meta property of the method object.
Sorry (same argument as above). It is, and I have to agree, a *substantial* part of the source-to-compiled method messenger "RNA" enzymatic process. But the result cannot tell which name its gene had for it (we human beings can, same for source/compiled code, to some extent). Take as a counter example #DoItwith:with:... and the cause for having CompiledMethod>>defaultSelector.
But anyways, I do not see a clash here and it's only a matter implementation.
Having said that, was is really missing is " an ObsoleteCompiledMethod(4711) ", mantained by MethodDictionary with key := oldMethod and value := aClass -> aSelector. A new global variable ObsoleteCompiledMethods can hold, with advantage, an instance of WeakIdentityKeyDictionary for these keys and values.
I propose to put the MethodProperties into the MethodDictionary where they belong (same method can *and will* have different properties per class
Methods should be only installed in one class.
And the system should be transparent if not so. But again, it can be implemented the way you describe.
yes... Traits, sigh... the copying of Traits is evil... I think a better model for traits would have been to share Bytecode, not Methods. But then in Squeak these two concepts have been mangled into one for space reasons, which was a bad idea in general. (Or: It was a bad idea not to harvest Tims fix in 1998...).
I have no problem with this. This will (perhaps) be reviewed once people take advantage of Squeak Trait(s).
= one MethodProperties in each class' method dictionary). As a consequence, today's MethodProperties would all be garbage collected (except for the few pragmas), because all they store is the selector. And CompiledMethod>>#selector is resolved by #methodClass and #keyAtValue: so #who can disappear as is intended.
After that, the *new* source code information can be put into the new MethodProperties, which allows an easy migration because the old source code pointer won't be touched until after migration.
You can do that even in the current structure. The sourcePointer is now per Method, too.
? I was referring to the old (as is now) and, at the same time, during migration, to the then new sourcePointer, with emphasis on having both available during migration. What was it what you meant?
/Klaus
Marcus
On 24.07.2006, at 18:21, Klaus D. Witzel wrote:
By the selector of a Method is not "meta"!
I'm sorry it is "meta" by tradition. Otherwise it would have been stored in the method (literals) since the 80's of the previous century. And if you cannot agree with that view:
only the VM knows about the relationship between the selector used for method lookup and the found and executed method. This *is* what meta is about, that's what I think.
If the model that we want to have is just "execution", then this is correct. This is what we have now: A model describing classes/ methods, where the description of the classes is both for the machine and the programmer, but for methods we only model execution. What I would like to see (now that machines have become so much more powerful) is to move this model up one level and make it's main goal to be a model of the program for engineering, while keeping this model executionable.
This is why a method should know it's class (this is what a developer thinks about a method) and it's name... the whole mode of the system should be on an abstraction level of that kind of model that you would design for a re- engineering or visualisation tool...
e.g. for execution, a class needs no name, just like a method needs no name. You could have nameless "CompiledClasses". But the objects encoding classes we do have a slot for the name... Because they have not only the interpreter as the client, but the developer tools (and the developer), too. Because it makes no sense to destinguish between the two: The higher level model with name is perfectly executable, too.
It makes the tools much simpler: In 3.9, methods know their class without having to search the whole system. As soon as that was done, I started to refactor a little... it turns out that this information makes the design simpler: e.g no need to call "decompileClass: aClass selector: selector" on the compiledMethod... just call decompile on the method, no need to funnel the extra information to all the places where they are needed. For the tools, it feels much more natural if the methods know more than nothing... as then the class/method structure will be your meta model that you can use directly.
Marcus
On Mon, 24 Jul 2006 19:44:01 +0200, Marcus Denker wrote:
On 24.07.2006, at 18:21, Klaus D. Witzel wrote:
By the selector of a Method is not "meta"!
I'm sorry it is "meta" by tradition. Otherwise it would have been stored in the method (literals) since the 80's of the previous century. And if you cannot agree with that view:
only the VM knows about the relationship between the selector used for method lookup and the found and executed method. This *is* what meta is about, that's what I think.
If the model that we want to have is just "execution", then this is correct. This is what we have now: A model describing classes/ methods, where the description of the classes is both for the machine and the programmer, but for methods we only model execution. What I would like to see (now that machines have become so much more powerful) is to move this model up one level and make it's main goal to be a model of the program for engineering, while keeping this model executionable.
I'm in sync.
This is why a method should know it's class (this is what a developer thinks about a method) and it's name... the whole mode of the system should be on an abstraction level of that kind of model that you would design for a re- engineering or visualisation tool...
But by using methodClass (to which I fully agree), every method knows it class, which knows its method dictionary, which knows the corresponding selector (all a matter of implementation). No need for MethodProperties here.
And yes (and to make it clear): every method nowadays has methodClass (except perhaps those circumscribed in a comment, similiar to 'bad case' in the other thread). Since any two ~= classes cannot be the same type, so their methods. So methodClass *must* be checked for by CompiledMethod>>#= ... (which, BTW, doesn't check the selector, in any version I've seen so far ;-)
e.g. for execution, a class needs no name, just like a method needs no name. You could have nameless "CompiledClasses". But the objects encoding classes we do have a slot for the name...
Good point. This is equivalent to (Smalltalk keyAtValue: aSlot), same as for (aMethodDictionary keyAtValue: aCompiledMethod), a matter of performance at the time of invention (CompiledMethod instanceCount > (... allObjectsdo: ... isBehavior ...) size, a subject for lengthy discussions in modern times ;-)
Because they have not only the interpreter as the client, but the developer tools (and the developer), too. Because it makes no sense to destinguish between the two: The higher level model with name is perfectly executable, too.
Sure, the higher level is "just" about *organization*, *reuse* and other elaborate concepts, a typical reason for introducing execution unrelated meta data (similiar to ClassDescription).
It makes the tools much simpler: In 3.9, methods know their class without having to search the whole system. As soon as that was done, I started to refactor a little... it turns out that this information makes the design simpler: e.g no need to call "decompileClass: aClass selector: selector" on the compiledMethod... just call decompile on the method, no need to funnel the extra information to all the places where they are needed.
Yes, when I saw that it was quite a surprise how easy things can be ;-) Good reason for marking these oldies with #deprecated:.
For the tools, it feels much more natural if the methods know more than nothing... as then the class/method structure will be your meta model that you can use directly.
O.K. but the level we are talking about here is equivalent to [at least] Behavior>>#allCallsOn: why the heck should Behavior know anything about calls to symbols in CompiledMethods? It delegates.
So can CompiledMethod (delegate) when asked the queries you mentioned. It doesn't need to store anything else than Blue Book's methodClass for obtaining these answers (by delegation).
I second implementing all possible queries and oops related to CompiledMethod in CompiledMethod. Perhaps as a separator for a redesign: oops which are unrelated to CompiledMethod>>#= (unrelated to type-, function-, computational-equivalency "PCE") do not belong to CompiledMethod. A few examples for the unrelated: #categoryOfElement:, #tempNames, source-origin-server, source-origin-path, source-origin-uri, source-origin-version, ..., source-origin-name (selector) they shouted when the baby was born ;-)
/Klaus
Marcus
Klaus D. Witzel wrote:
So can CompiledMethod (delegate) when asked the queries you mentioned. It doesn't need to store anything else than Blue Book's methodClass for obtaining these answers (by delegation).
Not quite true. It is easy to see that in a situation like here:
Object compile:'foo [42]'. fooBlock := Object new foo. Object compile:'foo ^21'.
you can't (without major modifications of classes) determine the properties of "fooBlock home" (say, for source retrieval). This is a relevant pattern as I've found in the past when dealing with changes to compiled methods - while not exactly common it is also not rare enough that it can be easily ignored.
Cheers, - Andreas
On Tue, 25 Jul 2006 08:53:49 +0200, Andreas Raab wrote:
Klaus D. Witzel wrote:
So can CompiledMethod (delegate) when asked the queries you mentioned. It doesn't need to store anything else than Blue Book's methodClass for obtaining these answers (by delegation).
Not quite true. It is easy to see that in a situation like here:
Object compile:'foo [42]'. fooBlock := Object new foo. Object compile:'foo ^21'.
you can't (without major modifications of classes) determine the properties of "fooBlock home" (say, for source retrieval).
This one (and recompiles while in debugger, fileIn's, another browser, parallel to a forked process, etc) can easily be addressed by the new ObsoleteCompiledMethods dictionary which I mentioned earlier. No major change (no change at all :) to classes necessary, just a proper action during aMethodDictionary xyzAt:ifAbsent:.
I could give more [boring] details if that's wanted.
This is a relevant pattern as I've found in the past when dealing with changes to compiled methods - while not exactly common it is also not rare enough that it can be easily ignored.
Yes, neglection of these would have to be categorized under 'major-failure'.
/Klaus
Cheers,
- Andreas
Klaus D. Witzel wrote:
I could give more [boring] details if that's wanted.
Actually, if you want my attention, talk about the new source code subsystem ;-)
The method properties discussion is a red herring in my eyes - we have a functioning version today and if that needs changing or not is in the eye of the beholder. Personally (having done it both ways) I strongly prefer the current solution since it has some nice properties (like that all relevant attributes are reachable via direct references from the method and therefore garbage collection will do The Right Thing without additional help; that a generic copying mechanism doesn't have to know that some vital parts of compiled methods are stored elsewhere; that extensions can be done in one place rather than many) but things can certainly be done differently. Discussing those merits would require some more data points other than saying "yes, it can be done" (because I know it can, I have done it before ;-) and would likely mean some fairly subjective comparisons of what is "nicer" or "more understandable" or somesuch.
In light of that, I'd rather discuss an area that actually needs fixing, like the source code subsystem.
Cheers, - Andreas
On Tue, 25 Jul 2006 17:19:11 +0200, Andreas Raab wrote:
Klaus D. Witzel wrote:
I could give more [boring] details if that's wanted.
Actually, if you want my attention, talk about the new source code subsystem ;-)
So we'll take branch here, the new source code subsystem topic will get a new thread.
The method properties discussion is a red herring in my eyes - we have a functioning version today and if that needs changing or not is in the eye of the beholder. Personally (having done it both ways) I strongly prefer the current solution since it has some nice properties (like that all relevant attributes are reachable via direct references from the method and therefore garbage collection will do The Right Thing without additional help; that a generic copying mechanism doesn't have to know that some vital parts of compiled methods are stored elsewhere; that extensions can be done in one place rather than many) but things can certainly be done differently. Discussing those merits would require some more data points other than saying "yes, it can be done" (because I know it can, I have done it before ;-) and would likely mean some fairly subjective comparisons of what is "nicer" or "more understandable" or somesuch.
Good points. May I stress that my concern is the *current* *integrity* of
(nil systemNavigation allMethodsSelect: [:method | method hasNewPropertyFormat == false]) size => 0 |x y | x := y := 0. nil systemNavigation allMethodsSelect: [:method | method literals last isVariableBinding ifTrue: [x := 1 + x] ifFalse: [y := 1 + y]. false]. x @ y => 49487@0 |x y | x := y := 0. CompiledMethod allInstancesDo: [:z | z literals last isVariableBinding ifTrue: [x := 1 + x] ifFalse: [y := 1 + y]]. x@y => 49035@5
(note the discrepancy, 49487 versus 49035, 0 versus 5, relative to 3.9#7048)
As discussed with Marcus last night (1st Squeak Stammtisch in Bern, 4 Squeakers :) I'm no longer in opposition to how and where MethodProperties are stored and maintained. And if I understand your comment above correctly, that item is hereby closed.
But aCompiledMethod methodClass (aMethod literals last isVariableBinding) must be solved (and perhaps the related implementation must be reviewed) before I'm interested in going a step further into the direction of a new source code subsystem. My intention here is that there is consensus on WHO exactly tells which class a method belongs to; when can (if at all) that change; how are obsolete methods managed in the future. This must be as reliable as possible, otherwise we can forget about aCompiledMethod>>#= for use in any other (sub)system. No, it's not my intention to invent something which is ersatz (<= WordNet/2.1) for aCompiledMethod>>#=.
A nice and bad (perhaps no *that* close but nevertheless frightening) example is ChangeRecord>>#methodClass, it asks Smalltalk (regardless of obsolence)...
In other words and in a single statement: clean up the #methodClass mess in the image.
Thereafter we can introduce (and/or reuse) concepts which support a source code view on current, obsolete, historized and future CompiledMethods, a' la distributed authoring and versioning.
Please nobody misunderstand me here: propose a fix for #methodClass (obsolete methods as well) and we can immediately turn over to the more interesting things like managing the future (so to speak).
In light of that, I'd rather discuss an area that actually needs fixing, like the source code subsystem.
Me too; let's fix first things first.
/Klaus
Cheers,
- Andreas
On 26.07.2006, at 07:36, Klaus D. Witzel wrote:
On Tue, 25 Jul 2006 17:19:11 +0200, Andreas Raab wrote:
Klaus D. Witzel wrote:
I could give more [boring] details if that's wanted.
Actually, if you want my attention, talk about the new source code subsystem ;-)
So we'll take branch here, the new source code subsystem topic will get a new thread.
The method properties discussion is a red herring in my eyes - we have a functioning version today and if that needs changing or not is in the eye of the beholder. Personally (having done it both ways) I strongly prefer the current solution since it has some nice properties (like that all relevant attributes are reachable via direct references from the method and therefore garbage collection will do The Right Thing without additional help; that a generic copying mechanism doesn't have to know that some vital parts of compiled methods are stored elsewhere; that extensions can be done in one place rather than many) but things can certainly be done differently. Discussing those merits would require some more data points other than saying "yes, it can be done" (because I know it can, I have done it before ;-) and would likely mean some fairly subjective comparisons of what is "nicer" or "more understandable" or somesuch.
Good points. May I stress that my concern is the *current* *integrity* of
(nil systemNavigation allMethodsSelect: [:method | method hasNewPropertyFormat == false]) size => 0
All methods have been recompiled to use the new format (with class binding in last literal)
|x y | x := y := 0. nil systemNavigation allMethodsSelect: [:method | method literals last isVariableBinding ifTrue: [x := 1 + x] ifFalse: [y := 1 + y]. false]. x @ y => 49487@0
|x y | x := y := 0. CompiledMethod allInstancesDo: [:z | z literals last isVariableBinding ifTrue: [x := 1 + x] ifFalse: [y := 1 + y]]. x@y => 49035@5
(note the discrepancy, 49487 versus 49035,
non-locally intalled traits methods.
0 versus 5, relative to 3.9#7048)
There are some methods on the stack that have not yet finished executing... should be fixed. It's on the never-ending list of stuff on the todo.
As discussed with Marcus last night (1st Squeak Stammtisch in Bern, 4 Squeakers :) I'm no longer in opposition to how and where MethodProperties are stored and maintained. And if I understand your comment above correctly, that item is hereby closed.
But aCompiledMethod methodClass (aMethod literals last isVariableBinding) must be solved (and perhaps the related implementation must be reviewed) before I'm interested in going a step further into the direction of a new source code subsystem. My intention here is that there is consensus on WHO exactly tells which class a method belongs to; when can (if at all) that change; how are obsolete methods managed in the future. This must be as reliable as possible, otherwise we can forget about aCompiledMethod>>#= for use in any other (sub)system. No, it's not my intention to invent something which is ersatz (<= WordNet/2.1) for aCompiledMethod>>#=.
#= does not take the Class of the method into account, as any client for equality would not want that. You want to ask "which methods are the same in the system". If we take the Class into account, you will never find any equeal methods...
A nice and bad (perhaps no *that* close but nevertheless frightening) example is ChangeRecord>>#methodClass, it asks Smalltalk (regardless of obsolence)... words and in a single statement: clean up the #methodClass mess in the image.
The whole fileOut / Changes stuff os a huge huge mess. But #methodClass in ChangeRecord has not much to do with #methodClass on CompiledMethod...
Seriously, the only way to improve the situation is to improve it... we had many people telling that only perfect solutions are acceptable. Or worthy of Squeak or something. This is wrong: We need *better* solutions, not perfect ones. If we would have just made everything continously a little bit better over the last 10 years, we would have quite a cool system now.
The idea of searching for the "perfect" solution only (while letting everything rot) is the reason for a lot of what we see now.
Marcus
On Wed, 26 Jul 2006 08:48:21 +0200, Marcus Denker wrote:
On 26.07.2006, at 07:36, Klaus D. Witzel wrote:
... May I stress that my concern is the *current* *integrity* of
(nil systemNavigation allMethodsSelect: [:method | method hasNewPropertyFormat == false]) size => 0
All methods have been recompiled to use the new format (with class binding in last literal)
|x y | x := y := 0. nil systemNavigation allMethodsSelect: [:method | method literals last isVariableBinding ifTrue: [x := 1 + x] ifFalse: [y := 1 + y]. false]. x @ y => 49487@0
|x y | x := y := 0. CompiledMethod allInstancesDo: [:z | z literals last isVariableBinding ifTrue: [x := 1 + x] ifFalse: [y := 1 + y]]. x@y => 49035@5
(note the discrepancy, 49487 versus 49035,
non-locally intalled traits methods.
Aha, I already assumed this was related to traits. Thank you for the confirmation.
0 versus 5, relative to 3.9#7048)
There are some methods on the stack that have not yet finished executing... should be fixed. It's on the never-ending list of stuff on the todo.
O.K. will have a look at what can be stolen from Pavel's kernel w.r.t. re-instantiating the as yet not terminated processes.
In adition, would you please review this line of thought: if we already had the new ObsoleteMethods feature, we'd know exactly which ones (of the still executing methods) that are. Thank you for you time.
But aCompiledMethod methodClass ... This must be as reliable as possible, otherwise we can forget about aCompiledMethod>>#= for use in any other (sub)system. No, it's not my intention to invent something which is ersatz (<= WordNet/2.1) for aCompiledMethod>>#=.
#= does not take the Class of the method into account, as any client for equality would not want that.
I repeat my argument ( >= 3 postings ago): any two classes must have (by definition) ~= type, so their methods. Please confirm or refute, if an answer is possible, thank you in advance.
You want to ask "which methods are the same in the system".
Do I? Do I want to know that two methods which are defined for two ~= types (classes) both access the i-th instance variable in their local (hidden, encapsulated) state?
If we take the Class into account, you will never find any equeal methods...
Hhm. Do you suggest that we can get rid of aCompiledMethod>>#= because there is not much sensible use of it anyways (see my point on the i-th instVar of two ~= types)?
So what do we need for finding out whether or not the compiler has made two methods from exactly the same or from guaranteed different entries in the new source code subsystem (a question which cannot be answered by querying the [most likely remote] source code subsystem)? Do we need "just" meta data and never ask aCompiledMethod>>#= ?
When does a method become a new (a different) version? Because the [most likely remote] new source code subsystem dictates so <joke> http push: sourceCode newVersion: true </joke>. Or because any two aCompiledMethod>>#= tell so?
[take my apologies for the many ?'s]
A nice and bad (perhaps no *that* close but nevertheless frightening) example is ChangeRecord>>#methodClass, it asks Smalltalk (regardless of obsolence)... words and in a single statement: clean up the #methodClass mess in the image.
The whole fileOut / Changes stuff os a huge huge mess. But #methodClass in ChangeRecord has not much to do with #methodClass on CompiledMethod...
OhhKay, lame example ;-) But frightening, isn't it.
Seriously, the only way to improve the situation is to improve it... we had many people telling that only perfect solutions are acceptable.
I never had the time ( == my customers never had the time) for waiting for / or doing a perfect solution. We're in sync (I know you knew that).
Or worthy of Squeak or something. This is wrong: We need *better* solutions, not perfect ones.
Yes, better quality than quantity.
If we would have just made everything continously a little bit better over the last 10 years, we would have quite a cool system now.
The idea of searching for the "perfect" solution only (while letting everything rot) is the reason for a lot of what we see now.
...while letting everything *else* rot... yes, agreed.
/Klaus
Marcus
O.K. will have a look at what can be stolen from Pavel's kernel w.r.t. re-instantiating the as yet not terminated processes.
I think that this is important that we learned and indeed use what is good.
This discussion about method = is quite interesting (especially in presence of bytecode rewriting) please continue....
Stef
Marcus,
can you confirm that the following collection is not empty in #7048
(nil systemNavigation allMethodsSelect: [:cm | (cm methodClass methodDictionary includesIdentity: cm) not ]) asArray inspect
These methods have methodClass whose method dictionary doesn't include the method :-(
Did a #compileAll on the respective classes and now the collection is empty. Please put on the list for the next to come image. Thanks in advance.
/Klaus
On Wed, 26 Jul 2006 12:14:31 +0200, stéphane ducasse wrote:
O.K. will have a look at what can be stolen from Pavel's kernel w.r.t. re-instantiating the as yet not terminated processes.
I think that this is important that we learned and indeed use what is good.
This discussion about method = is quite interesting (especially in presence of bytecode rewriting) please continue....
Stef
On 27.07.2006, at 18:48, Klaus D. Witzel wrote:
Marcus,
can you confirm that the following collection is not empty in #7048
(nil systemNavigation allMethodsSelect: [:cm | (cm methodClass methodDictionary includesIdentity: cm) not ]) asArray inspect
These methods have methodClass whose method dictionary doesn't include the method :-(
Did a #compileAll on the respective classes and now the collection is empty. Please put on the list for the next to come image. Thanks in advance.
Ok, I will recompile them.
Marcus
Here's a shortcut for letting the COMPUTER do it:
(nil systemNavigation allMethodsSelect: [:method | (method methodClass methodDictionary includesIdentity: method) not]) do: [:cltr | cltr actualClass compileAll]
It presupposes the (already discussed) script from Andreas embraced by my nilifying (for throwing non-methodClass methods out of the image into nil-heaven):
ProcessBrowser classPool associations do: [:each | each value: nil]. WeakArray restartFinalizationProcess. MethodChangeRecord allInstancesDo: [:each | each noteNewMethod: nil]. Delay startTimerInterruptWatcher. WorldState allInstancesDo: [:each | each convertAlarms; convertStepList]. ExternalDropHandler initialize. ScrollBar initializeImagesCache. "SokobanMorph initFields." Vocabulary initialize. GradientFillStyle initPixelRampCache. Smalltalk garbageCollect. ProcessBrowser initialize.
/Klaus
On Fri, 28 Jul 2006 16:24:18 +0200, Marcus Denker wrote:
On 27.07.2006, at 18:48, Klaus D. Witzel wrote:
Marcus,
can you confirm that the following collection is not empty in #7048
(nil systemNavigation allMethodsSelect: [:cm | (cm methodClass methodDictionary includesIdentity: cm) not ]) asArray inspect
These methods have methodClass whose method dictionary doesn't include the method :-(
Did a #compileAll on the respective classes and now the collection is empty. Please put on the list for the next to come image. Thanks in advance.
Ok, I will recompile them.
Marcus
On 29.07.2006, at 13:56, Klaus D. Witzel wrote:
Here's a shortcut for letting the COMPUTER do it:
(nil systemNavigation allMethodsSelect: [:method | (method methodClass methodDictionary includesIdentity: method) not]) do: [:cltr | cltr actualClass compileAll]
It presupposes the (already discussed) script from Andreas embraced by my nilifying (for throwing non-methodClass methods out of the image into nil-heaven):
ProcessBrowser classPool associations do: [:each | each value: nil]. WeakArray restartFinalizationProcess. MethodChangeRecord allInstancesDo: [:each | each noteNewMethod: nil]. Delay startTimerInterruptWatcher. WorldState allInstancesDo: [:each | each convertAlarms; convertStepList]. ExternalDropHandler initialize. ScrollBar initializeImagesCache. "SokobanMorph initFields." Vocabulary initialize. GradientFillStyle initPixelRampCache. Smalltalk garbageCollect. ProcessBrowser initialize.
Both are in the new 7049
Marcus
Marcus Denker wrote:
There are some methods on the stack that have not yet finished executing... should be fixed. It's on the never-ending list of stuff on the todo.
Try the following - for Croquet (3.8) this fixed all methods still in use:
WeakArray restartFinalizationProcess. MethodChangeRecord allInstancesDo:[:x| x noteNewMethod: nil]. Delay startTimerInterruptWatcher. WorldState allInstancesDo:[:ws| ws convertAlarms; convertStepList]. ExternalDropHandler initialize. ScrollBar initializeImagesCache. SokobanMorph initFields. Vocabulary initialize. GradientFillStyle initPixelRampCache. Smalltalk garbageCollect. (CompiledMethod allInstances reject:[:cm| cm hasNewPropertyFormat]) inspect. Cheers, - Andreas
Thank you Andreas. After nilifying the classVars of ProcessBrowser the inspector showed an empty array :-)
/Klaus
On Wed, 26 Jul 2006 10:58:18 +0200, Andreas Raab andreas.raab@gmx.de wrote:
WeakArray restartFinalizationProcess. MethodChangeRecord allInstancesDo:[:x| x noteNewMethod: nil]. Delay startTimerInterruptWatcher. WorldState allInstancesDo:[:ws| ws convertAlarms; convertStepList]. ExternalDropHandler initialize. ScrollBar initializeImagesCache. SokobanMorph initFields. Vocabulary initialize. GradientFillStyle initPixelRampCache. Smalltalk garbageCollect. (CompiledMethod allInstances reject:[:cm| cm hasNewPropertyFormat]) inspect.
Marcus,
attached is the related test case. Would you please condition the next to come image with Andreas' *massive* know how and my humble nilifying addition, thanks in advance.
/Klaus
On Wed, 26 Jul 2006 11:16:28 +0200, Klaus D. Witzel wrote:
Thank you Andreas. After nilifying the classVars of ProcessBrowser the inspector showed an empty array :-)
/Klaus
On Wed, 26 Jul 2006 10:58:18 +0200, Andreas Raab wrote:
WeakArray restartFinalizationProcess. MethodChangeRecord allInstancesDo:[:x| x noteNewMethod: nil]. Delay startTimerInterruptWatcher. WorldState allInstancesDo:[:ws| ws convertAlarms; convertStepList]. ExternalDropHandler initialize. ScrollBar initializeImagesCache. SokobanMorph initFields. Vocabulary initialize. GradientFillStyle initPixelRampCache. Smalltalk garbageCollect. (CompiledMethod allInstances reject:[:cm| cm hasNewPropertyFormat]) inspect.
Actually, if you want my attention, talk about the new source code subsystem ;-)
The method properties discussion is a red herring in my eyes - we have a functioning version today and if that needs changing or not is in the eye of the beholder. Personally (having done it both ways) I strongly prefer the current solution since it has some nice properties (like that all relevant attributes are reachable via direct references from the method and therefore garbage collection will do The Right Thing without additional help; that a generic copying mechanism doesn't have to know that some vital parts of compiled methods are stored elsewhere; that extensions can be done in one place rather than many) but things can certainly be done differently. Discussing those merits would require some more data points other than saying "yes, it can be done" (because I know it can, I have done it before ;-) and would likely mean some fairly subjective comparisons of what is "nicer" or "more understandable" or somesuch.
In light of that, I'd rather discuss an area that actually needs fixing, like the source code subsystem.
me too ;) I would really like to understand/learn the problem of having a more parcel like system for squeak (ie having source and image (bytecode + object) chunk as a delivery system)
Klaus D. Witzel puso en su mail :
3] use a service based (network based) mechanism
+1
And again I ask if we could have a official class repository from where all could dowload
Edgar
__________________________________________________ Pregunt�. Respond�. Descubr�. Todo lo que quer�as saber, y lo que ni imaginabas, est� en Yahoo! Respuestas (Beta). �Probalo ya! http://www.yahoo.com.ar/respuestas
Comments, suggestions, critique, all appreciated.
Just one: The idea sounds great but can we keep the entirely unrelated issues of a new source code subsystem and the particular implementation of per-method properties separate? I don't see why a new source code subsystem would require a particular implementation of per-method properties and having the two issues separate allows us to discuss and assess the merits independently.
Cheers, - Andreas
Klaus D. Witzel wrote:
Hi Marcus,
since every other project (package) depends on the integrity of the source code mechanism, I hereby volunteer to get a new source code subsystem (incl. code history) done during this summer. We have no time to wait for a new (reliable, scalable, available) source code mechanism, so I intend to make resources available for this project.
Here's a draft for a new source code subsystem:
0] (digit for Wolfgang ;-) there are at least three preferences:
1] use the traditional subsystem as is
2] use Dan Ingalls Source Compression as suggested earlier
3] use a service based (network based) mechanism
I've studied the recent additions which where made to CompiledMethod w.r.t. MethodProperties, this has to be cleaned up before anything else can be done. No, I'm not a friend of storing any MetaData *in* the meta data's object. I propose to put the MethodProperties into the MethodDictionary where they belong (same method can *and will* have different properties per class = one MethodProperties in each class' method dictionary). As a consequence, today's MethodProperties would all be garbage collected (except for the few pragmas), because all they store is the selector. And CompiledMethod>>#selector is resolved by #methodClass and #keyAtValue: so #who can disappear as is intended.
After that, the *new* source code information can be put into the new MethodProperties, which allows an easy migration because the old source code pointer won't be touched until after migration.
Comments, suggestions, critique, all appreciated.
/Klaus
On Mon, 24 Jul 2006 09:16:58 +0200, Marcus wrote:
On 24.07.2006, at 08:46, Peace Jerome wrote:
Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the problems is that the .changes is limited to 32 MB, and 3.9 7048 is already 25MB, which means people would very soon see the limit in normal use (e.g. when loading a bunch of packages).
Another problem is that if we remove the linefeeds in those couple of hundrets of methods and do a bit of more _ conversion (which is intended), then we even hit the 32MB limit while in 3.9alpha.
One of the deliverables with 3dot9 final should be a 3.10 with all changes from 3.0.
This is not possible because of two reasons: - the MC based process is broken, we can't load from 3.0 to get 3.9beta. Yes, bad. I am taking full responsibility. (We have the complete code history, though. Just not easily loadable) - The changes file would hit the 32MB limite before you would get a complete 39beta even if it would work in theory.
Besides keeping the source, I'm sure you would want to keep the changeset information, too. Cost: 5MB of image size for 3.9 alone.
So just for a 3.9 with all changes, all in all we are talking about 30MB that nobody needs but system developers. Do we *really* want to ship that to all the users?
The history mechanism of Squeak does not scale anyway. Why only ship the history from 3.0? why not the *complete* history? What do you do in 10 years? 7MB left in the changeset is kind of small...
The whole history mechanism of Squeak does not scale: There is no need at all to have the old versions on the local disc at all. What we need is a server that has the complete history that then can be queried from the clients, everthing else makes no sense.
For now, we should keep of course the latest beta version with the history of 3.9 available. Just that it's not called "3.9", but "3.9 dev" or something like that.
And we should get a student to work on that code history project...
Marcus
On Mon, 24 Jul 2006 14:00:16 +0200, Andreas Raab wrote:
Comments, suggestions, critique, all appreciated.
Just one: The idea sounds great but can we keep the entirely unrelated issues of a new source code subsystem and the particular implementation of per-method properties separate? I don't see why a new source code subsystem would require a particular implementation of per-method properties and having the two issues separate allows us to discuss and assess the merits independently.
You're right, and thank you for pointing me to the independence of the two issues. As everybody else in software industry, I was suggesting a particular implementation (of new sourcePointer) instead of a sound and complete design ;-)
/Klaus
Cheers,
- Andreas
+ 1 Changes are necessary but costly so we should go stepwise.
Stef
Comments, suggestions, critique, all appreciated.
Just one: The idea sounds great but can we keep the entirely unrelated issues of a new source code subsystem and the particular implementation of per-method properties separate? I don't see why a new source code subsystem would require a particular implementation of per-method properties and having the two issues separate allows us to discuss and assess the merits independently.
Cheers,
- Andreas
Klaus D. Witzel wrote:
Hi Marcus, since every other project (package) depends on the integrity of the source code mechanism, I hereby volunteer to get a new source code subsystem (incl. code history) done during this summer. We have no time to wait for a new (reliable, scalable, available) source code mechanism, so I intend to make resources available for this project. Here's a draft for a new source code subsystem: 0] (digit for Wolfgang ;-) there are at least three preferences: 1] use the traditional subsystem as is 2] use Dan Ingalls Source Compression as suggested earlier 3] use a service based (network based) mechanism I've studied the recent additions which where made to CompiledMethod w.r.t. MethodProperties, this has to be cleaned up before anything else can be done. No, I'm not a friend of storing any MetaData *in* the meta data's object. I propose to put the MethodProperties into the MethodDictionary where they belong (same method can *and will* have different properties per class = one MethodProperties in each class' method dictionary). As a consequence, today's MethodProperties would all be garbage collected (except for the few pragmas), because all they store is the selector. And CompiledMethod>>#selector is resolved by #methodClass and #keyAtValue: so #who can disappear as is intended. After that, the *new* source code information can be put into the new MethodProperties, which allows an easy migration because the old source code pointer won't be touched until after migration. Comments, suggestions, critique, all appreciated. /Klaus On Mon, 24 Jul 2006 09:16:58 +0200, Marcus wrote:
On 24.07.2006, at 08:46, Peace Jerome wrote:
Hi all,
I've noticed that condensing sources is on the todo list of the 3dot9 gamma push.
One of the problems is that the .changes is limited to 32 MB, and 3.9 7048 is already 25MB, which means people would very soon see the limit in normal use (e.g. when loading a bunch of packages).
Another problem is that if we remove the linefeeds in those couple of hundrets of methods and do a bit of more _ conversion (which is intended), then we even hit the 32MB limit while in 3.9alpha.
One of the deliverables with 3dot9 final should be a 3.10 with all changes from 3.0.
This is not possible because of two reasons: - the MC based process is broken, we can't load from 3.0 to get 3.9beta. Yes, bad. I am taking full responsibility. (We have the complete code history, though. Just not easily loadable) - The changes file would hit the 32MB limite before you would get a complete 39beta even if it would work in theory.
Besides keeping the source, I'm sure you would want to keep the changeset information, too. Cost: 5MB of image size for 3.9 alone.
So just for a 3.9 with all changes, all in all we are talking about 30MB that nobody needs but system developers. Do we *really* want to ship that to all the users?
The history mechanism of Squeak does not scale anyway. Why only ship the history from 3.0? why not the *complete* history? What do you do in 10 years? 7MB left in the changeset is kind of small...
The whole history mechanism of Squeak does not scale: There is no need at all to have the old versions on the local disc at all. What we need is a server that has the complete history that then can be queried from the clients, everthing else makes no sense.
For now, we should keep of course the latest beta version with the history of 3.9 available. Just that it's not called "3.9", but "3.9 dev" or something like that.
And we should get a student to work on that code history project...
Marcus
One of the deliverables with 3dot9 final should be a 3.10 with all changes from 3.0.
This is not possible because of two reasons:
- the MC based process is broken, we can't load from 3.0 to get 3.9beta. Yes, bad. I am taking full responsibility.
I'm too. I invite everybody to build something and we will use it.
(We have the complete code history, though. Just noteasily loadable)
- The changes file would hit the 32MB limite before you would get a complete 39beta even if it would work in theory.
Besides keeping the source, I'm sure you would want to keep the changeset information, too. Cost: 5MB of image size for 3.9 alone.
So just for a 3.9 with all changes, all in all we are talking about 30MB that nobody needs but system developers. Do we *really* want to ship that to all the users?
The history mechanism of Squeak does not scale anyway. Why only ship the history from 3.0? why not the *complete* history? What do you do in 10 years? 7MB left in the changeset is kind of small...
The whole history mechanism of Squeak does not scale: There is no need at all to have the old versions on the local disc at all. What we need is a server that has the complete history that then can be queried from the clients, everthing else makes no sense.
For now, we should keep of course the latest beta version with the history of 3.9 available. Just that it's not called "3.9", but "3.9 dev" or something like that.
And we should get a student to work on that code history project...
Marcus
squeak-dev@lists.squeakfoundation.org
-
Andreas Raab -
Klaus D. Witzel -
Lic. Edgar J. De Cleene -
Marcus Denker -
Mathieu SUEN -
Peace Jerome -
stéphane ducasse