Hi All,

Sorry I’m joining this conversation late.

We are using ImageSegment for transferring croquet island state. Is this something covered in the changes from Bert or are we talking about dropping ImageSegment altogether?

All the best,

Ron Teitelbaum

From: Pharo-dev [mailto:pharo-dev-bounces@lists.pharo.org] On Behalf Of Eliot Miranda Sent: Thursday, September 22, 2016 2:47 PM To: Squeak Virtual Machine Development Discussion Cc: Pharo Development List; The general-purpose Squeak developers list Subject: Re: [Pharo-dev] [Vm-dev] Re: Nuking VM ImageSegment support (was Re: [squeak-dev] Daily Commit Log; System-bf.916)

Hi Max,

On Thu, Sep 22, 2016 at 11:42 AM, Max Leske maxleske@gmail.com wrote:

On 22 Sep 2016, at 20:28, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Bert, Hi All,

On Thu, Sep 15, 2016 at 2:55 PM, commits@source.squeak.org wrote:

[snip]

http://lists.squeakfoundation.org/pipermail/packages/2016-September/068930.h...

Name: System-bf.916 Ancestors: System-bf.915

Replace VM-level ImageSegment loading with a Smalltalk implementation for old (interpreter-era) projects.

Also removes support for writing segments.

This overrides the Spur support introduced in System-eem.758.

So one question is should we delete VM support for ImageSegment from the Spur VM? There's at least 1.5k of generated source for the Spur ImageSegment load and save support, some 2% of the interpreter/primitives source code. That's a lot, and the code is complex and ugly. If it never really worked before IMO we should nuke it asap. If it worked in some fashion perhaps we can schedule its demise for the 6.0 release's VM.

What do others think?

As long as you don’t remove it from the Cog VM’s until I no longer need it I’m fine with that.

And when would that be? Do you mean that you use it in ways not covered by Bert's modifications (which render the VM support superfluous), or do you mean that you use ImageSegment as a naive consumer and are happy just so long as it works?

Max

_,,,^..^,,,_

best, Eliot

Hi Eliot,

On 22 Sep 2016, at 20:46, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Max,

On Thu, Sep 22, 2016 at 11:42 AM, Max Leske <maxleske@gmail.com mailto:maxleske@gmail.com> wrote:

...

On 22 Sep 2016, at 20:28, Eliot Miranda <eliot.miranda@gmail.com mailto:eliot.miranda@gmail.com> wrote:

Hi Bert, Hi All,

On Thu, Sep 15, 2016 at 2:55 PM, <commits@source.squeak.org mailto:commits@source.squeak.org> wrote: [snip] http://lists.squeakfoundation.org/pipermail/packages/2016-September/068930.h... http://lists.squeakfoundation.org/pipermail/packages/2016-September/068930.html

Name: System-bf.916 Ancestors: System-bf.915

Replace VM-level ImageSegment loading with a Smalltalk implementation for old (interpreter-era) projects.

Also removes support for writing segments.

This overrides the Spur support introduced in System-eem.758.

So one question is should we delete VM support for ImageSegment from the Spur VM? There's at least 1.5k of generated source for the Spur ImageSegment load and save support, some 2% of the interpreter/primitives source code. That's a lot, and the code is complex and ugly. If it never really worked before IMO we should nuke it asap. If it worked in some fashion perhaps we can schedule its demise for the 6.0 release's VM.

What do others think?

As long as you don’t remove it from the Cog VM’s until I no longer need it I’m fine with that.

And when would that be?

Can’t really say but I'm hoping to get rid of ImageSegment within the next 2 years (very rough estimate).

Do you mean that you use it in ways not covered by Bert's modifications (which render the VM support superfluous), or do you mean that you use ImageSegment as a naive consumer and are happy just so long as it works?

Speed is important to me, as I use ImageSegment to create snapshots of our applications (and hence I need write support, which Bert apparently removed). Those snapshots can exceed 90 MB and the graphs include thousands of objects. I fear that a pure Smalltalk implementation would not be fast enough.

On the other hand, we would simply not move to a VM version without ImageSegment support, so that case may give me the boost I need to get rid of ImageSegment :) Currently we’re preparing to move to our first Cog VM in production. If you can give me 2 or 3 months, so that I know the version we use works for us, you could then remove ImageSegment support and we would start replacing ImageSegment with something else so we could keep updating our VM.

Cheers, Max

Max

...

_,,,^..^,,,_ best, Eliot

-- _,,,^..^,,,_ best, Eliot

Hi Bert, Hi Tim,

On Thu, Sep 15, 2016 at 2:55 PM, commits@source.squeak.org wrote:

[snip]

...

http://lists.squeakfoundation.org/pipermail/packages/2016-Se ptember/068930.html

Name: System-bf.916 Ancestors: System-bf.915

Replace VM-level ImageSegment loading with a Smalltalk implementation for old (interpreter-era) projects.

Also removes support for writing segments.

This overrides the Spur support introduced in System-eem.758.

I am currently putting back the ImageSegment support for saving projects and loading native segments produced in this way. I''ve added subclasses of ImageSegment called NativeImageSegment and LegacyImageSegment. I have a few questions.

1. I'm curious about the ImageSegment>>scanFrom: method:

!ImageSegment methodsFor: 'fileIn/Out' stamp: 'RAA 6/22/2000 17:49'! scanFrom: aStream "Move source code from a fileIn to the changes file for classes in an ImageSegment. Do not compile the methods. They already came in via the image segment. After the ImageSegment in the file, !!ImageSegment new!! captures control, and scanFrom: is called." | val chunk |

[aStream atEnd] whileFalse: [aStream skipSeparators. val := (aStream peekFor: $!!) ifTrue: ["Move (aStream nextChunk), find the method or class comment, and install the file location bytes" (Compiler evaluate: aStream nextChunk logged: false) scanFromNoCompile: aStream forSegment: self] ifFalse: [chunk := aStream nextChunk. aStream checkForPreamble: chunk. Compiler evaluate: chunk logged: true]. aStream skipStyleChunk]. "regular fileIn will close the file" ^ val! !

I don't see this in the new (legacy, System-bf.916) image segment loading code. Do Etoys projects not include source code and hence its irrelevant? The question boils down to whether this should be a method on ImageSegment, and hence inherited by both NativeImageSegment and LegacyImageSegment, or just NativeImageSegment. Also, I don't see how this method attaches sources to the methods brought in. Is it in fact useless and hence that's why you've left it out? If so, would it not be useful to add code that does update the source pointers in loaded methods correctly?

2. Should I pull in the Etoys extensions deleted in EToys-bf.234? (ImageSegment>>(classOrganizersBeRoots:,rehashDictionaries:,rehashMethodDictionaries:)? I'm guessing they'll be needed for native loading.

3. Same for the SMBase extension ImageSegment>>writeForExportOn:. I guess this should come back in too.

4. Where should I look for tests that may have been removed?

_,,,^..^,,,_ best, Eliot

Hi Bert,

On Tue, Jul 4, 2017 at 5:01 AM, Bert Freudenberg bert@freudenbergs.de wrote:

On Mon, Jul 3, 2017 at 11:43 PM, Eliot Miranda eliot.miranda@gmail.com wrote:

...

Hi Bert, Hi Tim,

On Thu, Sep 15, 2016 at 2:55 PM, commits@source.squeak.org wrote:

...

[snip]

...

http://lists.squeakfoundation.org/pipermail/packages/2016-Se ptember/068930.html

Name: System-bf.916 Ancestors: System-bf.915

Replace VM-level ImageSegment loading with a Smalltalk implementation for old (interpreter-era) projects.

Also removes support for writing segments.

This overrides the Spur support introduced in System-eem.758.

I am currently putting back the ImageSegment support for saving projects and loading native segments produced in this way.

​Why?​ I thought we agreed that ImageSegments are not a great idea as a file format, for their lack of compatibility. Serialization speed isn't an issue anymore thanks to your fast jit.

Quite, but there's legacy...

1. Tert is being ported to Squeak 6.x. Terf (nee Teleplace, nee Qwaq Forums, nee Croquet) makes heavy use of image segments; they're used to transfer room contents from server to entering client.

2. Max Leske is using segments for some application in Pharo and has been waiting patiently for a working version in Spur

3. I myself wanted to be able to transfer projects from working image to fresh VMMaker image build

4. the code is mature and works, so if it ain't broke why break it ? :-)

What ImageSegments *are* good at is for splitting up an image into segments

that do not have to be in memory at the same time - but in that case they would not be an independent entity but be considered part of one particular image. And for that we need only very few methods (that I removed too, see storeSegment).

Well, we should indeed clean up, but I am for keeping the ability to save projects and for applications like Terf to continue.

I note that image segments can only be loaded into a VM using an identical heap representation, hence, even though 64 & 32 bit Spur have identical object header formats, loading a 32-bit segment into 64-bit Spur or vice verse needs to be done by parsing, adding to the suite of parsers written by Bert.

So while the store-side primitive may be useful, the load-time primitive is less useful. At some stage we could discard it in favor of a pure Smalltalk loader, provided performance is good enough, e.g. for Terf.

I''ve added subclasses of ImageSegment called NativeImageSegment and

...

LegacyImageSegment. I have a few questions.

1. I'm curious about the ImageSegment>>scanFrom: method:

!ImageSegment methodsFor: 'fileIn/Out' stamp: 'RAA 6/22/2000 17:49'! scanFrom: aStream "Move source code from a fileIn to the changes file for classes in an ImageSegment. Do not compile the methods. They already came in via the image segment. After the ImageSegment in the file, !!ImageSegment new!! captures control, and scanFrom: is called." | val chunk |

[aStream atEnd] whileFalse:
    [aStream skipSeparators.
    val := (aStream peekFor: $!!)
        ifTrue: ["Move (aStream nextChunk), find the method or class
                    comment, and install the file location bytes"
                (Compiler evaluate: aStream nextChunk logged: false)
                    scanFromNoCompile: aStream forSegment: self]
        ifFalse: [chunk := aStream nextChunk.
                aStream checkForPreamble: chunk.
                Compiler evaluate: chunk logged: true].
    aStream skipStyleChunk].
"regular fileIn will close the file"
^ val! !

I don't see this in the new (legacy, System-bf.916) image segment loading code. Do Etoys projects not include source code and hence its irrelevant?

​They do, but the code path to load them does not go through this method.

What path does it go through?

The question boils down to whether this should be a method on ImageSegment,

...

and hence inherited by both NativeImageSegment and LegacyImageSegment, or just NativeImageSegment. Also, I don't see how this method attaches sources to the methods brought in. Is it in fact useless and hence that's why you've left it out? If so, would it not be useful to add code that does update the source pointers in loaded methods correctly?

The ImageSegment nowadays only stores uniclasses as objects. Other changes to the system are simply filed in, and not included in the segment. This method appears to be unused. I tried to trim everything not strictly need​ed for project loading

OK, I'll nuke it.

2. Should I pull in the Etoys extensions deleted in EToys-bf.234?

...

(ImageSegment>>(classOrganizersBeRoots:,rehashDictionaries:,rehashMethodDictionaries:)? I'm guessing they'll be needed for native loading.

rehashDictionaries ​: is still there, not sure if we need the others.​

...

3. Same for the SMBase extension ImageSegment>>writeForExportOn:. I

guess this should come back in too.

​SqueakMap uses ImageSegments, yes. No idea if it's a good idea to write a new native segment.​

4. Where should I look for tests that may have been removed?

...

​There are no tests, unfortunately ...

The way I was testing is taking a project written by Etoys (from squeakland.org) and dropping it into a trunk image. Before your changes this did work, more or less: If you revert to System-pre.956 and make allocateMethodContext: and allocateBlockContext: simply return nil, you can load old projects again. These contexts aren't needed anyways for Etoys projects (if we needed them I'd recreate by recompiling).

I'm attaching a simple project that I just tested with this.

Thanks!

_,,,^..^,,,_ best, Eliot

On 23 September 2016 at 07:10, Vaidotas Didžbalis vaidasd@gmail.com wrote:

Hello all, To save morph and restore it is a basic behavior which is broken since Squeak 4.5. This is a bug. Beginning from 4.5 new Squeak user will get a debugger window open in a few clicks. This bug happened as a side effect of pushing Environments package into the Squeak image beginning from Squeak 4.5. No one uses Environments; there are significant use for morph saving. Squeak needs this ability. Maybe it is easier to remove Environments from stock Image and to fix the bug introduced by that package. Introduction of new packages need not to break useful things... regards, Vaidotas

It was pretty difficult for Colin Putney to perform the heart surgery necessary to get Environments into the image in the first place. The capabilities that Environments enable are pretty dang awesome, and it's a crying shame that people either don't see these capabilities (sandboxing, trivial resolution of class name clashes are the START of things) or (more likely) simply haven't had the time to build the _missing tooling support_ to make Environments work to their fullest.

I'm equally to blame here: I burned out a few years back, and have since been too busy doing other stuff to work on Environments (or Squeak at all).

But it would be tragic if we performed the equally painful surgery to remove Environments.

So instead of removing a super awesome half-implemented feature of Squeak, why don't we, I don't know, _fix the bug_. Raise a bug report. Give a patch. It will make Squeak better.

frank

Yay sandboxing Yay trivial solve name clashing Why hide it

Vat good iss a Dooms Day device If You Keep It A Zecret

Vhy Didn't You Tell Ze VVorld HAH? <---[ Doctor strange love ]

in my opinion if you make a function in a software and don't tell anyone you might as well not have made the function nobody knows it's in there in Smalltalk the code itself often tells anyone but in other languages not so much consequently they seem to have better documentation some or mostly in Smalltalk if a function is dispersed or complex such that the code does not speak very well or is muted then not spending a few minutes to comment ( like Andy Bower used to do for Dolphin every single method with a one line comment<---[ is priceless ][ no other help is needed ][ mostly ] most of the main classes big commented ) the main Classes with design and or usage notes is a crime of wasted effort in my opinion look at the fact that Dolphin has always been so bullet proof with images going for months even years without a crash or a memory leak and all or most of it made by just one or two guys can it be attributed to the comments at all well i bet you a good big chunk of that success rate can be I know those comments contributed a good big chunk of my not needing to ask for help with Dolphin ever such that i got a bad habit of never asking for help and was totally spoiled for anything that was not Dolphin

how about this idea a Class whose only job is to describe some subsystem where each of the Methods is just a comment or returns a String like a little book right in the code ( use Pier Seaside Aida Iliad etc if you had a wild hair to get fancy )<---[ not needed ]

so up with comments and up with Andy Bower is what I'm saying

and don't hand Google another brick in the wall of their world domination by hiding comments out on the internet

unless you link to it from comments in the code

in that case every single Class involved should contain such a link in its comment to its comment out on the web and Methods could have comment links to specific places in the web pages where it is commented or you hate the users and you hate cheaply gotten Maybe wider Smalltalk usage in my opinion

On Saturday, September 24, 2016, Stéphane Rollandin lecteur@zogotounga.net wrote:

It was pretty difficult for Colin Putney to perform the heart surgery

...

necessary to get Environments into the image in the first place. The capabilities that Environments enable are pretty dang awesome, and it's a crying shame that people either don't see these capabilities (sandboxing, trivial resolution of class name clashes are the START of things) or (more likely) simply haven't had the time to build the _missing tooling support_ to make Environments work to their fullest.

Where is the canonical documentation for Environments, where I would expect to find an exposition of its overall architecture, plus a detailed tour of its most important classes and methods, along with a couple of examples showing how to use it?

Stef

On 9/24/16, Frank Shearar frank.shearar@gmail.com wrote: ....

It was pretty difficult for Colin Putney to perform the heart surgery necessary to get Environments into the image in the first place. The capabilities that Environments enable are pretty dang awesome, and it's a crying shame that people either don't see these capabilities (sandboxing, trivial resolution of class name clashes are the START of things) or (more likely) simply haven't had the time to build the _missing tooling support_ to make Environments work to their fullest.

I'm equally to blame here: I burned out a few years back, and have since been too busy doing other stuff to work on Environments (or Squeak at all).

But it would be tragic if we performed the equally painful surgery to remove Environments.

So instead of removing a super awesome half-implemented feature of Squeak, why don't we, I don't know, _fix the bug_. Raise a bug report. Give a patch. It will make Squeak better.

Yes, this is the way to go. And add notes on the Squeak wiki

And simple use cases first. Maybe entering a new environment when entering a project and leaving it then leaving the environment is a simple use case.

Subclassing MorphicProject is neat these days and with overriding a few methods you can customize the project.

The question is what would be needed to set up a project specific environment. Maybe a new thread should be started for this.

--Hannes

On Fri, Sep 23, 2016 at 10:10 PM, Frank Shearar frank.shearar@gmail.com wrote:

On 23 September 2016 at 07:10, Vaidotas Didžbalis vaidasd@gmail.com wrote:

...

Hello all, To save morph and restore it is a basic behavior which is broken since Squeak 4.5. This is a bug. Beginning from 4.5 new Squeak user will get a debugger window open in a few clicks. This bug happened as a side effect of pushing Environments package into the Squeak image beginning from Squeak 4.5. No one uses Environments; there are significant use for morph saving. Squeak needs this ability. Maybe it is easier to remove Environments from stock Image and to fix the bug introduced by that package. Introduction of new packages need not to break useful things... regards, Vaidotas

It was pretty difficult for Colin Putney to perform the heart surgery necessary to get Environments into the image in the first place. The capabilities that Environments enable are pretty dang awesome, and it's a crying shame that people either don't see these capabilities (sandboxing, trivial resolution of class name clashes are the START of things) or (more likely) simply haven't had the time to build the _missing tooling support_ to make Environments work to their fullest.

I'm equally to blame here: I burned out a few years back, and have since been too busy doing other stuff to work on Environments (or Squeak at all).

But it would be tragic if we performed the equally painful surgery to remove Environments.

So instead of removing a super awesome half-implemented feature of Squeak, why don't we, I don't know, _fix the bug_. Raise a bug report. Give a patch. It will make Squeak better.

+1000

frank

_,,,^..^,,,_ best, Eliot