Hi Ben,

On Thu, Oct 25, 2018 at 8:54 PM Ben Coman btc@openinworld.com wrote:

Just a few thoughts. Take with a grain of salt.

On Fri, 26 Oct 2018 at 01:31, Eliot Miranda eliot.miranda@gmail.com wrote:

...

Hi All,

efforts are underway to include the VMMaker source in the

opensmalltalk-vm repository.

Very nice.

...

I'm hoping to see all the Smalltalk source included in the Tonel format (one file per class, and one file per extended class), with support for Pharo and Squeak quite soon. This is therefore an opportunity to also reorganize the structure of the repository to have a more comprehensible and less cluttered top-level directory. I'm interested in people's ideas. here are two suggestions.

1. move all source under src and all builds under build. So src/smalltalk/ all the VMMaker-related packages AioPlugin BytecodeSets Cog VMMaker

Could you explain what would be in VMMaker? I thought all the plugins were a part of VMMaker rather than siblings. i.e. why not src/vmmaker/AioPlugin & src/vmmaker/vm (or src/vmmaker/Core)

Also what would be in Cog? just the JIT stuff?

The above structure is merely a mirror of the existing package structure. See e.g. image/BuildSqueakSpurTrunkVMMakerImage.st

...

src/generated/              all the generated code
                    plugins/     all generated plugins
                    v3              the V3 VM (now src/vm)

                    spur32       the 32-bit Spur VM (now spursrc/vm)

...

                    spur64       the 64-bit Spur VM (now spur64src/vm)

Those renames definitely add clarity and will help newcomers, particularly the v3 one. However I think "src" should just be for "raw" sources. A top level "generated" folder per your point[2.] would be good, but perhaps also sllmming the name down to "gensrc".

OK.

src/platforms the current platforms directory

...

src/third-party/

                    processors

...

Can "src" be kept just for "our" sources ? IIUC, processors/IA32 contains third-party source folder "bochs" as well our our code which looks like build script in folders "$(PLATFORM)bochs" Ideally that part should be split out under the build folder, and the perhaps the third-party source goes under root folder "othersrc" but the existing "processors" folder as a whole is not getting in the way, so for now leave it.

Though the term "processors" has broad meaning and it may make more sense to newcomers if the folder was "processor-sim" or "cpu-sim".

OK.

build/                             the current build directories

...

    macos32x86
    macos64x64
deploy/
image/

Maybe an idea for the future, "image" could be much more than just holding scripts to create the build.image. Initially it could hold image-side code for CI testing of the VM. But also it seed the growth of a common image-side codebase for the low level stuff between distributions. Pharo's plan is to modularize things to build their shipping image from multiple repositories, so it shouldn't matter to them if their bootstrap pulls some parts direct from opensmalltalk. Or maybe it would be better to have all that in a different repo https://github.com/OpenSmalltalk/opensmalltalk-image

In any case, "image" as things currently stand feels a bit odd at the top-level. It seems it would fit better under "scripts/image", or "scripts/dev.image"

btw, what is spec/locode.xml ? Is that something loaded by the image-side by a user? That might fit under image/specs/lowcode.xml ?

Ronie knows, but my limited understanding is that spec/locode.xml specifies the semantics of the locoed instruction set, such that versions in a specific language, assembler or machine code can be autogenerated. So the Smalltalk implementation of the locoed instructions in Sista are autogenerated from spec/locode.xml.

tests/

...

2. single top-level directories for generated source, and Smalltalk

source, everything else unchanged smalltalksrc/ all the VMMaker-related packages generatedsrc/ all the generated code as above in src/generated, but in generatedsrc instead

platforms

...

processors third-party tests deploy

Obviously #1 is more work but produces a much cleaner structure. Other suggestions? Reactions to the above? _,,,^..^,,,_ best, Eliot

3. So top level could look like... src/ our own root sources, native and smalltalk

VMMaker gensrc/ generated from VMMaker othersrc/ third-party libs kept intree (manually or via subtree or submodule) cachesrc/ sources downloaded during build (empty folder on github except readme.md) scripts/ build/ deploy/

with second level... src/smalltalk/ all the VMMaker-related packages src/platforms the current platforms directory src/third-party/ only the spec files

I'm leaning towards src smalltalk-src generated-src

with a good README ;-)

The main readme says "Another distinction is between Stack, Cog and Sista VMs", so would it be better to be explicit about which ones are Cog? Also would it be better to be explicit about v3 only being 32 bit (if I guess correctly) gensrc/plugins gensrc/v3cog32 old src gensrc/v3stack32 old stacksrc gensrc/spurstack32 old spurstacksrc gensrc/spurstack64 old spurstack64src gensrc/spurcog32 old spursrc gensrc/spurcog64 old spur64src gensrc/spursista32 old spursista64src gensrc/spursista64 old spursista64src gensrc/spurlowcode32 old spurlowcodesrc gensrc/spurlowcode64 old spurlowcode64src

othersrc/gdb othersrc/bochs an alternative is to mirror/fork bochs into opensmalltalk-vm/bochs and have a build script pull that to your machine e.g. https://github.com/search?q=bochs+mirror&type=Repositories you can have one branch called "mirror" that only *ever* sucks from upstream with any custom mods in a branch called "Cog". scripts/ci scripts/image scripts/git

build/linuxXXX build/winXXX

deploy/squeak deploy/pharo deploy/newspeak

cheers -ben

_,,,^..^,,,_ best, Eliot

May I ask - just out of curiosity - which arguments ultimately lead to the decision to use a new, non-Smalltalk syntax to store code in files? I had a discussion with a colleague today who just wonder why this is, too.

Best, Steffen

Am .10.2018, 09:01 Uhr, schrieb Tobias Pape Das.Linux@gmx.de:

Hi All,

...

On 25.10.2018, at 19:30, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi All,

efforts are underway to include the VMMaker source in the  

opensmalltalk-vm repository. I'm hoping to see all the Smalltalk source included in the Tonel format (one file per class, and one file per extended class), with support for Pharo and Squeak quite soon.

I know its not a constructive comment, but I'm not very fond of the Tonel format. Yes, it solves the need (or pecularities) of too-many-tiny-files. But so does good-old file-out or Cuis pkg.st formt (eg, https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/tree/master/Packages/) . I really do not like the introduction of a new, incompatible Syntax, that also introduces a mismatch between what is written in the browser and what is written to disk[1]. I also do not think that using STON as a metadata format within Tonel is a good match. Moreover, Tonel is comparatively young and I'd be cautious to just yet use it in such a rather large, complex project as VMMaker, where bugs in the formats implementation would only surface slowly.

That being said, if there's consensus to go with Tonel, well, so be it.

Regarding the rest: sounds good, with a slight preference to variant #1.

Best regards -Tobias

[1]: Think I have a method with a very long selector, eg MCClassDefinition:

initializeWithName: nameString superclassName: superclassString traitComposition: traitCompositionString classTraitComposition: classTraitCompositionString category: categoryString instVarNames: ivarArray classVarNames: cvarArray poolDictionaryNames: poolArray classInstVarNames: civarArray type: typeSymbol comment: commentString commentStamp: stampStringOrNil name := nameString asSymbol. superclassName := superclassString ifNil: ['nil'] ifNotNil: [superclassString asSymbol]. traitComposition := traitCompositionString. classTraitComposition := classTraitCompositionString. category := categoryString. type := typeSymbol. comment := commentString withSqueakLineEndings. commentStamp := stampStringOrNil ifNil: [self defaultCommentStamp]. variables := OrderedCollection new. self addVariables: ivarArray ofType: MCInstanceVariableDefinition. self addVariables: cvarArray sorted ofType: MCClassVariableDefinition. self addVariables: poolArray sorted ofType: MCPoolImportDefinition. self addVariables: civarArray ofType: MCClassInstanceVariableDefinition

Seeing the Tonel format, this formatting of the selector, even if it is uncommon, would not be preserved, right?

...

This is therefore an opportunity to also reorganize the structure of the repository to have a more comprehensible and less cluttered top-level directory. I'm interested in people's ideas. here are two suggestions.

1. move all source under src and all builds under build. So src/smalltalk/ all the VMMaker-related packages AioPlugin BytecodeSets Cog VMMaker src/generated/ all the generated code plugins/ all generated plugins v3 the V3 VM (now src/vm) spur32 the 32-bit Spur VM (now spursrc/vm) spur64 the 64-bit Spur VM (now

spur64src/vm) src/platforms the current platforms directory src/third-party/ processors build/ the current build directories macos32x86 macos64x64 deploy/ image/ tests/

2. single top-level directories for generated source, and Smalltalk

source, everything else unchanged smalltalksrc/ all the VMMaker-related packages generatedsrc/ all the generated code as above in src/generated, but in generatedsrc instead platforms processors third-party tests deploy

Obviously #1 is more work but produces a much cleaner structure. Other suggestions? Reactions to the above? _,,,^..^,,,_ best, Eliot

Am 26.10.2018 um 20:33 schrieb Todd Blanchard tblanchard@mac.com:

I suspect it had something to do with making things git merge friendly.

I think the granularity is overly fine at one method per file as it makes browsing a git repo from a web browser annoyingly clicky. But there it is.

One file per method does not work on windows. So if we do not want to loose them one file per class seems to be the way to go, no?

Norbert

...

On Oct 26, 2018, at 11:02 AM, Steffen Märcker merkste@web.de wrote:

May I ask - just out of curiosity - which arguments ultimately lead to the decision to use a new, non-Smalltalk syntax to store code in files? I had a discussion with a colleague today who just wonder why this is, too.

Best, Steffen

Am .10.2018, 09:01 Uhr, schrieb Tobias Pape Das.Linux@gmx.de:

...

Hi All,

...

On 25.10.2018, at 19:30, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi All,

efforts are underway to include the VMMaker source in the opensmalltalk-vm repository. I'm hoping to see all the Smalltalk source included in the Tonel format (one file per class, and one file per extended class), with support for Pharo and Squeak quite soon.

I know its not a constructive comment, but I'm not very fond of the Tonel format. Yes, it solves the need (or pecularities) of too-many-tiny-files. But so does good-old file-out or Cuis pkg.st formt (eg, https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/tree/master/Packages/) . I really do not like the introduction of a new, incompatible Syntax, that also introduces a mismatch between what is written in the browser and what is written to disk[1]. I also do not think that using STON as a metadata format within Tonel is a good match. Moreover, Tonel is comparatively young and I'd be cautious to just yet use it in such a rather large, complex project as VMMaker, where bugs in the formats implementation would only surface slowly.

That being said, if there's consensus to go with Tonel, well, so be it.

Regarding the rest: sounds good, with a slight preference to variant #1.

Best regards -Tobias

[1]: Think I have a method with a very long selector, eg MCClassDefinition:

initializeWithName: nameString superclassName: superclassString traitComposition: traitCompositionString classTraitComposition: classTraitCompositionString category: categoryString instVarNames: ivarArray classVarNames: cvarArray poolDictionaryNames: poolArray classInstVarNames: civarArray type: typeSymbol comment: commentString commentStamp: stampStringOrNil name := nameString asSymbol. superclassName := superclassString ifNil: ['nil'] ifNotNil: [superclassString asSymbol]. traitComposition := traitCompositionString. classTraitComposition := classTraitCompositionString. category := categoryString. type := typeSymbol. comment := commentString withSqueakLineEndings. commentStamp := stampStringOrNil ifNil: [self defaultCommentStamp]. variables := OrderedCollection new. self addVariables: ivarArray ofType: MCInstanceVariableDefinition. self addVariables: cvarArray sorted ofType: MCClassVariableDefinition. self addVariables: poolArray sorted ofType: MCPoolImportDefinition. self addVariables: civarArray ofType: MCClassInstanceVariableDefinition

Seeing the Tonel format, this formatting of the selector, even if it is uncommon, would not be preserved, right?

...

This is therefore an opportunity to also reorganize the structure of the repository to have a more comprehensible and less cluttered top-level directory. I'm interested in people's ideas. here are two suggestions.

1. move all source under src and all builds under build. So

src/smalltalk/ all the VMMaker-related packages AioPlugin BytecodeSets Cog VMMaker src/generated/ all the generated code plugins/ all generated plugins v3 the V3 VM (now src/vm) spur32 the 32-bit Spur VM (now spursrc/vm) spur64 the 64-bit Spur VM (now spur64src/vm) src/platforms the current platforms directory src/third-party/ processors build/ the current build directories macos32x86 macos64x64 deploy/ image/ tests/

2. single top-level directories for generated source, and Smalltalk source, everything else unchanged

smalltalksrc/ all the VMMaker-related packages generatedsrc/ all the generated code as above in src/generated, but in generatedsrc instead platforms processors third-party tests deploy

Obviously #1 is more work but produces a much cleaner structure. Other suggestions? Reactions to the above? _,,,^..^,,,_ best, Eliot

On 26.10.2018, at 22:02, Jakob Reschke forums.jakob@resfarm.de wrote:

Am Fr., 26. Okt. 2018 um 21:13 Uhr schrieb Norbert Hartl norbert@hartl.name:

...

Am 26.10.2018 um 20:33 schrieb Todd Blanchard tblanchard@mac.com:

I think the granularity is overly fine at one method per file as it makes browsing a git repo from a web browser annoyingly clicky. But there it is.

One file per method does not work on windows. So if we do not want to loose them one file per class seems to be the way to go, no?

Actually it works fine on Windows, until people come up with ultra-long method names, typically taking lots of arguments. ;-)

If I remember correctly, Dale's argument for one method per file was that you can then ask Git (or whatever file-based VCS) about the history of one method, which is not possible directly with any coarser format. Yes, browsing through one method per file without a Smalltalk IDE is daunting. But the same applies to browsing any code base of a certain size without a proper IDE.

On the topic of the directory structure, I would prefer option 1 and keeping the "generated" name in favor of "gensrc". Spelling it out makes it clearer, IMHO. I also prefer third-party over othersrc because the latter is too generic. I could also agree on having /third-party as a top-level folder, so the /src folder only contains stuff that is genuinely opensmalltalk-vm.

The /tests directory currently does not really contain any tests, right? Yet, it is also some kind of source code, isn't it? So why not put it under /src as well?

By the way, once the "real" source code is in the repo, and the generated C source code is as well, the generated sources will lack behind the Smalltalk sources most of the time, won't they? That might be confusing for people who come to the repo, clone or download it and just want to compile the VM. "Hey I pulled commit abc that fixes xyz, yet I still have the bug when I build it!?" Are there any plans to tackle this?

Well said, all of it.

+1 -t

Kind regards, Jakob

Interesting to read through this thread! Thanks.

Am .10.2018, 04:59 Uhr, schrieb Ben Coman btc@openinworld.com:

And can also contribute to shortening the useful life of your SSD if you do a lot of those tiny writes frequently.

On Sun, Oct 28, 2018, 3:46 AM Stephan Eggermont stephan@stack.nl wrote:

Jakob Reschke forums.jakob@resfarm.de wrote:

...

Am Fr., 26. Okt. 2018 um 21:13 Uhr schrieb Norbert Hartl <

norbert@hartl.name

...

...

One file per method does not work on windows. So if we do not want to loose them one file per class seems to be the way to go, no?

Actually it works fine on Windows, until people come up with ultra-long method names, typically taking lots of arguments. ;-)

It actually doesn’t really work on any operating system with current hardware. Methods are small, and disk blocks are large. The read/write amplification kills performance.

Stephan

On Sun, Oct 28, 2018 at 2:41 AM Norbert Hartl norbert@hartl.name wrote:

...

Am 28.10.2018 um 08:44 schrieb Stephan Eggermont stephan@stack.nl:

Jakob Reschke forums.jakob@resfarm.de wrote:

...

Am Fr., 26. Okt. 2018 um 21:13 Uhr schrieb Norbert Hartl <

norbert@hartl.name

...

...

...

One file per method does not work on windows. So if we do not want to loose them one file per class seems to be the way to go, no?

Actually it works fine on Windows, until people come up with ultra-long method names, typically taking lots of arguments. ;-)

It actually doesn’t really work on any operating system with current hardware. Methods are small, and disk blocks are large. The read/write amplification kills performance.

There seem to be very different definitions of „it works“. We should just decide what we want to change: People and hardware or a storage format.

:) :) :) +1

The only thing I'd like to see in Tonel is per-method timestamps a la

Class { #name : #MCGitTonelGenericLoaderTest, #superclass : #ProtoObject, #category : #'MonticelloTonel-GitTests' }

{ #category : #resources, #timeStamp : 'ThierryGoubier 3/2/2018 19:24:09' }

MCGitTonelGenericLoaderTest >> getTestRepository: repositoryName [ | repo dir | dir := utilityClass directoryFromPath: repositoryName relativeTo: self getTestRepositoriesDirectory. repo := MCTonelGitRepository new directory: dir. ^ MCRepositoryGroup default repositories detect: [ :each | each = repo ] ifNone: [ repo ] ]

or

Class { #name : #MCGitTonelGenericLoaderTest, #superclass : #ProtoObject, #category : #'MonticelloTonel-GitTests' }

{ #category : #resources, #timeStamp : 'ThierryGoubier 3/2/2018 19:24:09' }

MCGitTonelGenericLoaderTest >> getTestRepository: repositoryName [ | repo dir | dir := utilityClass directoryFromPath: repositoryName relativeTo: self getTestRepositoriesDirectory. repo := MCTonelGitRepository new directory: dir. ^ MCRepositoryGroup default repositories detect: [ :each | each = repo ] ifNone: [ repo ] ]

but I know there is active opposition to this idea.

_,,,^..^,,,_ best, Eliot

On 28.10.2018, at 20:52, tim Rowledge tim@rowledge.org wrote:

...

On 2018-10-28, at 12:37 PM, Eliot Miranda eliot.miranda@gmail.com wrote:

...

The only thing I'd like to see in Tonel is per-method timestamps a la

[snip]

...

but I know there is active opposition to this idea.

Why on earth is there any opposition to what appears a perfectly reasonable idea? Within, of course, the context of an idea I find rather daft, that of trying to force Smalltalk source code into a model that seems completely at odds.

I think the reasoning is that the time stamp can trivially reified from the VCS history (ie, git blame…) The data is still there, yet no more completely tied to the code.

Best regards -Tobias

PS: also, the time stamp format we use up until now is not very portable (in a i18n kind of way)

On 2018-10-28, at 5:15 PM, Ben Coman btc@openinworld.com wrote:

Here are some historical discussion of merge problems experienced in the move to support git...

OK, I haven't really paid much attention to this in some time so my opinion, despite having originally written VMMaker, isn't all that important. I do wonder why anyone feels particular need to support git for the Smalltalk source for VMMaker. Sure, for the generated C code, as a way to make it easy for mundane tools to download and build etc, git is as tolerable as anything. But the Smalltalk code is our stuff, in our format, and we already have tools for that. Is it some wish to have everything in a single place, perhaps? I guess I could understand that urge.

tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: BYEBYE: Store in Write-Only Storage

Am 26.10.2018 um 20:02 schrieb Steffen Märcker merkste@web.de:

May I ask - just out of curiosity - which arguments ultimately lead to the decision to use a new, non-Smalltalk syntax to store code in files? I had a discussion with a colleague today who just wonder why this is, too.

Can you elaborate on why you think tonel is not smalltalk syntax and the others are?

Norbert

Best, Steffen

Am .10.2018, 09:01 Uhr, schrieb Tobias Pape Das.Linux@gmx.de:

...

Hi All,

...

On 25.10.2018, at 19:30, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi All,

efforts are underway to include the VMMaker source in the opensmalltalk-vm repository. I'm hoping to see all the Smalltalk source included in the Tonel format (one file per class, and one file per extended class), with support for Pharo and Squeak quite soon.

I know its not a constructive comment, but I'm not very fond of the Tonel format. Yes, it solves the need (or pecularities) of too-many-tiny-files. But so does good-old file-out or Cuis pkg.st formt (eg, https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/tree/master/Packages/) . I really do not like the introduction of a new, incompatible Syntax, that also introduces a mismatch between what is written in the browser and what is written to disk[1]. I also do not think that using STON as a metadata format within Tonel is a good match. Moreover, Tonel is comparatively young and I'd be cautious to just yet use it in such a rather large, complex project as VMMaker, where bugs in the formats implementation would only surface slowly.

That being said, if there's consensus to go with Tonel, well, so be it.

Regarding the rest: sounds good, with a slight preference to variant #1.

Best regards -Tobias

[1]: Think I have a method with a very long selector, eg MCClassDefinition:

initializeWithName: nameString superclassName: superclassString traitComposition: traitCompositionString classTraitComposition: classTraitCompositionString category: categoryString instVarNames: ivarArray classVarNames: cvarArray poolDictionaryNames: poolArray classInstVarNames: civarArray type: typeSymbol comment: commentString commentStamp: stampStringOrNil name := nameString asSymbol. superclassName := superclassString ifNil: ['nil'] ifNotNil: [superclassString asSymbol]. traitComposition := traitCompositionString. classTraitComposition := classTraitCompositionString. category := categoryString. type := typeSymbol. comment := commentString withSqueakLineEndings. commentStamp := stampStringOrNil ifNil: [self defaultCommentStamp]. variables := OrderedCollection new. self addVariables: ivarArray ofType: MCInstanceVariableDefinition. self addVariables: cvarArray sorted ofType: MCClassVariableDefinition. self addVariables: poolArray sorted ofType: MCPoolImportDefinition. self addVariables: civarArray ofType: MCClassInstanceVariableDefinition

Seeing the Tonel format, this formatting of the selector, even if it is uncommon, would not be preserved, right?

...

This is therefore an opportunity to also reorganize the structure of the repository to have a more comprehensible and less cluttered top-level directory. I'm interested in people's ideas. here are two suggestions.

1. move all source under src and all builds under build. So src/smalltalk/ all the VMMaker-related packages AioPlugin BytecodeSets Cog VMMaker src/generated/ all the generated code plugins/ all generated plugins v3 the V3 VM (now src/vm) spur32 the 32-bit Spur VM (now spursrc/vm) spur64 the 64-bit Spur VM (now spur64src/vm) src/platforms the current platforms directory src/third-party/ processors build/ the current build directories macos32x86 macos64x64 deploy/ image/ tests/

2. single top-level directories for generated source, and Smalltalk source, everything else unchanged

smalltalksrc/ all the VMMaker-related packages generatedsrc/ all the generated code as above in src/generated, but in generatedsrc instead platforms processors third-party tests deploy

Obviously #1 is more work but produces a much cleaner structure. Other suggestions? Reactions to the above? _,,,^..^,,,_ best, Eliot

On Fri, 26 Oct 2018, Tobias Pape wrote:

Think I have a method with a very long selector, eg MCClassDefinition:

initializeWithName: nameString superclassName: superclassString traitComposition: traitCompositionString classTraitComposition: classTraitCompositionString category: categoryString instVarNames: ivarArray classVarNames: cvarArray poolDictionaryNames: poolArray classInstVarNames: civarArray type: typeSymbol comment: commentString commentStamp: stampStringOrNil name := nameString asSymbol. superclassName := superclassString ifNil: ['nil'] ifNotNil: [superclassString asSymbol]. traitComposition := traitCompositionString. classTraitComposition := classTraitCompositionString. category := categoryString. type := typeSymbol. comment := commentString withSqueakLineEndings. commentStamp := stampStringOrNil ifNil: [self defaultCommentStamp]. variables := OrderedCollection new. self addVariables: ivarArray ofType: MCInstanceVariableDefinition. self addVariables: cvarArray sorted ofType: MCClassVariableDefinition. self addVariables: poolArray sorted ofType: MCPoolImportDefinition. self addVariables: civarArray ofType: MCClassInstanceVariableDefinition

Seeing the Tonel format, this formatting of the selector, even if it is uncommon, would not be preserved, right?

I just checked, and the formatting is preserved, even comments and white spaces between the selector part and the argument names appear as written. The only nuance is that no parser can optimally separate the method body from its signature, so there are edge cases with unexpected results. E.g.:

If you add type information as comments like below, the last comment will appear if it were part of the method body:

Object >> foo: foo "Integer" bar: bar "String"

^foo asString, bar

will be stored as

Object >> foo: foo "Integer" bar: bar [ "String"

^foo asString, bar ]

Levente