Hi Holger, thanks for reminding this. So we must regenerate source from *VMConstruction-Plugins-OSProcessPlugin.oscog-dtl.56* I just did that for nice.55 (for a 2nd time), a pity that I missed the more recent one... It's better to send such message to vm-dev as you did, I thnk audience is broader for VM concerns than pharo-dev

2017-06-01 11:32 GMT+02:00 Holger Freyther holger@freyther.de:

...

On 16. May 2017, at 20:57, Max Leske maxleske@gmail.com wrote:

Hi,

Hi!

...

We're building the pharo.cog.spur VM on Debian (32 bits, itimer, on 64

with compat libs). Unfortunately, we're running into one particular problem with OSProcess. Some primitives like primitiveFileStat work but forking a child process (e.g. with #waitForCommand) produces a segfault. We've tried to find a difference between our build and the VM downloaded from bintray but we weren't very successful. One difference we do see is that our libc6 version is 2.19, while the one used in the travis build is 2.15 (although both binaries use the 2.19 version at runtime).

...

I'm aware that OSProcess hasn't been tagged as officially ready for

Pharo 6 (which is where I'm using it). However, all of my tests with the prebuilt VM's have been successful.

I wanted to look at the issue and it seems that the compiler warning about out of bound array access[1] in restoreDefaultSignalHandlers remains unfixed? I reported it in February and the plugin code doesn't seem to be regenerated since then. Some of my PRs are still unreviewed so does somebody else want to try her or his luck?

I think this[2] and that[3] are candidates to be merged?

holger

[1] http://lists.pharo.org/pipermail/pharo-dev_lists. pharo.org/2017-February/127495.html [2] http://www.squeaksource.com/@ZzgxlHh7fhZ1n9Zo/rdfZ1xy3 [3] http://www.squeaksource.com/@ZzgxlHh7fhZ1n9Zo/wVMJyP1K

Hi David,

what would help here is if all the VMConstruction-Plugins- packages lived in a single repository. So instead of http://www.squeaksource.com/%7BAioPlugin,OSProcessPlugin,XDCP%7D there was http://www.squeaksource.com/OSConnectivity or http://www.squeaksource.com/AdditionalPlugins or some such. Checking lots of different repositories for changes is tedious. If there was one I could keep a repository inspector open and refresh.

Alternatively we need Update URLs instead of Update URL.

On Thu, Jun 1, 2017 at 5:38 AM, Nicolas Cellier < nicolas.cellier.aka.nice@gmail.com> wrote:

Hi Holger, thanks for reminding this. So we must regenerate source from *VMConstruction-Plugins-OSProcessPlugin.oscog-dtl.56* I just did that for nice.55 (for a 2nd time), a pity that I missed the more recent one... It's better to send such message to vm-dev as you did, I thnk audience is broader for VM concerns than pharo-dev

2017-06-01 11:32 GMT+02:00 Holger Freyther holger@freyther.de:

...

...

On 16. May 2017, at 20:57, Max Leske maxleske@gmail.com wrote:

Hi,

Hi!

...

We're building the pharo.cog.spur VM on Debian (32 bits, itimer, on 64

with compat libs). Unfortunately, we're running into one particular problem with OSProcess. Some primitives like primitiveFileStat work but forking a child process (e.g. with #waitForCommand) produces a segfault. We've tried to find a difference between our build and the VM downloaded from bintray but we weren't very successful. One difference we do see is that our libc6 version is 2.19, while the one used in the travis build is 2.15 (although both binaries use the 2.19 version at runtime).

...

I'm aware that OSProcess hasn't been tagged as officially ready for

Pharo 6 (which is where I'm using it). However, all of my tests with the prebuilt VM's have been successful.

I wanted to look at the issue and it seems that the compiler warning about out of bound array access[1] in restoreDefaultSignalHandlers remains unfixed? I reported it in February and the plugin code doesn't seem to be regenerated since then. Some of my PRs are still unreviewed so does somebody else want to try her or his luck?

I think this[2] and that[3] are candidates to be merged?

holger

[1] http://lists.pharo.org/pipermail/pharo-dev_lists.pharo.org/ 2017-February/127495.html [2] http://www.squeaksource.com/@ZzgxlHh7fhZ1n9Zo/rdfZ1xy3 [3] http://www.squeaksource.com/@ZzgxlHh7fhZ1n9Zo/wVMJyP1K

On 2. Jun 2017, at 18:43, K K Subbu kksubbu.ml@gmail.com wrote:

Hey!

+1. I would go further and propose that plugins, other than those which are integral to VM[1], be moved into separate projects and built in their own tree with no references to vm source paths. They should be free to organize their platform-specific and platform-independent code and makefiles.

why? How many plugins are used that don't ship with the pharo-vm (or squeakvm)? I have to say I like the pharo-vm approach a lot where the VMMaker* packages and generated sources sit next to each other.

Let's compare two options with the OSProcess timeline at hand

What happened:

* I compile with a new compiler, look at the compiler warning and agree with it * David Lewis agrees with my analysis and fixes it ... * Max runs into a crash * Max debugs a crash.. hours? days? * Max involves others ... * I have to debug the crash myself.. * I notice that the fix is not there yet * I have to wait for someone privileged to regenerate the source (black magic?)

My preferred option

* I compile with a new compiler, look at the compiler warning and agree with it * David Lewis agrees with my analysis and commits/pushes the fix to mc/ directory * New VM will be build and regenerates sources on the fly * Max never runs into the problem

Let's do it the Smalltalk way? Make simple things simple and complex (allow externally maintained plugins) possible?

have a nice weekend

holger

Hi Eliot,

The missing step is that the plugin source is regenerated . As a Smalltalk-side-of-the-business person David knows how to regenerate source. David, why didn't you regenerate? [honest Q, not an accusation]

To be honest, I just didn't think of it because I assumed that you were doing the code generation.

Agreed. No need to complicate source generation. But we still need people who fix the Smalltalk to be able to regenerate source. Several of us, Ronie, Clément, Nicolas, myself. David, you should feel free to do so also.

Noted, will do :-)

Thanks, Dave

On 2. Jun 2017, at 22:40, Eliot Miranda eliot.miranda@gmail.com wrote:

Hi Holger,

Hey Eliot!

I have no disagreement with your time lines, but I do disagree with the black magic accusation, and you introduce your own piece of magic (see below).

in the end it is a word and a very subjective thing. I used to spend a lot of time working on OpenEmbedded to build Linux distributions. If one didn't understand what needs to be done to build a distribution and didn't know how the system worked, it seemed like black magic.

and then checks in the new sources. David should have been able to do this. The process is not documented.

Let's say I saw the code was not regenerated, what could I have done? Figure out the not documented process, make a PR and hope it is being reviewed within a month or two? It might not be black magic but quite a high barrier and maybe higher than necessary?

...

My preferred option

• I compile with a new compiler, look at the compiler warning and agree with it
• David Lewis agrees with my analysis and commits/pushes the fix to mc/ directory

You're glossing over some steps here. Note that the fix is in the Smalltalk source, upstream of the C source.

I commit my Smalltalk code to the mc/ directory and the CI will build a VMMaker image and generate the C sources.

The missing step is that the plugin source is regenerated . As a Smalltalk-side-of-the-business person David knows how to regenerate source. David, why didn't you regenerate? [honest Q, not an accusation]

Done by the CI in pharo-vm. And besides being done by the CI, the script that builds the VMMaker image and generates the C sources provide an executable documentation of the process. So anyone capable of committing Smalltalk code to the mc/ directory can generate new sources.

One *mustn't* manually fix the generated source because doing so means

I didn't argue about that. That would be the beginning of the end.

One *must* regenerate from an unmodified checked in Monticello package because doing so means

• the C source is version stamped as originating from that package and so one can locate the correct version of the Smalltalk from which it derived

* The CI works by definition from an unmodified checked in Monticello package. * tags, source uploads, debian source packages can provide as 1:1 mapping from version to archived source

skipping stuff here to keep the answer short.

...

• New VM will be build and regenerates sources on the fly

How does this work? Seems like magic to me. Source has to be generated from a VMMaker image. We need to regenerate source only for that which has changed, not for the entire generated source. But right now that's problematic because of the time issues I mention above.

We are in the phase of computing where human time is (once again) more expensive that computing time. So the magic is that some labor can be outsourced to a shell script. The nice thing with a shell script is one can observe each step (set -x) it is doing. For "stable" code generation review one can still check-in and compare the generated code.

I do my deployments on Debian with debian packages. Since a couple of weeks commits to the pharo-vm will generate a debian source package with generated C source and upload it. At the place below one can see the uploads and the difference in generated code from one version to another.

https://build.opensuse.org/package/revisions/devel:languages:pharo:latest/ph...

have a nice day

holger

On Fri, Jun 2, 2017 at 9:54 AM, K K Subbu kksubbu.ml@gmail.com wrote:

On Friday 02 June 2017 04:50 PM, Holger Freyther wrote:

...

+1. I would go further and propose that plugins, other than those

...

which are integral to VM[1], be moved into separate projects and built in their own tree with no references to vm source paths. They should be free to organize their platform-specific and platform-independent code and makefiles.

why? How many plugins are used that don't ship with the pharo-vm (or squeakvm)? I have to say I like the pharo-vm approach a lot where the VMMaker* packages and generated sources sit next to each other.

My proposal is to loosen build time coupling, not packaging. Nothing prevents plugins built out of tree from being built, installed and shipped together. Supporting out of tree plugins allows others to extend a VM even long after its release.

Eliot has already responded to your example in detail, so I will not dwell upon it further, except to point out that we need a command line tool to generate a .c file directly from the plugin's mcz file. Then we can create a Make rule to compile a .o and get rid of .c file. e.g.

FooPlugin.o : FooPlugin.mcz squeak -headless vmmaker.image gencc.st FooPlugin.mcz cc -c ... FooPlugin.c rm -f FooPlugin.c

Over my dead body. The make must *NEVER* be done like this. The problems are many: - the VM so produced is undebuggable. There is no source for FooPlugin.c; it has been deleted - the build is slow; building a VMMaker image (the necessary prerequisite for st2c) takes a long time, mush longer than the st2c step - unless FooPlugin.mcz is versioned then the VM so produced is unversioned and if FooPlugin.mcz /is/ versioned then there's no reason why the plugin can't be produced using the normal workflow, generate C source, check it in to opensmalltalk/vm, build, and that way we have a versioned entity and break the dependency of builds requiring VMMaker

Look, I just spent several years persuading the Paro community that generating source and then building was a really bad idea and they should shelve it and now someone is trying to bring it back. No, no, and thrice times, no.

where st2c should generate the .c file for FooPlugin from its Slang code in its package or from a .st file. There is no need for .c file to be human readable or be subject to version control. That will be very messy.

_yes_. _there_. _is_.

Regards .. Subbu

Hi Ben,

On Jun 3, 2017, at 2:22 AM, Ben Coman btc@openinworld.com wrote:

...

On Sat, Jun 3, 2017 at 5:57 AM, Eliot Miranda eliot.miranda@gmail.com wrote:

...

On Fri, Jun 2, 2017 at 9:54 AM, K K Subbu kksubbu.ml@gmail.com wrote:

...

On Friday 02 June 2017 04:50 PM, Holger Freyther wrote:

...

+1. I would go further and propose that plugins, other than those which are integral to VM[1], be moved into separate projects and built in their own tree with no references to vm source paths. They should be free to organize their platform-specific and platform-independent code and makefiles.

why? How many plugins are used that don't ship with the pharo-vm (or squeakvm)? I have to say I like the pharo-vm approach a lot where the VMMaker* packages and generated sources sit next to each other.

My proposal is to loosen build time coupling, not packaging. Nothing prevents plugins built out of tree from being built, installed and shipped together. Supporting out of tree plugins allows others to extend a VM even long after its release.

Eliot has already responded to your example in detail, so I will not dwell upon it further, except to point out that we need a command line tool to generate a .c file directly from the plugin's mcz file. Then we can create a Make rule to compile a .o and get rid of .c file. e.g.

FooPlugin.o : FooPlugin.mcz squeak -headless vmmaker.image gencc.st FooPlugin.mcz cc -c ... FooPlugin.c rm -f FooPlugin.c

Over my dead body. The make must *NEVER* be done like this. The problems are many:

• the VM so produced is undebuggable. There is no source for FooPlugin.c; it has been deleted
• the build is slow; building a VMMaker image (the necessary prerequisite for st2c) takes a long time, mush longer than the st2c step
• unless FooPlugin.mcz is versioned then the VM so produced is unversioned and if FooPlugin.mcz /is/ versioned then there's no reason why the plugin can't be produced using the normal workflow, generate C source, check it in to opensmalltalk/vm, build, and that way we have a versioned entity and break the dependency of builds requiring VMMaker

And I remember it mentioned a while ago, the bug is sometimes in the generation-code, so you want historical access to both mcz and generated-C to help debug that.

+1

...

Look, I just spent several years persuading the Paro community that generating source and then building was a really bad idea and they should shelve it and now someone is trying to bring it back. No, no, and thrice times, no.

I think Pharo has not completely abandoned generating C-sources, although its now a sideline activity...

It's a CI activity to test the current state of the VMMaker package. This is useful.

...

On Wed, May 31, 2017 at 2:48 PM, Esteban Lorenzano estebanlm@gmail.com wrote:

In any case, in the “pharo side” of things, we keep a parallel CI process who does this:

1. there is a development branch that keep all sources we need: the osvm *and* all monticello packages (filetree format) acting as a mirror that merges changes and executes all building process (including generating sources) and testing (we run all pharo tests as a way to test the VM). This is done just for being sure the CI is green and no artifact is created. This is how I have early reported problem in VM generation before.
2. in parallel the build follows the “normal” process on opensmalltalk-vm and everything is build there. This generates a “latest” build which is published in pharo servers so people can test (but this binary could be wrong as is not something declared as “stable”

I do wonder from time to time is why the C-generation can't be done as part of an opensmalltalk-vm CI process? The manual process is a bit opaque, and a CI process would be better than documentation. One possibility of doing it via CI could be generating from both Squeak and Pharo and checking the output is identical (assuming that might be useful).

Yes this would be useful. Also there's still at least one place in the CoInterpreter source that instability in the type inference code that flips the type of a variable between sqInt & usqInt and automation generation might help track that down.

IIRC Eliot didn't want the generated-C updated in the Cog branch for every VMMaker commit.

Right. No point doing so for changes to simulation only code, or to experimental code. Or on every commit to a package that contains a translated primitive, a form of primitives we need to eliminate by rewriting as conventional ones.

But the real issue is in not committing changes to generated code that hasn't changed. See scripts/revertUnchangedPlugins.

If there is, say, a Slang change that has limited effects (only a subset of plugins, or only the interpreter file, etc) we don't want to commit new versions of the essentially unchanged code because it obscures which commit leads to a bug.

Plus there's your observation about Slang bugs. I want to eyeball the changes in at least one representative generated file to look for unintended consequences and assure myself that source generation is sane.

A way to conform to this would be the CI updating the generated C in a side branch, and only when deemed worthy it could be integrated into the Cog branch via a simple merge rather than a one-time manual generation run.

Eliot, If you're receptive to this idea, I'd be interested in working on this. What concerns would you have with it?

- the process identifies files that only change in version stamp (if essentially unchanged) and /not/ commit this subset. Right now that happens with a script on plugins (cuz they're relatively simple) and by my VMMaker image that tracks commits only producing source for classes whose timestamps have changed

- that the process maintains the separation between generation followed by check in from build from checked in source.

- that people who work on the Smalltalk side of the business not get lazy in eyeballing the generation process cuz Slang issues can be subtle and tedious to track down. IME, human supervision remains useful here

cheers -ben

...

...

where st2c should generate the .c file for FooPlugin from its Slang code in its package or from a .st file. There is no need for .c file to be human readable or be subject to version control. That will be very messy.

_yes_. _there_. _is_.

...

Regards .. Subbu

-- _,,,^..^,,,_ best, Eliot

Hi Subbu,

On Jun 3, 2017, at 9:47 AM, K K Subbu kksubbu.ml@gmail.com wrote:

...

On Saturday 03 June 2017 03:27 AM, Eliot Miranda wrote:

Over my dead body. The make must *NEVER* be done like this. The problems are many:

Ouch! :-). Please don't take my example too literally. Smalltalk does not have a concept of a file-based compilation unit like C, so I had to use an extension to denote a collection of st code that goes into generating a .c compilation unit. I may well have used a .st or .zip for source code.

...

• the VM so produced is undebuggable. There is no source for

FooPlugin.c; it has been deleted

This is not the first time C-code gets generated on the fly. This technique has been used by Lex, Yacc etc. since early 70s. Bison generates a FSM interpreter in C from a Yacc spec. My example makefile segment deleted the file, but it can also cache it for debugging.

Caching it is not good enough. If there are bugs in generation then there's no guarantee that regenerating it at a later time produces the same source. Since the generated source contains time and version stamps it won't be the same anyway.

The inputs to any production vm /must/ be fully versioned at the point the source is consumed by the production compilation system.

The time taken to rebuild after a change should be as short as possible to keep debugging productivity high. Mandating a translation step defeats this.

...

• the build is slow; building a VMMaker image (the necessary

prerequisite for st2c) takes a long time, mush longer than the st2c step

The image is just a smalltalk app that translates slang to C. It does not have to be regenerated in every build anymore than gcc or cmake. In earlier posts, I had floated the idea of a bootstrap VM - a st machine with a built-in image to build and launch larger VMs. It can even be a separate project.

Sure. This is also what my work image is.

The .st code has to be translated to C at some time. If it is done within a single VM, then we won't be able to exploit multi-cores. With an st2c app, we can translate in parallel for a quicker build.

That could be nice.

...

• unless FooPlugin.mcz is versioned then the VM so produced is

unversioned and if FooPlugin.mcz /is/ versioned then there's no reason why the plugin can't be produced using the normal workflow, generate C source, check it in to opensmalltalk/vm, build, and that way we have a versioned entity and break the dependency of builds requiring VMMaker

Anyway, my point is that only the smalltalk "compilation unit" should be under version control. If a .c file is generated from it, there is no need to subject it to version control. It can continue to remain in a cache. Like the way we preserve .i (preprocessed) or .s (assembly) files for debugging .c code but never check-in these i. or .s files.

And my point is that input source to compilers producing the VM (e.g. .c .h .cpp .m and .s) whether generated or manually prepared /must/ be versioned, and must change only when changes are made to them (e.g. regenerated plugin source is only checked in when either the Plugin Smalltalk has changed or Slang has changed in a way that affects the plugin).

Okay, I had no idea I was stepping into a minefield of raw nerves here ;-). I will step back from these threads and do some prototyping and come up with specific code changes. The code seems to have grown too large for a lone hobbyist but it is still worth a try.

Supporting hobby projects, or rather, keeping the system open for people to add plugins, platforms, new forms of garbage collection, new vm configurations such as packaging as a DLL, or what ever else is /absolutely/ something we should support and encourage. But that doesn't mean throwing away reproducibility and versioning. It probably implies much better documentation. But things are in a good state to improve:

- anyone can clone the opensmalltalk/vm repository

- there is a script in images to build a current VMMaker image so anyone can easily produce a VMMaker image in which to simulate, author and generate. And the generation is fine to the right place in the opensmalltalk/v clone for it to be committed when ready

- because generation and check-in are separate steps one can get all the bits together (e.g. plugin generated under src/plugins, plugins.int or plugins.ext files updated on relevant platforms, special plugin majefiles containing support library references, etc) and make one commit instead of many noisy failing intermediate steps

So perhaps take a step back and ask what is the support you need, both in terms of documentation, code and infrastructure, that will help you experiment.

Regards .. Subbu

best, Eliot