On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
frank
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Levente
frank
On 30 December 2013 19:14, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Does RFB get used in a _headless_ manner? An alternative would be to add a new option on which I can switch. Maybe -exec? Then I can add a SmalltalkImage >> #isReallyHeadlessAsInHasNoGuiAtAllUnlikeRfb, and switch off that. We'd then invoke the image, in a CLI-friendly way, as
squeak -exec Foo.image the-script.st
Except that you can't just do that, because "Unknown option encountered!" and then a dialog pops up asking you to locate the image you want to run.
frank
Levente
frank
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 19:14, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Does RFB get used in a _headless_ manner? An alternative would be to
It depends. RFB can be used with -vm-display-null, or one can use xpra and use -vm-display-x11. We start our images with xpra, but still connect to them using RFB, because it's more responsive. xpra is still useful, because it's not safe to press alt+. through RFB, because you can interrupt the RFB process itself.
add a new option on which I can switch. Maybe -exec? Then I can add a SmalltalkImage >> #isReallyHeadlessAsInHasNoGuiAtAllUnlikeRfb, and switch off that. We'd then invoke the image, in a CLI-friendly way, as
squeak -exec Foo.image the-script.st
Except that you can't just do that, because "Unknown option encountered!" and then a dialog pops up asking you to locate the image you want to run.
Well, one can replace te ToolSet using a startup script after a successful startup. Being able to see the stack trace in case of a failure helps a lot. So I think you should push it to the Trunk.
Levente
frank
Levente
frank
On 30 December 2013 20:04, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 19:14, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Does RFB get used in a _headless_ manner? An alternative would be to
It depends. RFB can be used with -vm-display-null, or one can use xpra and use -vm-display-x11. We start our images with xpra, but still connect to them using RFB, because it's more responsive. xpra is still useful, because it's not safe to press alt+. through RFB, because you can interrupt the RFB process itself.
add a new option on which I can switch. Maybe -exec? Then I can add a SmalltalkImage >> #isReallyHeadlessAsInHasNoGuiAtAllUnlikeRfb, and switch off that. We'd then invoke the image, in a CLI-friendly way, as
squeak -exec Foo.image the-script.st
Except that you can't just do that, because "Unknown option encountered!" and then a dialog pops up asking you to locate the image you want to run.
Well, one can replace te ToolSet using a startup script after a successful startup. Being able to see the stack trace in case of a failure helps a lot. So I think you should push it to the Trunk.
I'll need to address Chris' bug report: I do need to set the ToolSet every time on image startup. That means that CommandLineToolSet >> #startUp: is possibly not the right place to put the switch.
frank
Levente
frank
Levente
frank
On Mon, Dec 30, 2013 at 09:04:05PM +0100, Levente Uzonyi wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 19:14, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Does RFB get used in a _headless_ manner? An alternative would be to
It depends. RFB can be used with -vm-display-null, or one can use xpra and use -vm-display-x11. We start our images with xpra, but still connect to them using RFB, because it's more responsive. xpra is still useful, because it's not safe to press alt+. through RFB, because you can interrupt the RFB process itself.
Our squeaksource.com image works this way. We run it as a headless image with -vm-display-null, and the image has RFBServer to support interactive access. It works very well, and I believe that this is a common configuration for Seaside application servers.
Dave
On 30 December 2013 21:33, David T. Lewis lewis@mail.msen.com wrote:
On Mon, Dec 30, 2013 at 09:04:05PM +0100, Levente Uzonyi wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 19:14, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Does RFB get used in a _headless_ manner? An alternative would be to
It depends. RFB can be used with -vm-display-null, or one can use xpra and use -vm-display-x11. We start our images with xpra, but still connect to them using RFB, because it's more responsive. xpra is still useful, because it's not safe to press alt+. through RFB, because you can interrupt the RFB process itself.
Our squeaksource.com image works this way. We run it as a headless image with -vm-display-null, and the image has RFBServer to support interactive access. It works very well, and I believe that this is a common configuration for Seaside application servers.
OK, so I misunderstood what "headless" meant. I thought it meant "doesn't use a GUI", but it really means "doesn't render a GUI to a physical device".
In that case, the commit's useless. I'll have to take a look at what the Pharo guys have done, and see how I can make that work.
What I need is to be able to treat the execution of an image as a strictly command-line process. I need to, without doing anything inside a startup script, to be able to signal to the image that it must use the CommandLineToolSet. The CLTS and DummyUIManager will then ensure that external interactions with the user happen only over stdout, stderr and std in.
Using SmalltalkImage >> #run: doesn't do what I need, because syntax errors are thrown outside the block, when the startup script is parsed. Although it might do the trick if we rewrote the way startup scripts are run. There's a bunch of wrongness in CodeLoader already (like parsing relative file paths relative to the image's location, rather than the current working directory).
Either way, it's much more work than I can stomach for the moment. I'll shelve this commit.
frank
Dave
One of the traditional challenges with code in .st scripts is that it's external to the IDE, so it rots a lot more quickly.
Which is why I ensure my .st scripts essentially fall into the pattern of sending just ONE message wrapped inside a Smalltalk run: (or variant). e.g., myscript.st would have, very simply:
Smalltalk run: [ : commandLineArg1 : arg2 : arg3 | SomeClass doWith: commandLineArg1 with: arg2 with: arg3 ]
That way they're so simple that syntax errors are never really an issue. Plus, as much of the code as possible is managed the IDE and Monticello.
Very recently I've gone one step further: I've begun putting even the *contents* of my .st scripts into the image too, inside Blocks whose code can be exported via #exportLinuxScripts! I have not submitted this hack to trunk (yet), but it means you *can't* have accidental syntax errors because the code-editor of the IDE catches it when you save the method. The other benefit already mentioned is that ALL senders, even the one in .st script, is recognized and managed in the IDE, and MC.
And so every image I deploy for a vertical purpose knows how to export the library of scripts needed to operate that purpose from the command-line.
On Tue, Dec 31, 2013 at 5:48 AM, Frank Shearar frank.shearar@gmail.com wrote:
On 30 December 2013 21:33, David T. Lewis lewis@mail.msen.com wrote:
On Mon, Dec 30, 2013 at 09:04:05PM +0100, Levente Uzonyi wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 19:14, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote: > >A new version of CommandLine was added to project The Inbox: >http://source.squeak.org/inbox/CommandLine-fbs.3.mcz > >==================== Summary ==================== > >Name: CommandLine-fbs.3 >Author: fbs >Time: 30 December 2013, 6:36:22.399 pm >UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 >Ancestors: CommandLine-fbs.2 > >If launched headless (with option -headless or -vm-display-null), use >the >CommandLineToolSet instead of the StandardToolSet. > >We must startUp: before AutoStart because AutoStart triggers the >processing of startup scripts (because it asks all registered >AbstractLaunchers - in particular, ProjectLauncher - to do their thing). > >=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Does RFB get used in a _headless_ manner? An alternative would be to
It depends. RFB can be used with -vm-display-null, or one can use xpra and use -vm-display-x11. We start our images with xpra, but still connect to them using RFB, because it's more responsive. xpra is still useful, because it's not safe to press alt+. through RFB, because you can interrupt the RFB process itself.
Our squeaksource.com image works this way. We run it as a headless image with -vm-display-null, and the image has RFBServer to support interactive access. It works very well, and I believe that this is a common configuration for Seaside application servers.
OK, so I misunderstood what "headless" meant. I thought it meant "doesn't use a GUI", but it really means "doesn't render a GUI to a physical device".
In that case, the commit's useless. I'll have to take a look at what the Pharo guys have done, and see how I can make that work.
What I need is to be able to treat the execution of an image as a strictly command-line process. I need to, without doing anything inside a startup script, to be able to signal to the image that it must use the CommandLineToolSet. The CLTS and DummyUIManager will then ensure that external interactions with the user happen only over stdout, stderr and std in.
Using SmalltalkImage >> #run: doesn't do what I need, because syntax errors are thrown outside the block, when the startup script is parsed. Although it might do the trick if we rewrote the way startup scripts are run. There's a bunch of wrongness in CodeLoader already (like parsing relative file paths relative to the image's location, rather than the current working directory).
Either way, it's much more work than I can stomach for the moment. I'll shelve this commit.
frank
Dave
On 31 December 2013 16:26, Chris Muller asqueaker@gmail.com wrote:
One of the traditional challenges with code in .st scripts is that it's external to the IDE, so it rots a lot more quickly.
Code rot happens because some Foo isn't used very often, and then the things that Foo use change. You only find out that Foo's broken when next you (very occasionally) run Foo and realise that you have to change stuff.
Which is why I ensure my .st scripts essentially fall into the pattern of sending just ONE message wrapped inside a Smalltalk run: (or variant). e.g., myscript.st would have, very simply:
Smalltalk run: [ : commandLineArg1 : arg2 : arg3 | SomeClass doWith: commandLineArg1 with: arg2 with: arg3 ]
That way they're so simple that syntax errors are never really an issue. Plus, as much of the code as possible is managed the IDE and Monticello.
... and you pollute the image state with a bunch of scaffolding. This is the biggest beef I have with Metacello: I don't want to assemble an image containing Metacello. Nor do I want to assemble an image containing the details of the script I want to run: it's just not interesting.
Very recently I've gone one step further: I've begun putting even the *contents* of my .st scripts into the image too, inside Blocks whose code can be exported via #exportLinuxScripts! I have not submitted this hack to trunk (yet), but it means you *can't* have accidental syntax errors because the code-editor of the IDE catches it when you save the method. The other benefit already mentioned is that ALL senders, even the one in .st script, is recognized and managed in the IDE, and MC.
I can see why _a developer_ might use the #run: stuff. It looks very convenient, and when you're hacking on your own stuff it looks just fine to use.
But let's look at CI scripts. Do you propose to put the Hudson stuff in the base 4.5 image, as well as all the paraphernalia around running tests in a CI environment? And then when we have other script-y things that we need - we shove those in the image too?
Ideally ReleaseBuilder wouldn't be in the image, for instance: because it's just scaffolding nonsense.
And so every image I deploy for a vertical purpose knows how to export the library of scripts needed to operate that purpose from the command-line.
Yes, "vertical" is the key word here: #run: and in-image scripts are great when you're working with an image tailored for a specific purpose.
But I want to act on images that are built for a _general_ purpose.
frank
On Tue, Dec 31, 2013 at 5:48 AM, Frank Shearar frank.shearar@gmail.com wrote:
On 30 December 2013 21:33, David T. Lewis lewis@mail.msen.com wrote:
On Mon, Dec 30, 2013 at 09:04:05PM +0100, Levente Uzonyi wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
On 30 December 2013 19:14, Levente Uzonyi leves@elte.hu wrote:
On Mon, 30 Dec 2013, Frank Shearar wrote:
>On 30 December 2013 18:36, commits@source.squeak.org wrote: >> >>A new version of CommandLine was added to project The Inbox: >>http://source.squeak.org/inbox/CommandLine-fbs.3.mcz >> >>==================== Summary ==================== >> >>Name: CommandLine-fbs.3 >>Author: fbs >>Time: 30 December 2013, 6:36:22.399 pm >>UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 >>Ancestors: CommandLine-fbs.2 >> >>If launched headless (with option -headless or -vm-display-null), use >>the >>CommandLineToolSet instead of the StandardToolSet. >> >>We must startUp: before AutoStart because AutoStart triggers the >>processing of startup scripts (because it asks all registered >>AbstractLaunchers - in particular, ProjectLauncher - to do their thing). >> >>=============== Diff against CommandLine-fbs.2 =============== > > >This allows headless things to report bad things happening, like >dumping syntax error notifications to stdout and exiting.
People who use RFB won't like this, because it quits the image on the first error.
Does RFB get used in a _headless_ manner? An alternative would be to
It depends. RFB can be used with -vm-display-null, or one can use xpra and use -vm-display-x11. We start our images with xpra, but still connect to them using RFB, because it's more responsive. xpra is still useful, because it's not safe to press alt+. through RFB, because you can interrupt the RFB process itself.
Our squeaksource.com image works this way. We run it as a headless image with -vm-display-null, and the image has RFBServer to support interactive access. It works very well, and I believe that this is a common configuration for Seaside application servers.
OK, so I misunderstood what "headless" meant. I thought it meant "doesn't use a GUI", but it really means "doesn't render a GUI to a physical device".
In that case, the commit's useless. I'll have to take a look at what the Pharo guys have done, and see how I can make that work.
What I need is to be able to treat the execution of an image as a strictly command-line process. I need to, without doing anything inside a startup script, to be able to signal to the image that it must use the CommandLineToolSet. The CLTS and DummyUIManager will then ensure that external interactions with the user happen only over stdout, stderr and std in.
Using SmalltalkImage >> #run: doesn't do what I need, because syntax errors are thrown outside the block, when the startup script is parsed. Although it might do the trick if we rewrote the way startup scripts are run. There's a bunch of wrongness in CodeLoader already (like parsing relative file paths relative to the image's location, rather than the current working directory).
Either way, it's much more work than I can stomach for the moment. I'll shelve this commit.
frank
Dave
One of the traditional challenges with code in .st scripts is that it's external to the IDE, so it rots a lot more quickly.
Code rot happens because some Foo isn't used very often, and then the things that Foo use change. You only find out that Foo's broken when next you (very occasionally) run Foo and realise that you have to change stuff.
With the .st script code compiled in the image, you would discover Foo when you browsed senders while making your changes, long before you run Foo.
By maintaining chunks of Smalltalk in external editors, you didn't even know you had a _syntax error_ until after an hour or something whatever you said..
Which is why I ensure my .st scripts essentially fall into the pattern of sending just ONE message wrapped inside a Smalltalk run: (or variant). e.g., myscript.st would have, very simply:
Smalltalk run: [ : commandLineArg1 : arg2 : arg3 | SomeClass doWith: commandLineArg1 with: arg2 with: arg3 ]
That way they're so simple that syntax errors are never really an issue. Plus, as much of the code as possible is managed the IDE and Monticello.
... and you pollute the image state with a bunch of scaffolding. This is the biggest beef I have with Metacello: I don't want to assemble an image containing Metacello. Nor do I want to assemble an image containing the details of the script I want to run: it's just not interesting.
I'm just talking about having clean entry-points from the OS command-line into the Smalltalk system. And, not a "bunch." Just 1 short method per script. The format of the method is like this:
scriptName ^ './vm/bin/squeak -mmap 1000m -vm display=none my.image scriptName.st $1 $2 $3 $4 >> scriptName.log 2>>scriptName.err' -> [ Smalltalk runAndQuit: [ : source : target : user : pw | SSRepository copyAllFrom: source to: target user: user password: pw ] ]
I write this method in the image in a particular category and run #exportLinuxScripts causes two files to be written. One called 'scriptName' which contains that command-line, the other called "scriptName.st", which contains the code inside that block.
So, it documents, "how the hell do I run this sucker?" right in the image (including how much memory to allocate for the VM!), exposes it to the IDE, and allows deployment of up-to-date set of scripts by merely only deploying a new image.
Very recently I've gone one step further: I've begun putting even the *contents* of my .st scripts into the image too, inside Blocks whose code can be exported via #exportLinuxScripts! I have not submitted this hack to trunk (yet), but it means you *can't* have accidental syntax errors because the code-editor of the IDE catches it when you save the method. The other benefit already mentioned is that ALL senders, even the one in .st script, is recognized and managed in the IDE, and MC.
I can see why _a developer_ might use the #run: stuff. It looks very convenient, and when you're hacking on your own stuff it looks just fine to use.
But let's look at CI scripts. Do you propose to put the Hudson stuff in the base 4.5 image, as well as all the paraphernalia around running tests in a CI environment?
Of course not.
And then when we have other script-y things that we need - we shove those in the image too?
Just the bits that contain Smalltalk code. Maintaining that code in the image means it stays in sync with the image code it calls, and syntax errors cannot ever happen.
Ideally ReleaseBuilder wouldn't be in the image, for instance: because it's just scaffolding nonsense.
("pollute"? "shove"? "nonsense"? Frank I'm trying to help not strike a nerve..)
I think we need to stay with self-documenting release images. The purpose of ReleaseBuilder is to tell us how that release image was built. That's not nonsense.
In fact, when I had problems with the CI image, it really brought into focus the trouble of having to "trust" the image we ultimately release. I had no idea what was in that image, nor how it was built. I couldn't possibly deploy it.
And so every image I deploy for a vertical purpose knows how to export the library of scripts needed to operate that purpose from the command-line.
Yes, "vertical" is the key word here: #run: and in-image scripts are great when you're working with an image tailored for a specific purpose.
But I want to act on images that are built for a _general_ purpose.
How can we know the exact ST code of how those general-purpose images are built?
CI is a vertical application -- it 1) runs test cases against latest code and 2) reports results. As a tertiary responsibility, we're investigating whether it should 3) be used to build release images. If _all_ the Smalltalk code which builds the release can stay with the image which has withstood the scrutiny of the trunk process, and CI would only "invoke" it, I think it would go a long way toward trusting CI for building release images.
Any individual should be able to "make" an equivalent 4.5 solely from a 4.4, via code visible in the image.
On 31 December 2013 20:26, Chris Muller asqueaker@gmail.com wrote:
One of the traditional challenges with code in .st scripts is that it's external to the IDE, so it rots a lot more quickly.
Code rot happens because some Foo isn't used very often, and then the things that Foo use change. You only find out that Foo's broken when next you (very occasionally) run Foo and realise that you have to change stuff.
With the .st script code compiled in the image, you would discover Foo when you browsed senders while making your changes, long before you run Foo.
By maintaining chunks of Smalltalk in external editors, you didn't even know you had a _syntax error_ until after an hour or something whatever you said..
Which is why I ensure my .st scripts essentially fall into the pattern of sending just ONE message wrapped inside a Smalltalk run: (or variant). e.g., myscript.st would have, very simply:
Smalltalk run: [ : commandLineArg1 : arg2 : arg3 | SomeClass doWith: commandLineArg1 with: arg2 with: arg3 ]
That way they're so simple that syntax errors are never really an issue. Plus, as much of the code as possible is managed the IDE and Monticello.
... and you pollute the image state with a bunch of scaffolding. This is the biggest beef I have with Metacello: I don't want to assemble an image containing Metacello. Nor do I want to assemble an image containing the details of the script I want to run: it's just not interesting.
I'm just talking about having clean entry-points from the OS command-line into the Smalltalk system. And, not a "bunch." Just 1 short method per script. The format of the method is like this:
scriptName ^ './vm/bin/squeak -mmap 1000m -vm display=none my.image scriptName.st $1 $2 $3 $4 >> scriptName.log 2>>scriptName.err' -> [ Smalltalk runAndQuit: [ : source : target : user : pw | SSRepository copyAllFrom: source to: target user: user password: pw ] ]
I write this method in the image in a particular category and run #exportLinuxScripts causes two files to be written. One called 'scriptName' which contains that command-line, the other called "scriptName.st", which contains the code inside that block.
So, it documents, "how the hell do I run this sucker?" right in the image (including how much memory to allocate for the VM!), exposes it to the IDE, and allows deployment of up-to-date set of scripts by merely only deploying a new image.
Very recently I've gone one step further: I've begun putting even the *contents* of my .st scripts into the image too, inside Blocks whose code can be exported via #exportLinuxScripts! I have not submitted this hack to trunk (yet), but it means you *can't* have accidental syntax errors because the code-editor of the IDE catches it when you save the method. The other benefit already mentioned is that ALL senders, even the one in .st script, is recognized and managed in the IDE, and MC.
I can see why _a developer_ might use the #run: stuff. It looks very convenient, and when you're hacking on your own stuff it looks just fine to use.
But let's look at CI scripts. Do you propose to put the Hudson stuff in the base 4.5 image, as well as all the paraphernalia around running tests in a CI environment?
Of course not.
And then when we have other script-y things that we need - we shove those in the image too?
Just the bits that contain Smalltalk code. Maintaining that code in the image means it stays in sync with the image code it calls, and syntax errors cannot ever happen.
Ideally ReleaseBuilder wouldn't be in the image, for instance: because it's just scaffolding nonsense.
("pollute"? "shove"? "nonsense"? Frank I'm trying to help not strike a nerve..)
I'm not using these words to rile you up. I just like calling a spade a spade.
I think we need to stay with self-documenting release images. The purpose of ReleaseBuilder is to tell us how that release image was built. That's not nonsense.
The Squeak user doesn't care about ReleaseBuilder. The trunk developer doesn't care about ReleaseBuilder. Only one person cares about ReleaseBuilder, and that's the release manager. And that's once per release cycle. It's scaffolding. It's a one-off janitorial task that, other than at the end of a release cycle, is deadweight in the image.
I didn't say that _ReleaseBuilder_ was nonsense. I said _it being in the base image_ was nonsense. I mean, come on, didn't your builders dismantle the scaffolding they used to build your house before you moved in? That scaffolding served a valuable purpose, but it's not part of the house!
In fact, when I had problems with the CI image, it really brought into focus the trouble of having to "trust" the image we ultimately release. I had no idea what was in that image, nor how it was built. I couldn't possibly deploy it.
I trust hand-rolled images far, far less than I do images produced by scripts. We can test the latter, at least.
Now I realise that the current CI base image has issues, and you know what? You know why? Precisely because it is _so hard_ to work with images from a command line. It doesn't have to be like this. It _shouldn't_ be like this.
The whole point of my work on automation is to _remove the human element_ from the process. "A little trust goes a long way. The less you have, the further you'll go" (Maxim 30 from The Seventy Maxims of Maximally Effective Mercenaries, http://www.schlockmercenary.com/2003-03-08)
And so every image I deploy for a vertical purpose knows how to export the library of scripts needed to operate that purpose from the command-line.
Yes, "vertical" is the key word here: #run: and in-image scripts are great when you're working with an image tailored for a specific purpose.
But I want to act on images that are built for a _general_ purpose.
How can we know the exact ST code of how those general-purpose images are built?
Because (a) it's the update stream and (b) the base image is built by a script, from a known base version, that can be debugged and reworked.
CI is a vertical application -- it 1) runs test cases against latest code and 2) reports results. As a tertiary responsibility, we're investigating whether it should 3) be used to build release images. If _all_ the Smalltalk code which builds the release can stay with the image which has withstood the scrutiny of the trunk process, and CI would only "invoke" it, I think it would go a long way toward trusting CI for building release images.
I've built CI stuff for Ruby, Smalltalk and C# code. I've used it with Java and Scala. Only one language on that list has been a pain in the axe every. single. step. of. the. way. And it's Smalltalk. Why? Because we have an obsession with hand-rolling hand-crafted binary artifacts lovingly assembled by hand. That's fine for an individual user's image. It's rubbish when you want to try and understand what this blob of stuff actually contains.
Any individual should be able to "make" an equivalent 4.5 solely from a 4.4, via code visible in the image.
An individual _should not have to_. They should be able to download a random CI-built image, knowing that it _works_ because the CI published the test results for that image. The image should be automatically built, tested and deployed by a completely deterministic build process.
Being able to update your image from some ancient thing is great, and we should preserve it, but _I do not want that_ as a means of building an artifact for the general public to consume.
But this discussion is just going to end up with you saying that unloading is how you get a minimal image and me saying that building up an image is how you get a full-fat image.
frank
I think we need to stay with self-documenting release images. The purpose of ReleaseBuilder is to tell us how that release image was built. That's not nonsense.
The Squeak user doesn't care about ReleaseBuilder. The trunk developer doesn't care about ReleaseBuilder. Only one person cares about ReleaseBuilder, and that's the release manager. And that's once per release cycle. It's scaffolding. It's a one-off janitorial task that, other than at the end of a release cycle, is deadweight in the image.
I didn't say that _ReleaseBuilder_ was nonsense. I said _it being in the base image_ was nonsense. I mean, come on, didn't your builders dismantle the scaffolding they used to build your house before you moved in? That scaffolding served a valuable purpose, but it's not part of the house!
Non sequitir, an image is not a house.
In fact, when I had problems with the CI image, it really brought into focus the trouble of having to "trust" the image we ultimately release. I had no idea what was in that image, nor how it was built. I couldn't possibly deploy it.
I trust hand-rolled images far, far less than I do images produced by scripts. We can test the latter, at least.
Either can be tested, but I never said hand-rolled, where did you get that?
Now I realise that the current CI base image has issues, and you know what? You know why? Precisely because it is _so hard_ to work with images from a command line. It doesn't have to be like this. It _shouldn't_ be like this.
That's why I tried to show you my method for managing the seam between the OS and Smalltalk; because it's simple with other benefits too.
The whole point of my work on automation is to _remove the human element_ from the process.
Yes, I want that too. YOU said I wanted to hand-roll images, not me.
And so every image I deploy for a vertical purpose knows how to export the library of scripts needed to operate that purpose from the command-line.
snip <<<
How can we know the exact ST code of how those general-purpose images are built?
Because (a) it's the update stream and (b) the base image is built by a script, from a known base version, that can be debugged and reworked.
Right, I just prefer that "script" to be a method in ReleaseBuilder so it is visible, trunk-maintained and scrutinized and recorded for posterity in our MC repositories.
CI is a vertical application -- it 1) runs test cases against latest code and 2) reports results. As a tertiary responsibility, we're investigating whether it should 3) be used to build release images. If _all_ the Smalltalk code which builds the release can stay with the image which has withstood the scrutiny of the trunk process, and CI would only "invoke" it, I think it would go a long way toward trusting CI for building release images.
I've built CI stuff for Ruby, Smalltalk and C# code. I've used it with Java and Scala. Only one language on that list has been a pain in the axe every. single. step. of. the. way. And it's Smalltalk. Why? Because we have an obsession with hand-rolling hand-crafted binary artifacts lovingly assembled by hand. That's fine for an individual user's image. It's rubbish when you want to try and understand what this blob of stuff actually contains.
YOU don't have that obsession, so why are you having pain?
I don't have that obsession either; I don't think anyone does but we've been slow to make our system self-building. I just wish you were more open to Squeak playing a role in its own advancement by putting the Smalltalk-portions of the upgrade script in the image rather than by some invisible text file out on the internet somewhere that isn't tracked or scrutinized.
Any individual should be able to "make" an equivalent 4.5 solely from a 4.4, via code visible in the image.
An individual _should not have to_.
I said "should be _able_ to". I totally agree with you about "should not have to."
They should be able to download a random CI-built image, knowing that it _works_ because the CI published the test results for that image. The image should be automatically built, tested and deployed by a completely deterministic build process.
What is "deployed?" To me that means, pushed to the location for public-consumption (e.g., ftp.squeak.org). If so, just the other week you said that the "backup copy" out on ftp.squeak.org should be verified manually.
http://lists.squeakfoundation.org/pipermail/squeak-dev/2013-December/175304.... http://lists.squeakfoundation.org/pipermail/squeak-dev/2013-December/175330....
I just read those again -- So, you want images to be built by code in GitHub. Really, that's fine, but _all I'm asking_ is to move lines 10-13 of release.st into the image, perhaps to the top of #prepareNewBuild. That's it!
(Note: Line 12 should be changed to use the in-image Form from, (MorphicProject class>>#defaultBackgroundForm)).
Being able to update your image from some ancient thing is great, and we should preserve it, but _I do not want that_ as a means of building an artifact for the general public to consume.>
But this discussion is just going to end up with you saying that unloading is how you get a minimal image and me saying that building up an image is how you get a full-fat image.
frank
On Tue, Dec 31, 2013 at 5:31 PM, Chris Muller asqueaker@gmail.com wrote:
Right, I just prefer that "script" to be a method in ReleaseBuilder so
it is visible, trunk-maintained and scrutinized and recorded for posterity in our MC repositories.
OK, let's make that into a more concrete proposal. Rather than have Smalltalk code being written with a text editor and versioned with Git, we could have that code be written with the Squeak IDE and versioned with Monticello. These CI packages would stored in a repository on source.squeak.org, but NOT the trunk repository, and would not be part of the trunk image. They'd be responsible for generating script files that could be passed to the image on the command line for execution. (Rather like VMMaker, in effect.)
Frank, I think that might actually make your life easier, without burdening the trunk image or release artifacts with scaffolding. It's also orthogonal to your proposal for activating CommandLineToolSet with some sort of parameter from the command line. (If we could accomplish that without breaking the has-a-Display-but-draws-it-to-the-network-instead-of-a-window deployment mode, that would be great!)
Colin
On 1 January 2014 07:22, Colin Putney colin@wiresong.com wrote:
On Tue, Dec 31, 2013 at 5:31 PM, Chris Muller asqueaker@gmail.com wrote:
Right, I just prefer that "script" to be a method in ReleaseBuilder so it is visible, trunk-maintained and scrutinized and recorded for posterity in our MC repositories.
OK, let's make that into a more concrete proposal. Rather than have Smalltalk code being written with a text editor and versioned with Git, we could have that code be written with the Squeak IDE and versioned with Monticello. These CI packages would stored in a repository on source.squeak.org, but NOT the trunk repository, and would not be part of the trunk image. They'd be responsible for generating script files that could be passed to the image on the command line for execution. (Rather like VMMaker, in effect.)
The Smalltalk part of the CI stuff is tiny. It's the HudsonReportTool or whatever it's called, and two tiny scripts, update-base-image.st and release.st. Release.st boils down, now, to "ReleaseBuilder prepareNewBuild" wrapped in ToolSet setting boilerplate. It needs to be outside the image because _something_ needs to get the ball rolling from outside the image.
The bulk of the CI scripts, if you look, consists of small load scripts for Smalltalk packages (OSProcess, Xtreams, FileSystem, ...), scripts to invoke tests on same, and then Ruby to wire everything together.
Frank, I think that might actually make your life easier, without burdening the trunk image or release artifacts with scaffolding. It's also orthogonal to your proposal for activating CommandLineToolSet with some sort of parameter from the command line. (If we could accomplish that without breaking the has-a-Display-but-draws-it-to-the-network-instead-of-a-window deployment mode, that would be great!)
Yep. It is more complicated than I'd hoped, though. We need to add a new flag to the VM/scripts that run the VM, or something. -headless needs to stay untouched by this new thing.
frank
Colin
Am 30.12.2013 20:45, schrieb Frank Shearar:
Does RFB get used in a _headless_ manner?
Yes on a Linux server without graphical UI running Aidaweb. I tunnel RFB through Putty. I'd love to keep that option.
Cheers
Herbert
An alternative would be to add a new option on which I can switch. Maybe -exec? Then I can add a SmalltalkImage >> #isReallyHeadlessAsInHasNoGuiAtAllUnlikeRfb, and switch off that. We'd then invoke the image, in a CLI-friendly way, as
One issue is it doesn't do the opposite when starting headful. If the headless image _saves_ and exits (a use-case I have now, today), then is relaunched headful later, the default ToolSet is not put back to the preferred headful tool.
I think setting the CommandLineToolSet needs to be an explicit action by the command-line user, whether from the .st script or by virtue of using SmalltalkImage>>#run: (or equivalents). Since that is part of the command-line interface so the user could expect command-line enhancements to be done.
I think startup: should only do implementation-level stuff -- e.g., unhibernating type activities. Domain-level stuff (e.g., preferences and other image domain objects) should be controlled via explicitly by the user, such as through command-line arguments and/or .st script invoking a top-level method.
On Mon, Dec 30, 2013 at 12:42 PM, Frank Shearar frank.shearar@gmail.com wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
frank
On 30 December 2013 20:27, Chris Muller asqueaker@gmail.com wrote:
One issue is it doesn't do the opposite when starting headful. If the headless image _saves_ and exits (a use-case I have now, today), then is relaunched headful later, the default ToolSet is not put back to the preferred headful tool.
Oh, I see what you mean. If you run headlessly, save & quit, and launch headfully, you'll still have the CommandLineToolSet. Nice catch.
I think setting the CommandLineToolSet needs to be an explicit action by the command-line user, whether from the .st script or by virtue of using SmalltalkImage>>#run: (or equivalents). Since that is part of the command-line interface so the user could expect command-line enhancements to be done.
Setting the ToolSet inside the .st script is a fail. I wasted hours yesterday trying to debug the CI's release.st because the script _when working_ takes an age to run, as it checks all packages for dirtiness. The problem is that there was a missing [ in the script, and the only way to find out that the script was _broken_ was to wait even longer.
So the point of the above commit is to ensure that _before the startup script is run_ (i.e., before AutoStart starts up), that the appropriate ToolSet is configured. Hence my attempt at signalling which ToolSet to use through a command line option.
frank
I think startup: should only do implementation-level stuff -- e.g., unhibernating type activities. Domain-level stuff (e.g., preferences and other image domain objects) should be controlled via explicitly by the user, such as through command-line arguments and/or .st script invoking a top-level method.
On Mon, Dec 30, 2013 at 12:42 PM, Frank Shearar frank.shearar@gmail.com wrote:
On 30 December 2013 18:36, commits@source.squeak.org wrote:
A new version of CommandLine was added to project The Inbox: http://source.squeak.org/inbox/CommandLine-fbs.3.mcz
==================== Summary ====================
Name: CommandLine-fbs.3 Author: fbs Time: 30 December 2013, 6:36:22.399 pm UUID: a8ce5a55-8bb5-ac4b-a112-eb308ce3bf53 Ancestors: CommandLine-fbs.2
If launched headless (with option -headless or -vm-display-null), use the CommandLineToolSet instead of the StandardToolSet.
We must startUp: before AutoStart because AutoStart triggers the processing of startup scripts (because it asks all registered AbstractLaunchers - in particular, ProjectLauncher - to do their thing).
=============== Diff against CommandLine-fbs.2 ===============
This allows headless things to report bad things happening, like dumping syntax error notifications to stdout and exiting.
frank
squeak-dev@lists.squeakfoundation.org
-
Chris Muller -
Colin Putney -
David T. Lewis -
Frank Shearar -
Herbert König -
Levente Uzonyi