VM/image backwards compatibility (& high dpi) [was Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn]
Hi Jaromir,
On May 11, 2022, at 5:49 AM, Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas mail@jaromir.net wrote: Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the new primitive.
Thanks again; this brings me to another question: if you go simpler, i.e. with the ifFalse clause only, it would mean the method wouldn't work correctly with older VMs with the older prim 88 suspend semantics. I was under the impression to keep compatibility with older VMs but maybe it's a nonsensical assumption on my part (The reason I'm asking is the #terminate I wrote is compatible with both the new and old VMs - but could be simplified a bit if the old VMs were not supposed to be used with the new images).
That’s the way backwards compatibility works, with images and VMs in the same way as binaries and processors. A VM is expected to be backwards compatible with all older images of the same basic version (eg the Spur format, the V3 format). So all Spur 4.6, 5.x & current 6.x images should run on the current tip VM. But a new image may expect a feature added to a contemporary VM that is absent in previous versions of the VM.
This is analogous to, for example, binaries compiled against newer instructions added to the x86 such as mmx, sse 1, 2 & 3. We expect that the new processor can run older binaries; we do not expect new binaries to run on older processors unless we compile them specially.
Now, at a major release we may decide to remove VM features, but at that point we would change the VM’s image format number so that older images do not load. Since this effectively introduces a new VM variant it is not something we do lightly and do only when compelling, as the performance and functionality improvements obtained from the Spur architecture are over V3.
We’re currently thinking about this with high dpi support. Can we afford to provide backwards compatibility with older “high dpi unaware” (HDU) images or not? We can either freeze Spur VM support for older images at around March this year, and drop support for HDU images in the platform sources, or we can add a flag bit to the image flags in the latest VMs which if set identifies an image as “high dpi aware” (HDA), and set the bit, eg in a Monticello package load script. And then the VM can remain backwards compatible by testing the HDA bit, behaving like older VMs if unset.
It is desirable to maintain backwards compatibility if we can afford the engineering effort. But note that considerable engineering effort must be extended either way. For example, the Virtend application would require considerable work to migrate to HDA, being HDU currently.
This is a structural issue. Putting the effort into the HDA VM to provide backwards compatibility with HDU images supports all older HDU applications. Not putting the effort means that all older HDU applications must put effort in to move forwards. (This is because the VM is the platform supporting applications above it, just as the trunk image is a platform that supports applications built within it). Hence it’s my preference to put the effort into the VM (selfishly as I’m committed to keeping the Virtend application running).
What kinds of effort are we talking about? It’s not just having a flag to test and having the VM answer/respond appropriately for HDU if unset and for HDA if set. The flag needs to be tested very early in start up, potentially in the current sources before the image has been loaded. Eg on Linux one can supply a command line argument specifying the display subsystem to use which comes before the image name. So command line parsing must be able to make a pass through the arguments to find the image, locate the flags in the header, determine the value of the HDA flag, and then go back and actually process the command line arguments. Tedious work ;-)
Thanks,
Jaromir
-- Jaromír Matas mail@jaromir.net
From: Eliot Miranda Sent: Tuesday, May 10, 2022 23:21 To: The general-purpose Squeak developers list Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas mail@jaromir.net wrote: Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the new primitive.
Thanks again, Jaromir
--
Jaromír Matas
+420 777 492 777 mail@jaromir.net
From: Eliot Miranda Sent: Tuesday, May 10, 2022 21:17 To: The general-purpose Squeak developers list Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Hi Jaromir,
On Tue, May 10, 2022 at 8:31 AM Jaromir Matas mail@jaromir.net wrote: Hi Eliot, Nicolas,
(Eliot) Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the primitive most convenient. Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive.
(Nicolas) Yes, I propose the selector
Process>>suspendAndUnblock <primitive: 78> snip... Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent). The question remaining is if we really need two new primitives 578 (V1) and 588 (V2). It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
(Eliot) +1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Thank you, I wasn't sure how to proceed with this :) Do I understand correctly #suspend will have the new suspend semantics (either V1 or preferably V2) in new images (new release) and the previous semantics (via prim 88) will be available for backward compatibility (or special usage)?
Yes, exactly. In that case the tests I submitted in KernelTests-jar.421 should work without changes...
Cool!
The next question is what to do with: Process >> suspendPrimitivelyOrFail "Test support. Execute primitive 88, or fail."
<primitive: 88> ^self primitiveFailedChange it or remove it? Replace with Nicolas’s Process>>suspendAndUnblock ?
Well it appears to be used only in testAtomicSuspend. So I would change it to use the same primitive number as suspend, which I guess is 578, primitiveSuspendBackingUpV2, which has the more consistent semantics.
Thanks!
Best,
Jaromir
From: Eliot Miranda Sent: Monday, May 9, 2022 23:23 To: The general-purpose Squeak developers list Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On May 9, 2022, at 1:04 PM, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:
Hi Eliot, Le lun. 9 mai 2022 à 20:36, Eliot Miranda eliot.miranda@gmail.com a écrit : Hi Jaromir, Hi Nicolas,
On Sun, May 8, 2022 at 8:17 AM Jaromir Matas mail@jaromir.net wrote: Hi Nicolas,
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88).
Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88). It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
Yes, thanks, well that was the original idea for Smalltalk processSuspensionUnblocks to answer about the behavior of the primitiveSuspend #88; which was before the primitives #568 and #578 were introduced. (And which was when I wrote the tests using this original logic). However that approach ran into problems with some existing implementations (e.g. Virtend) and as a result the two new primitives were introduced (#568 and #578) - and apparently the logic of what Smalltalk processSuspensionUnblocks answers about changed. Since then I haven't heard from Eliot about his plans how to proceed with the new suspend semantics or whether this is it :)
Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the primitive most convenient. Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive. Yes, I propose the selector Process>>suspendAndUnblock <primitive: 78> snip... Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent). The question remaining is if we really need two new primitives 578 (V1) and 588 (V2). It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
+1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Sorry about not renaming processSuspensionUnblocks. This should be something like Smalltalk hasExtendedProcessSuspensionSemantics or some such.
It's not too late, there are not many usages, except inbox experimentations.
In case this is still WIP I'd quite like an approach similar to Smalltalk processPreemptionYields - you'd set Smalltalk processSuspensionUnblocks true or false depending on how you want the VM to behave and the VM would use the right semantics (unless this is too naive :) ).
Regarding the tests failing with the newest VM: testAtomicSuspend - we can remove this updated version and leave the existing one for the moment testRevisedSuspendExpectations - we can leave this one out too testTerminateBlockedInNestedEnsure1 - I'll take a look at these two and try to adjust the logic testTerminateBlockedInNestedEnsure2
All remaining test in KernelTests-jar.421 work as intended.
Thanks again, Jaromir
--
Jaromír Matas
mail@jaromir.net
From: Nicolas Cellier Sent: Sunday, May 8, 2022 16:22 To: The general-purpose Squeak developers list Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Le dim. 8 mai 2022 à 15:59, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com a écrit :
Le dim. 8 mai 2022 à 14:16, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com a écrit : Hi Jaromir
Le sam. 7 mai 2022 à 14:45, Jaromir Matas mail@jaromir.net a écrit : Hi Marcel, Nicolas, Eliot,
(Nicolas) I have merged Kernel-jar.1446 which is trivial (re-signal an Exception).
Thanks!
(Marcel) Hmm... #testSimpleResignalVsOuter1 is still failing.
In my latest trunk image it passes ok (5/7/22)
(Marcel) I will also take a look at KernelTests-jar.421 to check whether any new semantics are okay.
(Nicolas) Yes, I did that and got two failing tests testTerminateTerminatingProcess testResumeTerminatingProcess
both failures look the same, in second self assert: terminator isSuspended.
Yes, they are supposed to fail at the moment so I suggest to make them expected failures/feature requests. However there's a stupid bug in both I’ve fixed now: they were indeed supposed to fail the following assertion: ``` self should: [terminatee terminate] raise: Error. ```
Apologies for the confusion - an updated changeset is enclosed.
However, there's another issue regarding the revised #suspend semantics Eliot has been working on. I've updated the process tests in KernelTests-jar.421 to test both the old and the new #suspend semantics. The two semantics should be distinguishable via Smalltalk processSuspensionUnblocks flag answering true in case of the old semantics and false in case of the revised one; my updated tests use this logic. However, unfortunately the latest VM answers "false" but uses the OLD suspend semantics in #suspend prim 88.
So I'm surprised you haven't observed more tests failing due to the new suspend semantics... What VM have you used – an older one? I'm on the latest trunk's VM 3183. And what is the answer of Smalltalk processSuspensionUnblocks? :) I'm utterly confused...
Hi Eliot - I may be confused here but if the current VM uses the old #suspend prim 88 semantics, shouldn't ```Smalltalk processSuspensionUnblocks``` answer true?
Indeed, I can confirm that the latest VM does answer false to (Smalltalk processSuspensionUnblocks). Though, process suspension still unblocks as shown by the example at https://github.com/pharo-project/pharo/issues/10669
A bit more simply, this answers false - it shouldn't:
s := Semaphore new. p := [s wait] newProcess. p resume. 100 milliSeconds wait. p suspend; resume. 100 milliSeconds wait. p isTerminated = Smalltalk processSuspensionUnblocks.
technically, this answers false - it shouldn't:
s := Semaphore new. p := [s wait] newProcess. p resume. 100 milliSeconds wait. p suspend == s = Smalltalk processSuspensionUnblocks
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88). Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88). It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
The new Behavior is implemented by primitiveSuspendBackingUpV1 (#578) and primitiveSuspendBackingUpV2 (#588). V1 always answers the old list (even if a Semaphore/Mutex), V2 answers nil in case of Semaphore/Mutex.
So, only ((Process>>#suspend) primitive) can give you a clue about the behavior, assuming that the primitive in question is implemented...
So it depends if we want to make it a Preference, or make the image compatible with some less capable VM...
we could implement Process>>primitiveSuspendAndUnblock as <primitive: 88> Process>>primitiveSuspend as <primitive: 588> test the VM flag in suspend Process>>suspend ^(VMHasPrimitiveSuspendBackingUp ifNil: [VMHasPrimitiveSuspendBackingUp := Smalltalk processSuspensionUnblocks not]) ifTrue: [self primitiveSuspend] ifFalse: [self primitiveSuspendAndUnblock] Then arrange to reset VMHasPrimitiveSuspendBackingUp to nil at snapshot...
I will tell next week about the status of the VM which I used for testing, it's possibly a few months old.
(Nicolas) I would also consider 1445 1421 and most importantly 1447.
Regarding 1421: alternatively, you might consider a newer version 1415 where the only difference is #return:from: passes a block instead of nil to #resume:through: to cause a fresh search for the first unwind context; otherwise they are equivalent.
OK, I will check again.
(Marcel) Maybe we can just merge
KernelTests-jar.393 KernelTests-jar.418 KernelTests-jar.421
And go from there? Or are there any objections?
KernelTests-jar.393 is just a special case of a more general KernelTests-jar.418 test…
In my image all tests are green provided Smalltalk processSuspensionUnblocks answers true and the two above tests Nicolas mentioned are marked expected failures :)
Thanks very much for your feedback, Jaromir
--
Jaromír Matas
mail@jaromir.net
-- _,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
-- _,,,^..^,,,_ best, Eliot
Hi Eliot --
Can we afford to provide backwards compatibility with older “high dpi unaware” (HDU) images or not?
Similarly to 32-bit vs. 64-bit, it might as well the a responsibility of the underlying platform/OS to provide such compatibility.
Microsoft Windows shines here, still being able to run 32-bit and 64-bit applications. Still being able to run applications in "Low resolution" mode.
Apple macOS is a bummer here, not being able to run 32-bit applications. Not being able to even provide a "Low resolution" mode in macOS 12.
If we can avoid to tag images as "compatible" or "incompatible" to a certain VM regarding a specific feature, we are better off regarding maintenance cost.
In my opinion, it makes to sense *not* being able to open, e.g., Squeak 5.3 with the newest VM even if that image has no (good) support for high-resolution displays. There will always be measures to work around that. It's just inconvenient. Not broken.
Best, Marcel Am 11.05.2022 16:27:34 schrieb Eliot Miranda eliot.miranda@gmail.com: Hi Jaromir,
On May 11, 2022, at 5:49 AM, Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the new primitive.
Thanks again; this brings me to another question: if you go simpler, i.e. with the ifFalse clause only, it would mean the method wouldn't work correctly with older VMs with the older prim 88 suspend semantics. I was under the impression to keep compatibility with older VMs but maybe it's a nonsensical assumption on my part (The reason I'm asking is the #terminate I wrote is compatible with both the new and old VMs - but could be simplified a bit if the old VMs were not supposed to be used with the new images).
That’s the way backwards compatibility works, with images and VMs in the same way as binaries and processors. A VM is expected to be backwards compatible with all older images of the same basic version (eg the Spur format, the V3 format). So all Spur 4.6, 5.x & current 6.x images should run on the current tip VM. But a new image may expect a feature added to a contemporary VM that is absent in previous versions of the VM.
This is analogous to, for example, binaries compiled against newer instructions added to the x86 such as mmx, sse 1, 2 & 3. We expect that the new processor can run older binaries; we do not expect new binaries to run on older processors unless we compile them specially.
Now, at a major release we may decide to remove VM features, but at that point we would change the VM’s image format number so that older images do not load. Since this effectively introduces a new VM variant it is not something we do lightly and do only when compelling, as the performance and functionality improvements obtained from the Spur architecture are over V3.
We’re currently thinking about this with high dpi support. Can we afford to provide backwards compatibility with older “high dpi unaware” (HDU) images or not? We can either freeze Spur VM support for older images at around March this year, and drop support for HDU images in the platform sources, or we can add a flag bit to the image flags in the latest VMs which if set identifies an image as “high dpi aware” (HDA), and set the bit, eg in a Monticello package load script. And then the VM can remain backwards compatible by testing the HDA bit, behaving like older VMs if unset.
It is desirable to maintain backwards compatibility if we can afford the engineering effort. But note that considerable engineering effort must be extended either way. For example, the Virtend application would require considerable work to migrate to HDA, being HDU currently.
This is a structural issue. Putting the effort into the HDA VM to provide backwards compatibility with HDU images supports all older HDU applications. Not putting the effort means that all older HDU applications must put effort in to move forwards. (This is because the VM is the platform supporting applications above it, just as the trunk image is a platform that supports applications built within it). Hence it’s my preference to put the effort into the VM (selfishly as I’m committed to keeping the Virtend application running).
What kinds of effort are we talking about? It’s not just having a flag to test and having the VM answer/respond appropriately for HDU if unset and for HDA if set. The flag needs to be tested very early in start up, potentially in the current sources before the image has been loaded. Eg on Linux one can supply a command line argument specifying the display subsystem to use which comes before the image name. So command line parsing must be able to make a pass through the arguments to find the image, locate the flags in the header, determine the value of the HDA flag, and then go back and actually process the command line arguments. Tedious work ;-)
Thanks,
Jaromir
--
Jaromír Matas
mail@jaromir.net
From: Eliot Miranda [mailto:eliot.miranda@gmail.com]
Sent: Tuesday, May 10, 2022 23:21
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> wrote:
Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the new primitive.
Thanks again,
Jaromir
--
Jaromír Matas
+420 777 492 777
mail@jaromir.net [mailto:mail@jaromir.net]
From: Eliot Miranda [mailto:eliot.miranda@gmail.com]
Sent: Tuesday, May 10, 2022 21:17
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Hi Jaromir,
On Tue, May 10, 2022 at 8:31 AM Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> wrote:
Hi Eliot, Nicolas,
(Eliot) Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the primitive most convenient.
Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive.
(Nicolas) Yes, I propose the selector
Process>>suspendAndUnblock <primitive: 78> snip...
Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent).
The question remaining is if we really need two new primitives 578 (V1) and 588 (V2).
It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
(Eliot) +1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Thank you, I wasn't sure how to proceed with this :) Do I understand correctly #suspend will have the new suspend semantics (either V1 or preferably V2) in new images (new release) and the previous semantics (via prim 88) will be available for backward compatibility (or special usage)?
Yes, exactly.
In that case the tests I submitted in KernelTests-jar.421 should work without changes...
Cool!
The next question is what to do with:
Process >> suspendPrimitivelyOrFail
"Test support. Execute primitive 88, or fail."
<primitive: 88>
^self primitiveFailed
Change it or remove it? Replace with Nicolas’s Process>>suspendAndUnblock ?
Well it appears to be used only in testAtomicSuspend. So I would change it to use the same primitive number as suspend, which I guess is 578, primitiveSuspendBackingUpV2, which has the more consistent semantics.
Thanks!
Best,
Jaromir
From: Eliot Miranda [mailto:eliot.miranda@gmail.com]
Sent: Monday, May 9, 2022 23:23
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On May 9, 2022, at 1:04 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com [mailto:nicolas.cellier.aka.nice@gmail.com]> wrote:
Hi Eliot,
Le lun. 9 mai 2022 à 20:36, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> a écrit :
Hi Jaromir, Hi Nicolas,
On Sun, May 8, 2022 at 8:17 AM Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> wrote:
Hi Nicolas,
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88).
Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88).
It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
Yes, thanks, well that was the original idea for Smalltalk processSuspensionUnblocks to answer about the behavior of the primitiveSuspend #88; which was before the primitives #568 and #578 were introduced. (And which was when I wrote the tests using this original logic). However that approach ran into problems with some existing implementations (e.g. Virtend) and as a result the two new primitives were introduced (#568 and #578) - and apparently the logic of what Smalltalk processSuspensionUnblocks answers about changed. Since then I haven't heard from Eliot about his plans how to proceed with the new suspend semantics or whether this is it :)
Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the primitive most convenient. Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive.
Yes, I propose the selector
Process>>suspendAndUnblock <primitive: 78> snip...
Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent).
The question remaining is if we really need two new primitives 578 (V1) and 588 (V2).
It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
+1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Sorry about not renaming processSuspensionUnblocks. This should be something like Smalltalk hasExtendedProcessSuspensionSemantics or some such.
It's not too late, there are not many usages, except inbox experimentations.
In case this is still WIP I'd quite like an approach similar to Smalltalk processPreemptionYields - you'd set Smalltalk processSuspensionUnblocks true or false depending on how you want the VM to behave and the VM would use the right semantics (unless this is too naive :) ).
Regarding the tests failing with the newest VM:
testAtomicSuspend - we can remove this updated version and leave the existing one for the moment
testRevisedSuspendExpectations - we can leave this one out too
testTerminateBlockedInNestedEnsure1 - I'll take a look at these two and try to adjust the logic
testTerminateBlockedInNestedEnsure2
All remaining test in KernelTests-jar.421 work as intended.
Thanks again,
Jaromir
--
Jaromír Matas
mail@jaromir.net [mailto:mail@jaromir.net]
From: Nicolas Cellier [mailto:nicolas.cellier.aka.nice@gmail.com]
Sent: Sunday, May 8, 2022 16:22
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Le dim. 8 mai 2022 à 15:59, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com [mailto:nicolas.cellier.aka.nice@gmail.com]> a écrit :
Le dim. 8 mai 2022 à 14:16, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com [mailto:nicolas.cellier.aka.nice@gmail.com]> a écrit :
Hi Jaromir
Le sam. 7 mai 2022 à 14:45, Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> a écrit :
Hi Marcel, Nicolas, Eliot,
(Nicolas) I have merged Kernel-jar.1446 which is trivial (re-signal an Exception).
Thanks!
(Marcel) Hmm... #testSimpleResignalVsOuter1 is still failing.
In my latest trunk image it passes ok (5/7/22)
(Marcel) I will also take a look at KernelTests-jar.421 to check whether any new semantics are okay.
(Nicolas) Yes, I did that and got two failing tests
testTerminateTerminatingProcess
testResumeTerminatingProcess
both failures look the same, in second self assert: terminator isSuspended.
Yes, they are supposed to fail at the moment so I suggest to make them expected failures/feature requests. However there's a stupid bug in both I’ve fixed now: they were indeed supposed to fail the following assertion:
``` self should: [terminatee terminate] raise: Error. ```
Apologies for the confusion - an updated changeset is enclosed.
However, there's another issue regarding the revised #suspend semantics Eliot has been working on. I've updated the process tests in KernelTests-jar.421 to test both the old and the new #suspend semantics. The two semantics should be distinguishable via Smalltalk processSuspensionUnblocks flag answering true in case of the old semantics and false in case of the revised one; my updated tests use this logic. However, unfortunately the latest VM answers "false" but uses the OLD suspend semantics in #suspend prim 88.
So I'm surprised you haven't observed more tests failing due to the new suspend semantics... What VM have you used – an older one? I'm on the latest trunk's VM 3183. And what is the answer of Smalltalk processSuspensionUnblocks? :) I'm utterly confused...
Hi Eliot - I may be confused here but if the current VM uses the old #suspend prim 88 semantics, shouldn't ```Smalltalk processSuspensionUnblocks``` answer true?
Indeed, I can confirm that the latest VM does answer false to (Smalltalk processSuspensionUnblocks).
Though, process suspension still unblocks as shown by the example at
https://github.com/pharo-project/pharo/issues/10669 [https://github.com/pharo-project/pharo/issues/10669]
A bit more simply, this answers false - it shouldn't:
s := Semaphore new.
p := [s wait] newProcess.
p resume.
100 milliSeconds wait.
p suspend; resume.
100 milliSeconds wait.
p isTerminated = Smalltalk processSuspensionUnblocks.
technically, this answers false - it shouldn't:
s := Semaphore new.
p := [s wait] newProcess.
p resume.
100 milliSeconds wait.
p suspend == s = Smalltalk processSuspensionUnblocks
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88).
Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88).
It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
The new Behavior is implemented by primitiveSuspendBackingUpV1 (#578) and primitiveSuspendBackingUpV2 (#588).
V1 always answers the old list (even if a Semaphore/Mutex), V2 answers nil in case of Semaphore/Mutex.
So, only ((Process>>#suspend) primitive) can give you a clue about the behavior, assuming that the primitive in question is implemented...
So it depends if we want to make it a Preference, or make the image compatible with some less capable VM...
we could implement
Process>>primitiveSuspendAndUnblock as <primitive: 88>
Process>>primitiveSuspend as <primitive: 588>
test the VM flag in suspend
Process>>suspend
^(VMHasPrimitiveSuspendBackingUp ifNil: [VMHasPrimitiveSuspendBackingUp := Smalltalk processSuspensionUnblocks not])
ifTrue: [self primitiveSuspend]
ifFalse: [self primitiveSuspendAndUnblock]
Then arrange to reset VMHasPrimitiveSuspendBackingUp to nil at snapshot...
I will tell next week about the status of the VM which I used for testing, it's possibly a few months old.
(Nicolas) I would also consider 1445 1421 and most importantly 1447.
Regarding 1421: alternatively, you might consider a newer version 1415 where the only difference is #return:from: passes a block instead of nil to #resume:through: to cause a fresh search for the first unwind context; otherwise they are equivalent.
OK, I will check again.
(Marcel) Maybe we can just merge
KernelTests-jar.393
KernelTests-jar.418
KernelTests-jar.421
And go from there? Or are there any objections?
KernelTests-jar.393 is just a special case of a more general KernelTests-jar.418 test…
In my image all tests are green provided Smalltalk processSuspensionUnblocks answers true and the two above tests Nicolas mentioned are marked expected failures :)
Thanks very much for your feedback,
Jaromir
--
Jaromír Matas
mail@jaromir.net [mailto:mail@jaromir.net]
--
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
On May 11, 2022, at 7:38 AM, Marcel Taeumel marcel.taeumel@hpi.de wrote:
Hi Eliot --
Can we afford to provide backwards compatibility with older “high dpi unaware” (HDU) images or not?
Similarly to 32-bit vs. 64-bit, it might as well the a responsibility of the underlying platform/OS to provide such compatibility.
Microsoft Windows shines here, still being able to run 32-bit and 64-bit applications. Still being able to run applications in "Low resolution" mode.
Apple macOS is a bummer here, not being able to run 32-bit applications. Not being able to even provide a "Low resolution" mode in macOS 12.
If we can avoid to tag images as "compatible" or "incompatible" to a certain VM regarding a specific feature, we are better off regarding maintenance cost.
In my opinion, it makes to sense *not* being able to open, e.g., Squeak 5.3 with the newest VM even if that image has no (good) support for high-resolution displays. There will always be measures to work around that. It's just inconvenient. Not broken.
It’s horribly broken for me because it means I have to fork the vm and maintain a backwards-compatible fork if I want to move bug fixes from the current 6.x vm to the 5.x compatible one. I’m overloaded as it is. This makes things worse. It is indeed broken. It’s only inconvenient from your perspective cuz you to y have to maintain the vm.
Best, Marcel
Am 11.05.2022 16:27:34 schrieb Eliot Miranda eliot.miranda@gmail.com:
Hi Jaromir,
On May 11, 2022, at 5:49 AM, Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the new primitive.
Thanks again; this brings me to another question: if you go simpler, i.e. with the ifFalse clause only, it would mean the method wouldn't work correctly with older VMs with the older prim 88 suspend semantics. I was under the impression to keep compatibility with older VMs but maybe it's a nonsensical assumption on my part (The reason I'm asking is the #terminate I wrote is compatible with both the new and old VMs - but could be simplified a bit if the old VMs were not supposed to be used with the new images).
That’s the way backwards compatibility works, with images and VMs in the same way as binaries and processors. A VM is expected to be backwards compatible with all older images of the same basic version (eg the Spur format, the V3 format). So all Spur 4.6, 5.x & current 6.x images should run on the current tip VM. But a new image may expect a feature added to a contemporary VM that is absent in previous versions of the VM.
This is analogous to, for example, binaries compiled against newer instructions added to the x86 such as mmx, sse 1, 2 & 3. We expect that the new processor can run older binaries; we do not expect new binaries to run on older processors unless we compile them specially.
Now, at a major release we may decide to remove VM features, but at that point we would change the VM’s image format number so that older images do not load. Since this effectively introduces a new VM variant it is not something we do lightly and do only when compelling, as the performance and functionality improvements obtained from the Spur architecture are over V3.
We’re currently thinking about this with high dpi support. Can we afford to provide backwards compatibility with older “high dpi unaware” (HDU) images or not? We can either freeze Spur VM support for older images at around March this year, and drop support for HDU images in the platform sources, or we can add a flag bit to the image flags in the latest VMs which if set identifies an image as “high dpi aware” (HDA), and set the bit, eg in a Monticello package load script. And then the VM can remain backwards compatible by testing the HDA bit, behaving like older VMs if unset.
It is desirable to maintain backwards compatibility if we can afford the engineering effort. But note that considerable engineering effort must be extended either way. For example, the Virtend application would require considerable work to migrate to HDA, being HDU currently.
This is a structural issue. Putting the effort into the HDA VM to provide backwards compatibility with HDU images supports all older HDU applications. Not putting the effort means that all older HDU applications must put effort in to move forwards. (This is because the VM is the platform supporting applications above it, just as the trunk image is a platform that supports applications built within it). Hence it’s my preference to put the effort into the VM (selfishly as I’m committed to keeping the Virtend application running).
What kinds of effort are we talking about? It’s not just having a flag to test and having the VM answer/respond appropriately for HDU if unset and for HDA if set. The flag needs to be tested very early in start up, potentially in the current sources before the image has been loaded. Eg on Linux one can supply a command line argument specifying the display subsystem to use which comes before the image name. So command line parsing must be able to make a pass through the arguments to find the image, locate the flags in the header, determine the value of the HDA flag, and then go back and actually process the command line arguments. Tedious work ;-)
Thanks,
Jaromir
--
Jaromír Matas
mail@jaromir.net
From: Eliot Miranda
Sent: Tuesday, May 10, 2022 23:21
To: The general-purpose Squeak developers list
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the new primitive.
Thanks again,
Jaromir
--
Jaromír Matas
+420 777 492 777
mail@jaromir.net
From: Eliot Miranda
Sent: Tuesday, May 10, 2022 21:17
To: The general-purpose Squeak developers list
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Hi Jaromir,
On Tue, May 10, 2022 at 8:31 AM Jaromir Matas mail@jaromir.net wrote:
Hi Eliot, Nicolas,
(Eliot) Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the primitive most convenient.
Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive.
(Nicolas) Yes, I propose the selector
Process>>suspendAndUnblock <primitive: 78> snip...
Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent).
The question remaining is if we really need two new primitives 578 (V1) and 588 (V2).
It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
(Eliot) +1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Thank you, I wasn't sure how to proceed with this :) Do I understand correctly #suspend will have the new suspend semantics (either V1 or preferably V2) in new images (new release) and the previous semantics (via prim 88) will be available for backward compatibility (or special usage)?
Yes, exactly.
In that case the tests I submitted in KernelTests-jar.421 should work without changes...
Cool!
The next question is what to do with:
Process >> suspendPrimitivelyOrFail
"Test support. Execute primitive 88, or fail." <primitive: 88> ^self primitiveFailedChange it or remove it? Replace with Nicolas’s Process>>suspendAndUnblock ?
Well it appears to be used only in testAtomicSuspend. So I would change it to use the same primitive number as suspend, which I guess is 578, primitiveSuspendBackingUpV2, which has the more consistent semantics.
Thanks!
Best,
Jaromir
From: Eliot Miranda
Sent: Monday, May 9, 2022 23:23
To: The general-purpose Squeak developers list
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On May 9, 2022, at 1:04 PM, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com wrote:
Hi Eliot,
Le lun. 9 mai 2022 à 20:36, Eliot Miranda eliot.miranda@gmail.com a écrit :
Hi Jaromir, Hi Nicolas,
On Sun, May 8, 2022 at 8:17 AM Jaromir Matas mail@jaromir.net wrote:
Hi Nicolas,
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88).
Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88).
It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
Yes, thanks, well that was the original idea for Smalltalk processSuspensionUnblocks to answer about the behavior of the primitiveSuspend #88; which was before the primitives #568 and #578 were introduced. (And which was when I wrote the tests using this original logic). However that approach ran into problems with some existing implementations (e.g. Virtend) and as a result the two new primitives were introduced (#568 and #578) - and apparently the logic of what Smalltalk processSuspensionUnblocks answers about changed. Since then I haven't heard from Eliot about his plans how to proceed with the new suspend semantics or whether this is it :)
Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the primitive most convenient. Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive.
Yes, I propose the selector
Process>>suspendAndUnblock <primitive: 78> snip...
Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent).
The question remaining is if we really need two new primitives 578 (V1) and 588 (V2).
It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
+1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Sorry about not renaming processSuspensionUnblocks. This should be something like Smalltalk hasExtendedProcessSuspensionSemantics or some such.
It's not too late, there are not many usages, except inbox experimentations.
In case this is still WIP I'd quite like an approach similar to Smalltalk processPreemptionYields - you'd set Smalltalk processSuspensionUnblocks true or false depending on how you want the VM to behave and the VM would use the right semantics (unless this is too naive :) ).
Regarding the tests failing with the newest VM:
testAtomicSuspend - we can remove this updated version and leave the existing one for the moment
testRevisedSuspendExpectations - we can leave this one out too
testTerminateBlockedInNestedEnsure1 - I'll take a look at these two and try to adjust the logic
testTerminateBlockedInNestedEnsure2
All remaining test in KernelTests-jar.421 work as intended.
Thanks again,
Jaromir
--
Jaromír Matas
mail@jaromir.net
From: Nicolas Cellier
Sent: Sunday, May 8, 2022 16:22
To: The general-purpose Squeak developers list
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Le dim. 8 mai 2022 à 15:59, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com a écrit :
Le dim. 8 mai 2022 à 14:16, Nicolas Cellier nicolas.cellier.aka.nice@gmail.com a écrit :
Hi Jaromir
Le sam. 7 mai 2022 à 14:45, Jaromir Matas mail@jaromir.net a écrit :
Hi Marcel, Nicolas, Eliot,
(Nicolas) I have merged Kernel-jar.1446 which is trivial (re-signal an Exception).
Thanks!
(Marcel) Hmm... #testSimpleResignalVsOuter1 is still failing.
In my latest trunk image it passes ok (5/7/22)
(Marcel) I will also take a look at KernelTests-jar.421 to check whether any new semantics are okay.
(Nicolas) Yes, I did that and got two failing tests
testTerminateTerminatingProcess
testResumeTerminatingProcess
both failures look the same, in second self assert: terminator isSuspended.
Yes, they are supposed to fail at the moment so I suggest to make them expected failures/feature requests. However there's a stupid bug in both I’ve fixed now: they were indeed supposed to fail the following assertion:
``` self should: [terminatee terminate] raise: Error. ```
Apologies for the confusion - an updated changeset is enclosed.
However, there's another issue regarding the revised #suspend semantics Eliot has been working on. I've updated the process tests in KernelTests-jar.421 to test both the old and the new #suspend semantics. The two semantics should be distinguishable via Smalltalk processSuspensionUnblocks flag answering true in case of the old semantics and false in case of the revised one; my updated tests use this logic. However, unfortunately the latest VM answers "false" but uses the OLD suspend semantics in #suspend prim 88.
So I'm surprised you haven't observed more tests failing due to the new suspend semantics... What VM have you used – an older one? I'm on the latest trunk's VM 3183. And what is the answer of Smalltalk processSuspensionUnblocks? :) I'm utterly confused...
Hi Eliot - I may be confused here but if the current VM uses the old #suspend prim 88 semantics, shouldn't ```Smalltalk processSuspensionUnblocks``` answer true?
Indeed, I can confirm that the latest VM does answer false to (Smalltalk processSuspensionUnblocks).
Though, process suspension still unblocks as shown by the example at
https://github.com/pharo-project/pharo/issues/10669
A bit more simply, this answers false - it shouldn't:
s := Semaphore new.
p := [s wait] newProcess.
p resume.
100 milliSeconds wait.
p suspend; resume.
100 milliSeconds wait.
p isTerminated = Smalltalk processSuspensionUnblocks.
technically, this answers false - it shouldn't:
s := Semaphore new.
p := [s wait] newProcess.
p resume.
100 milliSeconds wait.
p suspend == s = Smalltalk processSuspensionUnblocks
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88).
Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88).
It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
The new Behavior is implemented by primitiveSuspendBackingUpV1 (#578) and primitiveSuspendBackingUpV2 (#588).
V1 always answers the old list (even if a Semaphore/Mutex), V2 answers nil in case of Semaphore/Mutex.
So, only ((Process>>#suspend) primitive) can give you a clue about the behavior, assuming that the primitive in question is implemented...
So it depends if we want to make it a Preference, or make the image compatible with some less capable VM...
we could implement
Process>>primitiveSuspendAndUnblock as <primitive: 88> Process>>primitiveSuspend as <primitive: 588>test the VM flag in suspend
Process>>suspend ^(VMHasPrimitiveSuspendBackingUp ifNil: [VMHasPrimitiveSuspendBackingUp := Smalltalk processSuspensionUnblocks not]) ifTrue: [self primitiveSuspend] ifFalse: [self primitiveSuspendAndUnblock]Then arrange to reset VMHasPrimitiveSuspendBackingUp to nil at snapshot...
I will tell next week about the status of the VM which I used for testing, it's possibly a few months old.
(Nicolas) I would also consider 1445 1421 and most importantly 1447.
Regarding 1421: alternatively, you might consider a newer version 1415 where the only difference is #return:from: passes a block instead of nil to #resume:through: to cause a fresh search for the first unwind context; otherwise they are equivalent.
OK, I will check again.
(Marcel) Maybe we can just merge
KernelTests-jar.393
KernelTests-jar.418
KernelTests-jar.421
And go from there? Or are there any objections?
KernelTests-jar.393 is just a special case of a more general KernelTests-jar.418 test…
In my image all tests are green provided Smalltalk processSuspensionUnblocks answers true and the two above tests Nicolas mentioned are marked expected failures :)
Thanks very much for your feedback,
Jaromir
--
Jaromír Matas
mail@jaromir.net
--
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
Hi Eliot --
I assumed that the matters changed since May 11. :-) macOS 12 puts unnecessary pressure on all of us. Thank you, Apple.
We agreed on implementing a vm-based (up-)scaling to make older images compatible with newer platforms where platform-based (up-)scaling is not supported such as on macOS 12.
It is still a lot of work, especially if we also want to support per-monitor scale changes in any yet-to-be-written platform-specific scaling code.
At the moment, there is that image-header flag in place where we can denote whether an image has support for high resolutions or not. Next step is to use this information and implement the necessary support per platform. X11, Windows (DirectDraw, OpenGL, ...), macOS (core-graphics, metal, opengl, ...)
I am in favor of maximizing backwards compatibility. I hear you.
Best, Marcel Am 02.07.2022 19:57:17 schrieb Eliot Miranda eliot.miranda@gmail.com:
On May 11, 2022, at 7:38 AM, Marcel Taeumel marcel.taeumel@hpi.de wrote:
Hi Eliot --
Can we afford to provide backwards compatibility with older “high dpi unaware” (HDU) images or not?
Similarly to 32-bit vs. 64-bit, it might as well the a responsibility of the underlying platform/OS to provide such compatibility.
Microsoft Windows shines here, still being able to run 32-bit and 64-bit applications. Still being able to run applications in "Low resolution" mode.
Apple macOS is a bummer here, not being able to run 32-bit applications. Not being able to even provide a "Low resolution" mode in macOS 12.
If we can avoid to tag images as "compatible" or "incompatible" to a certain VM regarding a specific feature, we are better off regarding maintenance cost.
In my opinion, it makes to sense *not* being able to open, e.g., Squeak 5.3 with the newest VM even if that image has no (good) support for high-resolution displays. There will always be measures to work around that. It's just inconvenient. Not broken.
It’s horribly broken for me because it means I have to fork the vm and maintain a backwards-compatible fork if I want to move bug fixes from the current 6.x vm to the 5.x compatible one. I’m overloaded as it is. This makes things worse. It is indeed broken. It’s only inconvenient from your perspective cuz you to y have to maintain the vm.
Best, Marcel
Am 11.05.2022 16:27:34 schrieb Eliot Miranda eliot.miranda@gmail.com: Hi Jaromir,
On May 11, 2022, at 5:49 AM, Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas mail@jaromir.net wrote:
Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the new primitive.
Thanks again; this brings me to another question: if you go simpler, i.e. with the ifFalse clause only, it would mean the method wouldn't work correctly with older VMs with the older prim 88 suspend semantics. I was under the impression
to keep compatibility with older VMs but maybe it's a nonsensical assumption on my part (The reason I'm asking is the #terminate I wrote is compatible with both the new and old VMs - but could be simplified a bit if the old VMs were not supposed to be used
with the new images).
That’s the way backwards compatibility works, with images and VMs in the same way as binaries and processors. A VM is expected to be backwards compatible with all older images of the same basic version (eg the Spur format, the V3 format). So all Spur 4.6, 5.x & current 6.x images should run on the current tip VM. But a new image may expect a feature added to a contemporary VM that is absent in previous versions of the VM.
This is analogous to, for example, binaries compiled against newer instructions added to the x86 such as mmx, sse 1, 2 & 3. We expect that the new processor can run older binaries; we do not expect new binaries to run on older processors unless we compile them specially.
Now, at a major release we may decide to remove VM features, but at that point we would change the VM’s image format number so that older images do not load. Since this effectively introduces a new VM variant it is not something we do lightly and do only when compelling, as the performance and functionality improvements obtained from the Spur architecture are over V3.
We’re currently thinking about this with high dpi support. Can we afford to provide backwards compatibility with older “high dpi unaware” (HDU) images or not? We can either freeze Spur VM support for older images at around March this year, and drop support for HDU images in the platform sources, or we can add a flag bit to the image flags in the latest VMs which if set identifies an image as “high dpi aware” (HDA), and set the bit, eg in a Monticello package load script. And then the VM can remain backwards compatible by testing the HDA bit, behaving like older VMs if unset.
It is desirable to maintain backwards compatibility if we can afford the engineering effort. But note that considerable engineering effort must be extended either way. For example, the Virtend application would require considerable work to migrate to HDA, being HDU currently.
This is a structural issue. Putting the effort into the HDA VM to provide backwards compatibility with HDU images supports all older HDU applications. Not putting the effort means that all older HDU applications must put effort in to move forwards. (This is because the VM is the platform supporting applications above it, just as the trunk image is a platform that supports applications built within it). Hence it’s my preference to put the effort into the VM (selfishly as I’m committed to keeping the Virtend application running).
What kinds of effort are we talking about? It’s not just having a flag to test and having the VM answer/respond appropriately for HDU if unset and for HDA if set. The flag needs to be tested very early in start up, potentially in the current sources before the image has been loaded. Eg on Linux one can supply a command line argument specifying the display subsystem to use which comes before the image name. So command line parsing must be able to make a pass through the arguments to find the image, locate the flags in the header, determine the value of the HDA flag, and then go back and actually process the command line arguments. Tedious work ;-)
Thanks,
Jaromir
--
Jaromír Matas
mail@jaromir.net
From: Eliot Miranda [mailto:eliot.miranda@gmail.com]
Sent: Tuesday, May 10, 2022 23:21
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On Tue, May 10, 2022 at 1:36 PM Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> wrote:
Hi Eliot,
Thanks a lot for your answers.
One last question: what about Process >> #signalException:? You wrote a fix (enclosed) – is it ok to merge it when the #suspend method is updated (Nicolas’s way)?
Yes, I think so. We have to test it but it seems right. And we can go even simpler, just have the first clause as guarded by processSuspensionUnblocks ifFalse:, once we're ready to release on VMs with the
new primitive.
Thanks again,
Jaromir
--
Jaromír Matas
+420 777 492 777
mail@jaromir.net [mailto:mail@jaromir.net]
From:
Eliot Miranda [mailto:eliot.miranda@gmail.com]
Sent: Tuesday, May 10, 2022 21:17
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Hi Jaromir,
On Tue, May 10, 2022 at 8:31 AM Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> wrote:
Hi Eliot, Nicolas,
(Eliot) Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the primitive most convenient.
Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive.
(Nicolas) Yes, I propose the selector
Process>>suspendAndUnblock <primitive: 78> snip...
Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent).
The question remaining is if we really need two new primitives 578 (V1) and 588 (V2).
It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
(Eliot) +1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Thank you, I wasn't sure how to proceed with this :) Do I understand correctly #suspend will have the new suspend semantics (either V1 or preferably V2) in new images (new release)
and the previous semantics (via prim 88) will be available for backward compatibility (or special usage)?
Yes, exactly.
In that case the tests I submitted in KernelTests-jar.421 should work without changes...
Cool!
The next question is what to do with:
Process >> suspendPrimitivelyOrFail
"Test support. Execute primitive 88, or fail."
<primitive: 88>
^self primitiveFailed
Change it or remove it? Replace with Nicolas’s Process>>suspendAndUnblock ?
Well it appears to be used only in testAtomicSuspend. So I would change it to use the same primitive number as suspend, which I guess is 578, primitiveSuspendBackingUpV2,
which has the more consistent semantics.
Thanks!
Best,
Jaromir
From:
Eliot Miranda [mailto:eliot.miranda@gmail.com]
Sent: Monday, May 9, 2022 23:23
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
On May 9, 2022, at 1:04 PM, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com [mailto:nicolas.cellier.aka.nice@gmail.com]> wrote:
Hi Eliot,
Le lun. 9 mai 2022 à 20:36, Eliot Miranda <eliot.miranda@gmail.com [mailto:eliot.miranda@gmail.com]> a écrit :
Hi Jaromir, Hi Nicolas,
On Sun, May 8, 2022 at 8:17 AM Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> wrote:
Hi Nicolas,
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88).
Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88).
It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
Yes, thanks, well that was the original idea for Smalltalk processSuspensionUnblocks to answer about the behavior of the primitiveSuspend #88; which was before the primitives #568
and #578 were introduced. (And which was when I wrote the tests using this original logic). However that approach ran into problems with some existing implementations (e.g. Virtend) and as a result the two new primitives were introduced (#568 and #578) - and
apparently the logic of what Smalltalk processSuspensionUnblocks answers about changed. Since then I haven't heard from Eliot about his plans how to proceed with the new suspend semantics or whether this is it :)
Apologies for not responding sooner. It seems to me the semantics must remain as they are, and it be left up to the image to bind suspend to the
primitive most convenient. Then backwards compatibility can be provided for cases such as DelayWaitTimeout by providing a renamed suspend which accesses the old primitive.
Yes, I propose the selector
Process>>suspendAndUnblock <primitive: 78> snip...
Process>>suspend would call the new primitive, and fallback to suspendAndUnblock in case of failure (for example if new primitive is absent).
The question remaining is if we really need two new primitives 578 (V1) and 588 (V2).
It seems like signalException: would require V1, while Jaromir terminate logic would require V2...
+1. Works for me. Having both 578 & 588 available in the VM is fine. I agree that surfacing only one of them is best.
Sorry about not renaming processSuspensionUnblocks. This should be something like Smalltalk hasExtendedProcessSuspensionSemantics or some such.
It's not too late, there are not many usages, except inbox experimentations.
In case this is still WIP I'd quite like an approach similar to Smalltalk processPreemptionYields - you'd set Smalltalk processSuspensionUnblocks true or false depending on how
you want the VM to behave and the VM would use the right semantics (unless this is too naive :) ).
Regarding the tests failing with the newest VM:
testAtomicSuspend - we can remove this updated version and leave the existing one for the moment
testRevisedSuspendExpectations - we can leave this one out too
testTerminateBlockedInNestedEnsure1 - I'll take a look at these two and try to adjust the logic
testTerminateBlockedInNestedEnsure2
All remaining test in KernelTests-jar.421 work as intended.
Thanks again,
Jaromir
--
Jaromír Matas
mail@jaromir.net [mailto:mail@jaromir.net]
From:
Nicolas Cellier [mailto:nicolas.cellier.aka.nice@gmail.com]
Sent: Sunday, May 8, 2022 16:22
To: The general-purpose Squeak developers list [mailto:squeak-dev@lists.squeakfoundation.org]
Subject: Re: [squeak-dev] SIGTRAP when Proceeding from BlockCannotReturn
Le dim. 8 mai 2022 à 15:59, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com [mailto:nicolas.cellier.aka.nice@gmail.com]> a écrit :
Le dim. 8 mai 2022 à 14:16, Nicolas Cellier <nicolas.cellier.aka.nice@gmail.com [mailto:nicolas.cellier.aka.nice@gmail.com]> a écrit :
Hi Jaromir
Le sam. 7 mai 2022 à 14:45, Jaromir Matas <mail@jaromir.net [mailto:mail@jaromir.net]> a écrit :
Hi Marcel, Nicolas, Eliot,
(Nicolas) I have merged Kernel-jar.1446 which is trivial (re-signal an Exception).
Thanks!
(Marcel) Hmm... #testSimpleResignalVsOuter1 is still failing.
In my latest trunk image it passes ok (5/7/22)
(Marcel) I will also take a look at KernelTests-jar.421 to check whether any new semantics are okay.
(Nicolas) Yes, I did that and got two failing tests
testTerminateTerminatingProcess
testResumeTerminatingProcess
both failures look the same, in second self assert: terminator isSuspended.
Yes, they are supposed to fail at the moment so I suggest to make them expected failures/feature requests. However there's a stupid bug in both I’ve fixed now: they were indeed
supposed to fail the following assertion:
``` self should: [terminatee terminate] raise: Error. ```
Apologies for the confusion - an updated changeset is enclosed.
However, there's another issue regarding the revised #suspend semantics Eliot has been working on. I've updated the process tests in KernelTests-jar.421 to test both the old and
the new #suspend semantics. The two semantics should be distinguishable via Smalltalk processSuspensionUnblocks flag answering true in case of the old semantics and false in case of the revised one; my updated tests use this logic. However, unfortunately the
latest VM answers "false" but uses the OLD suspend semantics in #suspend prim 88.
So I'm surprised you haven't observed more tests failing due to the new suspend semantics... What VM have you used – an older one? I'm on the latest trunk's VM 3183. And what is
the answer of Smalltalk processSuspensionUnblocks? :) I'm utterly confused...
Hi Eliot - I may be confused here but if the current VM uses the old #suspend prim 88 semantics, shouldn't ```Smalltalk processSuspensionUnblocks``` answer true?
Indeed, I can confirm that the latest VM does answer false to (Smalltalk processSuspensionUnblocks).
Though, process suspension still unblocks as shown by the example at
https://github.com/pharo-project/pharo/issues/10669 [https://github.com/pharo-project/pharo/issues/10669]
A bit more simply, this answers false - it shouldn't:
s := Semaphore new.
p := [s wait] newProcess.
p resume.
100 milliSeconds wait.
p suspend; resume.
100 milliSeconds wait.
p isTerminated = Smalltalk processSuspensionUnblocks.
technically, this answers false - it shouldn't:
s := Semaphore new.
p := [s wait] newProcess.
p resume.
100 milliSeconds wait.
p suspend == s = Smalltalk processSuspensionUnblocks
About the latest VM, (Smalltalk processSuspensionUnblocks) answers about the behavior of primitiveSuspend (#88).
Err, wrong wording (Smalltalk processSuspensionUnblocks) does not describe the behavior of primitiveSuspend #(88).
It answers whether the VM has primitives #578 and #588 (see #getCogVMFeatureFlags in VMMaker)...
The new Behavior is implemented by primitiveSuspendBackingUpV1 (#578) and primitiveSuspendBackingUpV2 (#588).
V1 always answers the old list (even if a Semaphore/Mutex), V2 answers nil in case of Semaphore/Mutex.
So, only ((Process>>#suspend) primitive) can give you a clue about the behavior, assuming that the primitive in question is implemented...
So it depends if we want to make it a Preference, or make the image compatible with some less capable VM...
we could implement
Process>>primitiveSuspendAndUnblock as <primitive: 88>
Process>>primitiveSuspend as <primitive: 588>
test the VM flag in suspend
Process>>suspend
^(VMHasPrimitiveSuspendBackingUp ifNil: [VMHasPrimitiveSuspendBackingUp := Smalltalk processSuspensionUnblocks not])
ifTrue: [self primitiveSuspend]
ifFalse: [self primitiveSuspendAndUnblock]
Then arrange to reset VMHasPrimitiveSuspendBackingUp to nil at snapshot...
I will tell next week about the status of the VM which I used for testing, it's possibly a few months old.
(Nicolas) I would also consider 1445 1421 and most importantly 1447.
Regarding 1421: alternatively, you might consider a newer version 1415 where the only difference is #return:from: passes a block instead of nil to #resume:through: to cause a fresh
search for the first unwind context; otherwise they are equivalent.
OK, I will check again.
(Marcel) Maybe we can just merge
KernelTests-jar.393
KernelTests-jar.418
KernelTests-jar.421
And go from there? Or are there any objections?
KernelTests-jar.393 is just a special case of a more general KernelTests-jar.418 test…
In my image all tests are green provided Smalltalk processSuspensionUnblocks answers true and the two above tests Nicolas mentioned are marked expected failures :)
Thanks very much for your feedback,
Jaromir
--
Jaromír Matas
mail@jaromir.net [mailto:mail@jaromir.net]
--
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
--
_,,,^..^,,,_
best, Eliot
vm-dev@lists.squeakfoundation.org
-
Eliot Miranda -
Marcel Taeumel