For a few builds, appveyor is failing for squeak.cog.v3 at final link stage:
i686-w64-mingw32-gcc -mwindows -m32 -mthreads
-Wl,--large-address-aware,--export-all-symbols -o
build/vm/SqueakUnstripped.exe \
...snip...
collect2: fatal error: ld terminated with signal 11 [Segmentation fault],
core dumped
https://ci.appveyor.com/project/OpenSmalltalk/vm/history shows it happened
since
https://ci.appveyor.com/project/OpenSmalltalk/vm/build/1.0.710
The same build pass on linux and osx.
The tools shall never crash, bu worse problem I have is that I can't
reproduce the segfault in my windows VM, even if updating properly cygwin...
Any idea of what's going on?
Branch: refs/heads/Cog
Home: https://github.com/OpenSmalltalk/opensmalltalk-vm
Commit: 3010e4465405f6ec7a289fc3a3d21eb324816a8f
https://github.com/OpenSmalltalk/opensmalltalk-vm/commit/3010e4465405f6ec7a…
Author: Eliot Miranda <eliot.miranda(a)gmail.com>
Date: 2017-06-15 (Thu, 15 Jun 2017)
Changed paths:
M build.macos64x64/squeak.sista.spur/Makefile
M nsspur64src/vm/cogit.h
M nsspur64src/vm/cogitX64SysV.c
M nsspur64src/vm/cogitX64WIN64.c
M nsspur64src/vm/cointerp.c
M nsspur64src/vm/cointerp.h
M nsspur64src/vm/gcc3x-cointerp.c
M nsspursrc/vm/cogit.h
M nsspursrc/vm/cogitARMv5.c
M nsspursrc/vm/cogitIA32.c
M nsspursrc/vm/cogitMIPSEL.c
M nsspursrc/vm/cointerp.c
M nsspursrc/vm/cointerp.h
M nsspursrc/vm/gcc3x-cointerp.c
M nsspurstack64src/vm/gcc3x-interp.c
M nsspurstack64src/vm/interp.c
M nsspurstacksrc/vm/gcc3x-interp.c
M nsspurstacksrc/vm/interp.c
M platforms/Cross/vm/sq.h
M spur64src/vm/cogit.h
M spur64src/vm/cogitX64SysV.c
M spur64src/vm/cogitX64WIN64.c
M spur64src/vm/cointerp.c
M spur64src/vm/cointerp.h
M spur64src/vm/gcc3x-cointerp.c
M spurlowcode64src/vm/cogit.h
M spurlowcode64src/vm/cogitX64SysV.c
M spurlowcode64src/vm/cogitX64WIN64.c
M spurlowcode64src/vm/cointerp.c
M spurlowcode64src/vm/cointerp.h
M spurlowcode64src/vm/gcc3x-cointerp.c
M spurlowcodesrc/vm/cogit.h
M spurlowcodesrc/vm/cogitARMv5.c
M spurlowcodesrc/vm/cogitIA32.c
M spurlowcodesrc/vm/cogitMIPSEL.c
M spurlowcodesrc/vm/cointerp.c
M spurlowcodesrc/vm/cointerp.h
M spurlowcodesrc/vm/gcc3x-cointerp.c
M spurlowcodestack64src/vm/gcc3x-interp.c
M spurlowcodestack64src/vm/interp.c
M spurlowcodestacksrc/vm/gcc3x-interp.c
M spurlowcodestacksrc/vm/interp.c
M spursista64src/vm/cogit.h
M spursista64src/vm/cogitX64SysV.c
M spursista64src/vm/cogitX64WIN64.c
M spursista64src/vm/cointerp.c
M spursista64src/vm/cointerp.h
M spursista64src/vm/gcc3x-cointerp.c
M spursistasrc/vm/cogit.h
M spursistasrc/vm/cogitARMv5.c
M spursistasrc/vm/cogitIA32.c
M spursistasrc/vm/cogitMIPSEL.c
M spursistasrc/vm/cointerp.c
M spursistasrc/vm/cointerp.h
M spursistasrc/vm/gcc3x-cointerp.c
M spursrc/vm/cogit.h
M spursrc/vm/cogitARMv5.c
M spursrc/vm/cogitIA32.c
M spursrc/vm/cogitMIPSEL.c
M spursrc/vm/cointerp.c
M spursrc/vm/cointerp.h
M spursrc/vm/gcc3x-cointerp.c
M spurstack64src/vm/gcc3x-interp.c
M spurstack64src/vm/interp.c
M spurstacksrc/vm/gcc3x-interp.c
M spurstacksrc/vm/interp.c
M src/plugins/B2DPlugin/B2DPlugin.c
M src/plugins/BitBltPlugin/BitBltPlugin.c
M src/plugins/CroquetPlugin/CroquetPlugin.c
M src/plugins/DSAPrims/DSAPrims.c
M src/plugins/FFTPlugin/FFTPlugin.c
M src/plugins/GeniePlugin/GeniePlugin.c
M src/plugins/JPEGReadWriter2Plugin/JPEGReadWriter2Plugin.c
M src/plugins/JPEGReaderPlugin/JPEGReaderPlugin.c
M src/plugins/MiscPrimitivePlugin/MiscPrimitivePlugin.c
M src/plugins/ScratchPlugin/ScratchPlugin.c
M src/plugins/SqueakFFIPrims/X64SysVFFIPlugin.c
M src/plugins/ZipPlugin/ZipPlugin.c
M src/vm/cogit.h
M src/vm/cogitARMv5.c
M src/vm/cogitIA32.c
M src/vm/cogitMIPSEL.c
M src/vm/cointerp.c
M src/vm/cointerp.h
M src/vm/cointerpmt.c
M src/vm/cointerpmt.h
M src/vm/gcc3x-cointerp.c
M src/vm/gcc3x-cointerpmt.c
M stacksrc/vm/gcc3x-interp.c
M stacksrc/vm/interp.c
Log Message:
-----------
CogVM source as per VMMaker.oscog-eem.2243
StackInterpreter:
Make statTenures, statAverageLivePagesWhenMapping & statMaxPageCountWhenMapping
(parameters 11, 68 & 69) writable to allow easier profiling. Allow parameters
expecting a float (statTenures, Sista CogCodeThreshold &
statAverageLivePagesWhenMapping: 17, 55, 68) to take an int.
Slang:
Fix a bug in inferTypesForImplicitlyTypedVariablesIn:. We cannot derive types
from variables assigned to until all assignments are typed. So exclude
variables assigned from as-yet-untyped methods. The old code would simply
ignore the types from as-yet-untyped methods, hence leaving the variable
with a chosen-at-random, unmerged type.
This fixes a number of cases. But there's still the weirdness that the return
type of mapEndFor: in cogitX64SysV.c is correct (usqInt) but incorrect (sqInt)
in cogitX64WIN64.c. Luckily this is benign, but still should be fixed asap.
Eliot Miranda uploaded a new version of VMMaker to project VM Maker:
http://source.squeak.org/VMMaker/VMMaker.oscog-eem.2243.mcz
==================== Summary ====================
Name: VMMaker.oscog-eem.2243
Author: eem
Time: 15 June 2017, 4:51:21.305055 pm
UUID: 2f909507-a192-4a51-8bff-786bc61737a5
Ancestors: VMMaker.oscog-eem.2242
StackInterpreter:
Make statTenures, statAverageLivePagesWhenMapping & statMaxPageCountWhenMapping (parameters 11, 68 & 69) writable to allow easier profiling. Allow parameters expecting a float (statTenures, Sista CogCodeThreshold & statAverageLivePagesWhenMapping: 17, 55, 68) to take an int.
Slang:
Fix a bug in inferTypesForImplicitlyTypedVariablesIn:. We cannot derive types from variables assigned to until all assignments are typed. So exclude variables assigned from as-yet-untyped methods. The old code would simply ignore the types from as-yet-untyped methods, hence leaving the variable with a chosen-at-random, unmerged type.
This fixes a number of cases. But there's still the weirdness that the return type of mapEndFor: in cogitX64SysV.c is correct (usqInt) but incorrect (sqInt) in cogitX64WIN64.c. Luckily this is benign, but still should be fixed asap.
=============== Diff against VMMaker.oscog-eem.2242 ===============
Item was changed:
----- Method: CogStackPages>>statAverageLivePagesWhenMapping (in category 'statistics') -----
statAverageLivePagesWhenMapping
<returnTypeC: #double>
+ ^statNumMaps = 0
+ ifTrue: [0.0]
+ ifFalse: [statPageCountWhenMappingSum asFloat / statNumMaps]!
- ^statPageCountWhenMappingSum asFloat / statNumMaps!
Item was added:
+ ----- Method: CogStackPages>>statAverageLivePagesWhenMapping: (in category 'statistics') -----
+ statAverageLivePagesWhenMapping: aFloat
+ <var: #aFloat type: #double>
+ aFloat == 0.0
+ ifTrue: [statPageCountWhenMappingSum := statNumMaps := 0]
+ ifFalse: [coInterpreter primitiveFailFor: PrimErrBadArgument]!
Item was added:
+ ----- Method: CogStackPages>>statMaxPageCountWhenMapping: (in category 'statistics') -----
+ statMaxPageCountWhenMapping: num
+ statMaxPageCountWhenMapping := num!
Item was added:
+ ----- Method: ObjectMemory>>statTenures: (in category 'accessing') -----
+ statTenures: aValue
+ statTenures := aValue!
Item was added:
+ ----- Method: SpurGenerationScavenger>>statTenures: (in category 'accessing') -----
+ statTenures: aValue
+ statTenures := aValue!
Item was added:
+ ----- Method: SpurMemoryManager>>statTenures: (in category 'accessing') -----
+ statTenures: aValue
+ <doNotGenerate>
+ scavenger statTenures: aValue!
Item was added:
+ ----- Method: StackInterpreterPrimitives>>noInlineLoadFloatOrIntFrom: (in category 'primitive support') -----
+ noInlineLoadFloatOrIntFrom: floatOrInt
+ <inline: #never>
+ ^objectMemory loadFloatOrIntFrom: floatOrInt!
Item was changed:
----- Method: StackInterpreterPrimitives>>primitiveVMParameter (in category 'system control primitives') -----
(excessive size, no diff calculated)
Item was changed:
----- Method: TMethod>>inferTypesForImplicitlyTypedVariablesIn: (in category 'type inference') -----
inferTypesForImplicitlyTypedVariablesIn: aCodeGen
"infer types for untyped variables from assignments and arithmetic uses.
For debugging answer a Dictionary from var to the nodes that determined types
This for debugging:
(self copy inferTypesForImplicitlyTypedVariablesIn: aCodeGen)"
+ | alreadyExplicitlyTypedOrNotToBeTyped asYetUntyped mustBeSigned newDeclarations effectiveNodes |
- | alreadyExplicitlyTypedOrNotToBeTyped asYetUntyped effectiveNodes |
aCodeGen maybeBreakForTestToInline: selector in: self.
alreadyExplicitlyTypedOrNotToBeTyped := declarations keys asSet.
asYetUntyped := locals copyWithoutAll: alreadyExplicitlyTypedOrNotToBeTyped.
+ mustBeSigned := Set new.
+ newDeclarations := Dictionary new.
effectiveNodes := Dictionary new. "this for debugging"
parseTree nodesDo:
[:node| | type var |
"If there is something of the form i >= 0, then i should be signed, not unsigned."
(node isSend
and: [(locals includes: (var := node receiver variableNameOrNil))
- and: [(alreadyExplicitlyTypedOrNotToBeTyped includes: var) not "don't be fooled by inferred unsigned types"
and: [(#(<= < >= >) includes: node selector)
and: [node args first isConstant
+ and: [node args first value = 0]]]]) ifTrue:
+ [mustBeSigned add: var.
+ effectiveNodes at: var put: { #signed. node }, (effectiveNodes at: var ifAbsent: [#()])].
- and: [node args first value = 0
- and: [(type := self typeFor: var in: aCodeGen) notNil
- and: [type first == $u]]]]]]]) ifTrue:
- [self declarationAt: var put: (aCodeGen signedTypeForIntegralType: type), ' ', var.
- effectiveNodes at: var put: { declarations at: var. node }].
"if an assignment to an untyped local of a known type, set the local's type to that type.
Only observe known sends (methods in the current set) and typed local variables."
(node isAssignment
and: [(locals includes: (var := node variable name))
and: [(alreadyExplicitlyTypedOrNotToBeTyped includes: var) not]]) ifTrue: "don't be fooled by previously inferred types"
[type := node expression isSend
ifTrue: [aCodeGen returnTypeForSend: node expression in: self ifNil: nil]
ifFalse: [self typeFor: node expression in: aCodeGen].
type "If untyped, then cannot type the variable yet. A subsequent assignment may assign a subtype of what this type ends up being"
+ ifNil: "Further, if the type derives from an as-yet-untyped method, we must defer."
+ [alreadyExplicitlyTypedOrNotToBeTyped add: var.
+ (node expression isSend
+ and: [(aCodeGen methodNamed: node expression selector) notNil]) ifTrue:
+ [newDeclarations removeKey: var ifAbsent: nil]]
- ifNil: [alreadyExplicitlyTypedOrNotToBeTyped add: var]
ifNotNil: "Merge simple types (but *don't* merge untyped vars); complex types must be defined by the programmer."
[(aCodeGen isSimpleType: type) ifTrue:
[(asYetUntyped includes: var)
+ ifTrue: [newDeclarations at: var put: type, ' ', var. asYetUntyped remove: var]
- ifTrue: [declarations at: var put: type, ' ', var. asYetUntyped remove: var]
ifFalse:
+ [aCodeGen mergeTypeOf: var in: newDeclarations with: type method: self].
+ effectiveNodes at: var put: { newDeclarations at: var. node }, (effectiveNodes at: var ifAbsent: [#()])]]]].
+ mustBeSigned do:
+ [:var|
+ (newDeclarations at: var ifAbsent: nil) ifNotNil:
+ [:decl| | type |
+ type := aCodeGen extractTypeFor: var fromDeclaration: decl.
+ type first == $u ifTrue:
+ [newDeclarations at: var put: (aCodeGen signedTypeForIntegralType: type), ' ', var]]].
+ newDeclarations keysAndValuesDo:
+ [:var :decl| declarations at: var put: decl].
- [aCodeGen mergeTypeOf: var in: declarations with: type method: self].
- effectiveNodes at: var put: { declarations at: var. node }, (effectiveNodes at: var ifAbsent: [#()])]]]].
^effectiveNodes!
In trying to troubleshoot an issue, I needed to bump up the stackpages
parameter. On 64-bit Linux, a value of 600 worked but 1000 segfaulted so I
was just wondering what the limit(s) are for it?
On Sun, Jun 4, 2017 at 1:10 AM, Eliot Miranda <eliot.miranda(a)gmail.com>
wrote:
>
> On Jun 3, 2017, at 2:22 AM, Ben Coman <btc(a)openinworld.com> wrote:
>
> On Sat, Jun 3, 2017 at 5:57 AM, Eliot Miranda <eliot.miranda(a)gmail.com>
> wrote:
>>
>>
>> On Fri, Jun 2, 2017 at 9:54 AM, K K Subbu <kksubbu.ml(a)gmail.com> wrote:
>>
>>>
>>> On Friday 02 June 2017 04:50 PM, Holger Freyther wrote:
>>>
>>>>
>>>> +1. I would go further and propose that plugins, other than those
>>>>> which are integral to VM[1], be moved into separate projects and
>>>>> built in their own tree with no references to vm source paths. They
>>>>> should be free to organize their platform-specific and
>>>>> platform-independent code and makefiles.
>>>>>
>>>>
>>>> why? How many plugins are used that don't ship with the pharo-vm (or
>>>> squeakvm)? I have to say I like the pharo-vm approach a lot where the
>>>> VMMaker* packages and generated sources sit next to each other.
>>>>
>>>
>>> My proposal is to loosen build time coupling, not packaging. Nothing
>>> prevents plugins built out of tree from being built, installed and shipped
>>> together. Supporting out of tree plugins allows others to extend a VM even
>>> long after its release.
>>>
>>> Eliot has already responded to your example in detail, so I will not
>>> dwell upon it further, except to point out that we need a command line tool
>>> to generate a .c file directly from the plugin's mcz file. Then we can
>>> create a Make rule to compile a .o and get rid of .c file. e.g.
>>>
>>> FooPlugin.o : FooPlugin.mcz
>>> squeak -headless vmmaker.image gencc.st FooPlugin.mcz
>>> cc -c ... FooPlugin.c
>>> rm -f FooPlugin.c
>>>
>>
>> Over my dead body. The make must *NEVER* be done like this. The
>> problems are many:
>> - the VM so produced is undebuggable. There is no source for
>> FooPlugin.c; it has been deleted
>> - the build is slow; building a VMMaker image (the necessary prerequisite
>> for st2c) takes a long time, mush longer than the st2c step
>> - unless FooPlugin.mcz is versioned then the VM so produced is
>> unversioned and if FooPlugin.mcz /is/ versioned then there's no reason why
>> the plugin can't be produced using the normal workflow, generate C source,
>> check it in to opensmalltalk/vm, build, and that way we have a versioned
>> entity and break the dependency of builds requiring VMMaker
>>
>>
> And I remember it mentioned a while ago, the bug is sometimes in the
> generation-code, so you want historical access to both mcz and generated-C
> to help debug that.
>
> +1
>
>
> Look, I just spent several years persuading the Pharo community that
>> generating source and then building was a really bad idea and they should
>> shelve it and now someone is trying to bring it back. No, no, and thrice
>> times, no.
>>
>
>
> I think Pharo has not completely abandoned generating C-sources, although
> its now a sideline activity...
>
>
> It's a CI activity to test the current state of the VMMaker package. This
> is useful.
>
So we should do this under the OpenSmalltalk-VM project?
On Wed, May 31, 2017 at 2:48 PM, Esteban Lorenzano <estebanlm(a)gmail.com>
> wrote:
>>
>>
>> In any case, in the “pharo side” of things, we keep a parallel CI process
>> who does this:
>>
>> 1) there is a development branch that keep all sources we need: the osvm
>> *and* all monticello packages (filetree format) acting as a mirror that
>> merges changes and executes all building process (including generating
>> sources) and testing (we run all pharo tests as a way to test the VM). This
>> is done just for being sure the CI is green and no artifact is created.
>> This is how I have early reported problem in VM generation before.
>> 2) in parallel the build follows the “normal” process on opensmalltalk-vm
>> and everything is build there. This generates a “latest” build which is
>> published in pharo servers so people can test (but this binary could be
>> wrong as is not something declared as “stable”
>>
>
> I do wonder from time to time is why the C-generation can't be done as
> part of an opensmalltalk-vm CI process? The manual process is a bit
> opaque, and a CI process would be better than documentation. One
> possibility of doing it via CI could be generating from both Squeak and
> Pharo and checking the output is identical (assuming that might be useful).
>
> Yes this would be useful. Also there's still at least one place in the
> CoInterpreter source that instability in the type inference code that flips
> the type of a variable between sqInt & usqInt and automation generation
> might help track that down.
>
> IIRC Eliot didn't want the generated-C updated in the Cog branch for every
> VMMaker commit.
>
>
> Right. No point doing so for changes to simulation only code, or to
> experimental code. Or on every commit to a package that contains a
> translated primitive, a form of primitives we need to eliminate by
> rewriting as conventional ones.
>
> But the real issue is in not committing changes to generated code that
> hasn't changed. See scripts/revertUnchangedPlugins.
>
> If there is, say, a Slang change that has limited effects (only a subset
> of plugins, or only the interpreter file, etc) we don't want to commit new
> versions of the essentially unchanged code because it obscures which commit
> leads to a bug.
>
> Plus there's your observation about Slang bugs. I want to eyeball the
> changes in at least one representative generated file to look for
> unintended consequences and assure myself that source generation is sane.
>
> A way to conform to this would be the CI updating the generated C in a
> side branch, and only when deemed worthy it could be integrated into the
> Cog branch via a simple merge rather than a one-time manual generation run.
>
>
> Eliot, If you're receptive to this idea, I'd be interested in working on
> this. What concerns would you have with it?
>
>
> 1. the process identifies files that only change in version stamp (if
> essentially unchanged) and /not/ commit this subset. Right now that
> happens with a script on plugins (cuz they're relatively simple) and by my
> VMMaker image that tracks commits only producing source for classes whose
> timestamps have changed
>
> 2. that the process maintains the separation between generation followed
> by check in from build from checked in source.
>
> 3. that people who work on the Smalltalk side of the business not get lazy
> in eyeballing the generation process cuz Slang issues can be subtle and
> tedious to track down. IME, human supervision remains useful here
>
One extra thing, there feels like as explosion of *src* folders in the root
directory. This could be an opportunity to clean these up into a
subdirectory (and a side benefit is I can experiment without touching
existing srcs)
So I'd like to run a possible structure past you...
opensmalltalk-vm/
VMMaker/
mc-mirror/ #versioned in filetree format
generated/ #versioned - initial generation checked into a separate
branch until manual review and merge.
src/
stacksrc/
nsspursrc/
nsspur64src/
nsspurstacksrc/
nsspurstack64src/
spursrc/
spur64src/
spurstack64src/
spurstacksrc/
spurlowcodesrc/
spurlowcodestacksrc/
spurlowcode64src/
spurlowcodestack64src/
spursistasrc/
spursista64src/
build/
Makefile
CI-build-scripts.sh
generated.squeak/ #not versioned, build host only
src/
spursrc/
spur64src/
...etc
generated.pharo/ #not versioned, build host only
src/
spursrc/
spur64src/
...etc
When http://source.squeak.org is updated, this would trigger the CI copy it
into "mc-mirror"
then build both "generated.squeak/" & "generated.pharo/". The two would
be cross compared and if identical,
then "generated/" would be updated with substantive changes only.
Both "mc-mirror/" and "generated/" would be checked into a
"source-generation" branch
that would kick off a regular CI vm build and test (only if there were
substantive changes).
The advantage being that github comments can be attached to individual
lines of code to facilitate public review and discussion.
After manual review, the source-generation branch can be merged via PR to
perform the regular CI test run inclusive of any platform changes.
cheers -ben
This might be good news for some. I'm not sure whether it helps JITing.
Apple made several changes to the App Store Review Guidelines during WWDC
this week, including an easing of the prohibition against downloading and
executing code on an iOS device. The ban on executable code remains intact,
but rule 2.5.2 [1] now also provides that: Apps designed to teach, develop,
or test executable code may, in limited circumstances, download code
provided that such code is not used for other purposes. Such apps must make
the source code provided by the Application completely viewable and
editable by the user.
cheers -ben
[1]
https://developer.apple.com/app-store/review/guidelines/#software-requireme…