Vm-dev March 2017

vm-dev@lists.squeakfoundation.org

39 participants
154 discussions

StackInterp/Cog>>transferTo: - assertValidExecutionPointe:r:s differences
by Ben Coman 06 May '18

06 May '18

Just curious about the difference in instructionPointer between these two... StackInterpreter>>transferTo: self assertValidExecutionPointe: instructionPointer + 1 r: framePointer s: stackPointer. CoIntepreter>>transferTo:from: self assertValidExecutionPointe: instructionPointer r: framePointer s: stackPointer. cheers -ben

3 8

64bits Pharo VM for windows
by Nicolas Cellier 19 May '17

19 May '17

Hi, I've built a 64bits pharo.stack.spur VM for windows on my machine, and I'm uploading the changes to opensmalltalk-vm in branch build_pharo_win32_with_cygwin If the appveyor job correctly succeed, I will emit a pull request. The VM does not have the SqueakSSL plugin yet. The 64bits squeak/pharo.cog.spur JIT for windows is still to come, but I did not work on it for a few months... One thing at a time. Let's cross finger Nicolas

4 18

Re: [Vm-dev] Freeze after Morph Activity
by Dan Norton 15 May '17

15 May '17

Hi Levente, I directed stdout to a file and prior to the freeze, did: pkill -USR1 -n -x squeak a few times and noticed "stack overflow" mentioned twice. After the freeze, pkill sent nothing else to stdout. Maybe something in stderr? I'll try again and concatenate stderr to the end of stdout. Meanwhile, the file with the stack overflows is attached. Thanks for the help. - Dan On Sat, 28 Jan 2017,*Levente Uzonyi* wrote: >Hi Dan, >You can send the USR1 signal to the VM process to make it write some debug >information to the console. With this information you can easily tell >what's happening. I suspect it's stuck in a long GC. >Levente >On Fri, 27 Jan 2017, Dan Norton wrote: >>//>>/On Debian8, CogSpur64 5.0-201612221637, Cuis 5.0 3043... />>//>>/after lots of user interaction, views opening and closing, and />>/animation, the image becomes unresponsive. Cmd+. does nothing and the />>/clock in the Cuis taskbar no longer updates. The length of time to />>/produce this varies from 5 to 20 minutes. />>//>>/There seems to be no dump and no log. 'Smalltalk garbageCollectMost' />>/reports 25550736 to 30517872 over 34 samples. />>//>>/Sorry to be so vague. This has occurred with several images - sometimes />>/scrolling through a senders list, or stepping through a debugger, but />>/repeatably with one of my images. I can supply this image, warmed up so />>/that it might not take so long to reproduce the problem if you would />>/like. Or give me a hint as to how to narrow down the problem /

10 65

Amazing ARM simulator experience
by Eliot Miranda 03 May '17

03 May '17

Hi All, just had to tell people about this morning's experience using the ARM simulator. I've been building Smalltalk VMs since 1983, so 33 years. My first on the Three Rivers PERQ was dog slow. My undergraduate project was done the the National Semiconductor 32016 based Whitechapel Computer Works workstation and its and 32032-based successor. This morning I was revamping register management in Cog's ARMv5/v6 back end, making more registers available by using two of the C argument registers for two of registers the Cogit assigns to various fixed tasks, and using store and load multiple instructions to save and restore concisely the registers around calls into the runtime. Remember the architecture here. The simulator generates ARM machine code into the ByteArray that re[resents the address space, holding all of generated machine code, a small C stack, and the Smalltalk heap. A plugin, derived from Gdb's ARM simulator written in C, interprets that machine code, running for a couple of milliseconds at a time in a loop, applying breakpoint calculations, and asserts on every call into the run-time, and that calls into the run-time and accesses of variables in the simulator is done by using illegal addresses in the generated machine code. Each illegal access causes the Gdb-derived machine code interpreter to return with an error, this error is turned into an exception, the handler for which maps the specific illegal address into a variable or message selector, accesses the variable or activates the message selector, providing the result, and allowing the execution to continue. In changing the register management I had a test case that worked, an image that prompted for an expression ands evaluated it, which worked both in the simulator and with the generated VM. But the real VM, crashed when used on a proper image. So to debug I started launching the real interactive image in the simulator. Well, the amazing experience is that that image, whose machine code is being _interpreted by a C program_ feels /faster/ than my 32016 based implementation back in 1984-ish. Quite amazing. I can open windows, type text, access source code (was playing with Message Names) etc. It's sluggish, but usable. Amazing how fast modern machines are. All on my 2012 vintage 2.2 Ghz Core i7 MacBook Pro. I'm blown away :-) _,,,^..^,,,_ best, Eliot

5 7

Crashes on snapshot with the new compactor
by Eliot Miranda 17 Apr '17

17 Apr '17

Hi All, a number of people are being affected by crashes on snapshotting the image, the worst possible time for a crash. There is a bug in the new compactor that unfortunately bites when saving. The compactor is invoked as part of a full garbage collect after the garbage collector has feed unreachable objects. Normally the new compactor makes only a single pass through the heap, which may not move all the objects that are possible to move. (The amount of objects that can be moved in a single pass is limited by available free space.) But on snapshot the compactor makes as may passes as are necessary to slide all movable objects down as far as possible. Unfortunately there is a bug in this second pass. Fixing this bug is now my priority. I have an example image from Esteban Lorenzano to test. I am asking anyone else that can provide an image that reliably crashes when trying to save it to make the image and changes available to me for testing if possible. In the mean time one may be able to work around the problem by doing a full garbage collect before snapshot. This should do a GC with a single compaction pass which should not fail, and then make it much more likely that the GC during snapshot will do a single compaction pass, since fewer objects should be mobile after the single pass compaction in the explicit GC. To do this in Pharo I would put a full gc here: SessionManager>>snapshot: save andQuit: quit | isImageStarting snapshotResult | ChangesLog default logSnapshot: save andQuit: quit. >> SmalltalkImage current primitiveGarbageCollect. self currentSession stop: quit. "Image not usable from here until the session is restarted!" ... In Squeak I would put a full GC here: snapshot: save andQuit: quit withExitCode: exitCode embedded: embeddedFlag "Mark the changes file and close all files as part of #processShutdownList. If save is true, save the current state of this Smalltalk in the image file. If quit is true, then exit to the outer OS shell. If exitCode is not nil, then use it as exit code. The latter part of this method runs when resuming a previously saved image. This resume logic checks for a document file to process when starting up." | resuming msg | Object flushDependents. Object flushEvents. ... Smalltalk processShutDownList: quit. >> SmalltalkImage current primitiveGarbageCollect. Cursor write show. save ifTrue: [resuming := embeddedFlag ifTrue: [self snapshotEmbeddedPrimitive] ifFalse: [self snapshotPrimitive]] "<-- PC frozen here on image file" ifFalse: [resuming := false]. I do apologise for the bug. I hope it will be fixed within a few days. _,,,^..^,,,_ best, Eliot

2 3

Idle time and sleep/wake notification
by Eliot Miranda 12 Apr '17

12 Apr '17

Hi All, I've just added tallying of idle time in the VM so that the system report can give more meaningful overheads to the GC information it reports. Currently the VM tallies idle time in the innards of relinquishProcessorForMicroseconds:. But it also needs to tally idle time across SLEEP/WAKE events. May I ask experts in the Mac OS, X11 and Windows worlds to have a loop at event processing and either implement tallying idle time here or reporting back on what work is required or (as I fear in the case of X11/Unix) whether it is unsupported. Here's an example of the nonsense one gets back after waking one's laptop: Virtual Machine Statistics -------------------------- uptime 14h 59m 26s (runtime 11h 53m 33s, idletime 3h 5m 53s) memory 338,690,048 bytes old 333,689,744 bytes (98.5%) young 4,111,360 bytes (1.2%) used 314,964,392 bytes (93%) free 21,991,672 bytes (6.5%) GCs 3,430 (15733.6 ms between GCs 12482 ms runtime between GCs) full 8 totalling 1,331 ms (0% runtime), avg 166.4 ms marking 536 ms (40.3%) avg 67 ms, compacting 795 ms (59.7%) avg 99.4 ms scavenges 3,422 totalling 2,511 ms (0.01% runtime), avg 0.7 ms tenures 818,889 (avg 239 tenures per scavenge) Since last view 2,463 (21394 ms between GCs, 17285 ms runtime between GCs) uptime 52695.2 s, runtime 42574 s, idletime 10121.2 s full 2 totalling 322 ms (0% runtime), avg 161 ms scavenge 2,461 totalling 2,029 ms (0% runtime), avg 0.8 ms tenures 16,733 (avg 6 tenures per scavenge) whereas while the machine stays awake we get good information: Virtual Machine Statistics -------------------------- uptime 1m 58s (runtime 0m 21s, idletime 1m 37s) memory 281,018,368 bytes old 276,018,064 bytes (98.2%) young 4,111,360 bytes (1.5%) used 251,951,208 bytes (89.7%) free 26,712,592 bytes (9.5%) GCs 37 (3203.5 ms between GCs 569.7 ms runtime between GCs) full 0 totalling 0 ms (0% runtime), avg 0 ms scavenges 37 totalling 13 ms (0.06% runtime), avg 0.4 ms tenures 648 (avg 17 tenures per scavenge) Since last view 30 (3498 ms between GCs, 590 ms runtime between GCs) uptime 104.9 s, runtime 17.7 s, idletime 87.2 s full 0 scavenge 30 totalling 11 ms (0.1% runtime), avg 0.4 ms tenures 0 _,,,^..^,,,_ best, Eliot

3 4

Unix heartbeat thread vs itimer
by Guillermo Polito 12 Apr '17

12 Apr '17

Hi, I was checking the code in sqUnixHeartbeat.c to see how the heartbeat thread/itimer worked. It somehow bothers me that there are different compiled artifacts, one per option. What do you think about having a VM that manages that as an argument provided when we launch the VM? This would add some flexibility that we don't have right now because we make the decision at compile time. The code in sqUnixHeartbeat.c is not a lot nor very complex, it should not be difficult to do... Also, what would be the drawbacks besides an increase on the vm size? Guille

11 35

Build pharo vm on ubuntu 16.04 64 bits ?
by Javier Pimás 03 Apr '17

03 Apr '17

Just trying to build the vm on a vanilla ubuntu 16.04 64bits. Cloned from opensmalltalk-vm entered build.linux32x86/pharo.cog.spur and run ./makeallclean It fails like this: ... checking for shared library run path origin... done checking for libssl... no checking for libgcrypt... no checking for libbcrypt... no checking for libcrypt32... no checking for ntdef.h... no checking for ntstatus.h... no checking whether SecureZeroMemory is declared... no configure: error: No crypto library found! Try --with-libssl-prefix=PATH or --with-libgcrypt-prefix=PATH or --with-wincng on Windows .,. I installed all the following packages but didn't help, any clue? sudo apt-get install cmake zip bash-completion ruby git xz-utils debhelper devscripts sudo apt-get install libc6-dev:i386 libasound2:i386 libasound2-dev:i386 libasound2-plugins:i386 libssl-dev:i386 libssl0.9.8:i386 libfreetype6-dev:i386 libx11-dev:i386 libsm-dev:i386 libice-dev:i386 sudo apt-get install build-essential gcc-multilib g++ sudo apt-get install libgl1-mesa-dev libgl1-mesa-glx:i386 sudo apt-get install binutils:i386 g++-4.6:i386 gcc-4.6:i386 sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so /usr/lib/i386-linux-gnu/libGL.so sudo ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/mesa/libGL.so sudo apt-get install uuid-dev:i386 sudo apt-get install subversion cheers, Javier -- Javier Pimás Ciudad de Buenos Aires

5 9

Re: [Vm-dev] [Pharo-dev] Problem with OSSubprocess / signals / heartbeat ?
by Guille Polito 02 Apr '17

02 Apr '17

Ok, so following a bit on this. I'll summarize some of our findings, some of them maybe obvious for some people in this list. I saw that there are actually two different kind of VMs for *nix [1] : - threaded heartbeat - itimer + signal heartbeat Specially, I'd like to cite the following paragraph for the lazy: A distinction on linux is between VMs with an itimer hearbeat or a threaded heartbeat. VMs with an itimer hearbeat use setitimer to deliver a SIGALRM signal at regular intervals to interrupt the VM to check for events. These signals can be troublesome, interrupting foreign code that cannot cope with such signals. VMs with a threaded heartbeat use a high-priority thread that loops, blocking on nanosleep and then interrupting the VM, performing the same function as the itimer heartbeat but without using signals. These VMs are to be preferred but suport for multiple thread priorities in user-level processes has only been available on linux in kernels later than 2.6.12. So, I downloaded the heartbeat squeak VM from bintray [2]. This VM requires so deploy some configuration files in /etc/security [2]. Under this configuration OSSubprocess worked like a charm (or I did not found the issue again so far). Now, this heartbeat threaded VM is the recommended in the README file, and we see that OSSubprocess generates the exact issue stated. The main problem remains for the moment since Pharo's default download includes not this VM but the itimer one. I talked with Esteban about it and he was aware of these two VM flavours, and the reason why we are using the itimer one is the need to deploy those permission files in /etc, which makes installation a bit less automatic. Cheers, Guille [1] https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/e17db79411cfec767e04… [2] https://bintray.com/opensmalltalk/vm/download_file?file_path=cog_linux32x86… [3] https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux -------- Original Message -------- > > >> I asked Santa an OSSubprocess that won't hang :) >> > > > I really appreciate to see such not so small polar elfves helping santa :) > > > > Stef > > > > > > > >> Thanks, >> >> Guille >> >> >> >> >> >> -------- Original Message -------- >> >>> >>> >>>> On 19 Dec 2016, at 14:41, Mariano Martinez Peck >>>> <marianopeck(a)gmail.com> wrote: >>>> >>>> >>>> >>>> Hi guys, >>>> >>>> >>>> >>>> Guille Polito kept one of these images if someone can give us a >>>> hand. He also proposed the great idea of using `strace` to see what >>>> was going on. He (together with Pable Tesone) suspected that the >>>> heartbeat could be interrupting the `clone()` function which is (I >>>> think) called internally by the 'posix_spawn()' which is the one >>>> used by OSSubprocess. >>>> >>>> >>>> >>>> When these images are "hung" they found at a infinitive loop like this: >>>> >>> Okay, how many child processes do you have at that point? How many >>> processes does the system have? >>> >>> >>> >>> >>> >>>> [pid 17477] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> [pid 17477] gettimeofday({1482152630, 593498}, NULL) = 0 >>>> >>>> [pid 17477] sigreturn() (mask []) = 120 >>>> >>>> [pid 17477] clone(child_stack=0, >>>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, >>>> child_tidptr=0xf7578768) = ? ERESTARTNOINTR (To be restarted) >>>> >>>> [pid 17477] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> [pid 17477] gettimeofday({1482152630, 600126}, NULL) = 0 >>>> >>>> [pid 17477] sigreturn() (mask []) = 120 >>>> >>>> [pid 17477] clone(child_stack=0, >>>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, >>>> child_tidptr=0xf7578768) = ? ERESTARTNOINTR (To be restarted) >>>> >>> so above.. 7ms between the two gettimeofday calls? Nothing else? Set >>> a breakpoint on clone/fork in gdb and look at the c-stack at this >>> point? Could you strace with timestamps to see how much time is >>> spent? Is the process suspicious in other ways? >>> >>> >>> >>> So yes.. sounds like clone doesn't complete.. the question is why? >>> Is it out of resources? Is something in the VM blocking longer than >>> the heartbeat, is the heartbeat more frequent than expected? >>> >>> >>> >>> >>> >>>> As you can see, there is a SIGALARM involved. It also looks like >>>> the `gettimeofday` is used by the heartbeat ? Could it be that >>>> someone the heartbeat is interrupting the `clone()` ? >>>> >>>> >>>> >>>> Guille also showed me the `strace` output with a regular / working >>>> image: >>>> >>>> >>>> >>>> >>>> >>>> [pid 18647] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> [pid 18647] gettimeofday({1482152829, 481014}, NULL) = 0 >>>> >>>> [pid 18647] sigreturn() (mask []) = -1 EINTR (Interrupted >>>> system call) >>>> >>>> [pid 18647] getitimer(ITIMER_REAL, {it_interval={0, 2000}, >>>> it_value={0, 1917}}) = 0 >>>> >>>> [pid 18647] recvmsg(3, 0xff7b0734, 0) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> [pid 18647] select(4, [3], [], [3], {0, 1000}) = 0 (Timeout) >>>> >>>> [pid 18647] getitimer(ITIMER_REAL, {it_interval={0, 2000}, >>>> it_value={0, 797}}) = 0 >>>> >>>> [pid 18647] recvmsg(3, 0xff7b0734, 0) = -1 EAGAIN (Resource >>>> temporarily unavailable) >>>> >>>> [pid 18647] select(4, [3], [], [3], {0, 1000}) = ? ERESTARTNOHAND >>>> (To be restarted if no handler) >>>> >>>> [pid 18647] --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} --- >>>> >>>> >>>> >>>> >>>> >>>> Does anyone have any hint here? >>>> >>> Get timestamps in there. How long does it take to fail/end in this >>> situation? >>> >>> >>> >>> holger >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >> >> >> >> >> >> > > > > > >

7 18

Re: [Vm-dev] [Pharo-dev] Iceberg with SSH on Linux
by Eliot Miranda 31 Mar '17

31 Mar '17

On Thu, Mar 30, 2017 at 8:20 PM, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: > Hi Alistair, > > On Thu, Mar 30, 2017 at 9:32 AM, Alistair Grant <akgrant0710(a)gmail.com> > wrote: > >> Hi Eliot, >> >> On 30 March 2017 at 18:15, Eliot Miranda <eliot.miranda(a)gmail.com> wrote: >> > >> > On Mar 30, 2017, at 2:58 AM, Pavel Krivanek <pavel.krivanek(a)gmail.com> >> > wrote: >> >> >> >> Yes, the same priority. See: >> >> >> >> The VM may have issues with clock jitter due to the heartbeat thread >> >> not running at elevated priority. >> > >> > >> > It better /not/ be. This is wrong. The heartbeat thread /must/ run at >> a >> > higher priority for the VM not to lock up when it becomes compute >> bound. I >> > thought the resolution was that the VM /will/ try and raise the >> priority if >> > the heartbeat thread but will /not/ exit if it fails. This is /very/ >> > different from not trying to raise the priority at all. >> >> As Esteban wrote, the vm is trying to raise the priority. You can see >> the change at: >> >> https://github.com/pharo-project/pharo-vm/commit/32f321583c6 >> 9ca27e61ffaff6decc2a3e4b6ca5e >> >> I'd like to just make sure I understand your comment correctly: >> >> Are you saying that the VM will lock up if it gets in to, e.g., an >> endless loop, so that Ctrl-. fails to interrupt. Or is it more >> serious than that? Can the VM become locked up for just a long >> running process (I guess it will be locked while the process is >> running, but will eventually go back to normal once the process >> completes)? >> > > The issue is that delays are recognized by periodically checking the time > against the next delay expiry. It is the heartbeat thread that causes > JITed Smalltalk code to break out of its execution to check for events, > including the next delay expiry. So if the heartbeat is at the same > priority as the JITed Smalltalk code, and the Smalltalk code is in tight > enough a loop (one that does not provoke a GC, does not do any process > switches) the heartbeat thread will not run and the delay expiry will never > be tested for. > Something like the following should run forever if the heartbeat has the same priority as the VM thread (and should be tried in a headless configuration too): [| infiniteLoop | infiniteLoop := [| sum | sum := 0. [sum < 10] whileTrue: [sum := sum + (sum even ifTrue: [1] ifFalse: [-1])]] newProcess. infiniteLoop resume. Processor activeProcess priority: Processor activePriority + 1. (Delay forSeconds: 1) wait. infiniteLoop terminate. Processor activeProcess priority: Processor activePriority - 1] timeToRun 1001 Also, a change was made to the travis test setup in: >> >> https://github.com/pharo-project/pharo-vm/commit/5418a415e92 >> 97f601f6d57ee732fd7fd942da08c >> >> The comment is: "No need to raise rtprio limit anymore" >> >> From your comments above, this doesn't seem accurate. >> > > Right. What seems to occur is that certain linux distress don't require a > conf file to allow raising the priority of a thread. Remember that rtprio > is not the issue. We only require that the heartbeat thread has higher > priority than the VM thread. > > >> I'm in the middle of adding some details to the text that is printed >> out by the threaded heartbeat vm, and from what you're saying, maybe >> we should add some more warnings. >> > > The VM should (& I believe already does) print an error if the attempt to > raise the heartbeat thread priority fails. My underatanding is that the > positive change is simply not to exit if the attempt fails. > > > >> >> >> > The resolution I thought we had reached means that if correctly >> installed >> > the VM can work correctly, and will continue to work correctly if and >> when >> > linux removes the annoyance of the conf files. The resolution you >> describe >> > Pavel (I hope inaccurately) means the threaded heartbeat VM will never >> work >> > correctly. >> >> Thanks, >> Alistair >> > > _,,,^..^,,,_ > best, Eliot > -- _,,,^..^,,,_ best, Eliot

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev March 2017