Vm-dev

vm-dev@lists.squeakfoundation.org

10 participants
17030 discussions

lowspace signalling and handling issues
by Tim Rowledge 04 May '05

04 May '05

After having problem trying to debug some TK4 code that blew up with lowspace problems but never let me catch and debug, I spent some time adding the lowspace-process stuff we recently discussed. I had to make a few alterations to match it up with the latest 64bit clean code but no problems with that part. After building a VM I started testing with some of the methods in SystemDictionary 'memory space' - in particular #useUpMemory. It is perhaps fortunate that I did since the other #useUp* methods pretty much work once the lowspace process is caught by the vm and passed up to the image. After a _lot_ of head scratching by John & I we found that with a gazillion tiny objects (Links are the smallest possible objects that can exist on their own, plain Objects would have to be contained in a collection and so would cost the same 3 words per object) cause a catastrophic GC explosion. What happens is that memory fills up until we get to signal lowspace and then we are in danger. Depending upon the exact size of object memory in use the 200kb used as the lowSpaceThreshold can be gobbled up in one swallow by the initializeMemoryFirstFree: method making sure there is a byte per object that survived the markPhase. In using useUpMemory we can get to having 4 bytes of free space when the next allocate is attempted.... Ka-Boom. Expanding the lowSpaceThreshold (along with the VM changes to report the process and avoid the accidental problem of interrupting eventTickler) to a couple of mb makes it ok on my machine and the threshold can be a lot lower with the other tests that create bigger (hence fewer) objects (hence smaller fwdTable needs). In the worst case, we could have a very large OM filled with very small objects all surviving markPhase; in such a case we would need an additional 1/12 of OM available for the fwdTable. So for a 30Mb objectmemory we ought to set the lowSPaceThreshold to 30/13 => 2.31Mb + actual space needed to run the notifier/debugger etc for reasonable safety. Or hide the 2.31 Mb away so the image never even knows it is there. If you are using virtual memory and a limit of 512Mb then you should perhaps secrete 40Mb some where safe. This assumes that we really need to have one byte per object of course. The original rationale was to keep the number of compact loops down to eight (see Dan's comment in initializeMemoryFirstFree:) for Alan's large demo image. The nicest solution would be to come up with a way to do our GC & compacting without needing any extra space. Commence headscratching now... John suggested making sure the fwd gets less than the byte-per-object if things are tight, and accpting the extra compaction loops. Good news- with the vm change and 2Mb lowSpaceThreshold I can probably go back and find my TK4 problem(s). Bad news- consider Tweak. With lots of processes whizzing away, merely stopping the one that did the allocation and triggered the lowspace is not going to be much good. Stopping everything except the utterly essential stuff to debug the lowspace will be needed. Probably. More bad news- somehow, going from VMM37b5 to b6 cost 40% of performance on my machine :-( Bugger. tim -- Tim Rowledge, tim(a)sumeru.stanford.edu, http://sumeru.stanford.edu/tim Every bug you find is the last one.

3 7

FloatArrayPlugin & 64bitting
by Tim Rowledge 04 May '05

04 May '05

FloatArrayPlugin doesn't feel well when we generate it from the 64bit VMMaker alpha. It has a clause:- } rcvrPtr = ((float *) (interpreterProxy->firstIndexableField(rcvr))); /* Check if any of the argument's values is zero */ argPtr = ((float *) (interpreterProxy->firstIndexableField(arg))); for (i = 0; i <= (length - 1); i += 1) { if ((longAt(argPtr + i)) == 0) { return interpreterProxy->primitiveFail(); } which upsets the typing of longAt since it expects an sqInt. I can trivially alter it to cast the rcvrPtr to an sqInt for the use of the longAt and it compiles/runs ok. I have no idea if a 64bit machine will have floats of 64 bits (ie the same as sqInt) or if they stay 32bit (ie we have to use plain int, which may upset longAt on a 64bit machine) or what. And since I have no 64bit setup, I can't say I'm massively bothered right now. But somebody probably is and if anyone cares to suggest the nicest fix I'll happily include it. tim -- Tim Rowledge, tim(a)sumeru.stanford.edu, http://sumeru.stanford.edu/tim No single raindrop believes it is to blame for the flood

3 9

Re: FloatArrayPlugin & 64bitting
by Dave Mason 30 Apr '05

30 Apr '05

>>>>> On Fri, 29 Apr 2005 16:32:09 -0400, Dave Mason <dmason(a)sarg.ryerson.ca> said: I have the source (without VMMaker because of a permission problem), and I looked at FloatArrayPlugin. The code there looks fine if the longAt is replaced with floatAt or intAt. It stores a block of floats starting at the first field in the object. However there definitely is *potential* for a problem if you had a collection of fields some of which were floats. Here's where there could be a problem on 64 bit, where sizeof(sqInt)==8 and sizeof(float)==4. Consider a block of memory: abcdefgh ijklmnop qrstuvwx 1stIndexableField returns the address of 'a'; assume the following code: sqInt argOop = oopForPointer(interpreterProxy->firstIndexableField(arg)); float* argFlo = (float *)(interpreterProxy->firstIndexableField(arg)); sqInt* argOopp = (sqInt *)(interpreterProxy->firstIndexableField(arg)); floatAt(argOop) gets abcd floatAtPointer(argFlo) gets abcd floatAtPointer(argOopp) gets abcd Fine so far, but: floatAt(argOop+1) gets bcde (or error) floatAtPointer(argFlo+1) gets efgh floatAtPointer(argOopp+1) gets ijkl floatAt(argOop+2) gets cdef (or error) floatAtPointer(argFlo+2) gets ijkl floatAtPointer(argOopp+2) gets qrst As far as I can see, abcd, ijkl, and qrst are the right answers (if they are fields containing floats), with efgh being garbage and ijkl not being the correct answer in the 3rd example. So *anywhere* you are skipping through a bunch of fields, you need to do the pointer arithmetic with sqInt*. As I said before, this is a pretty subtle error, but a quick grep through the code for 1stIndexableField shows a lot of _potential_ problems, although all in plugins, and many of them have to do with arrays, so they probably aren't. All of this may be obvious to Ian and the rest of you, but I'm trying to learn a bit about the code, and this was a useful exercise for me. ../Dave

1 0

VMMaker-tpr.14.mcz permission problems
by Dave Mason 29 Apr '05

29 Apr '05

OK, I found the sources... duh! But when I try to access VMMaker-tpr.14.mcz I get a 403: Forbidden ../Dave

1 0

vm crash in updatePointersinRangeFromto via primitiveArrayBecome (mac vm 3.8.7b5)
by John M McIntosh 28 Apr '05

28 Apr '05

Last night I had an vm crash as I was working on TK4, this is an unusual event so I'll bring it to everyone's attention. I recall I clicked on the squeak desktop and everything went away... This afternoon I'm still working with the image in question without issues. Lately we added some changes by Andreas for correct weak array handling, some changes to how become: works, and my work in VM GC statistical data, so I cann't say which is at fault, if any... In looking I found that my 3.8.7b5 VM had crashed in updatePointersinRangeFromto() at 21777 via calling from primitiveArrayBecome. 21776 fieldOop = longAt(oop + fieldOffset); 21777 if (((fieldOop & 1) == 0) && (((longAt(fieldOop)) & MarkBit) != 0)) { Which resolves to this assembler, and we croak on the (lwz) load word zero high order bits at + 268 attempting to get the word at 0x3618bca2 (illegal) + 256 lwzx r2,r10,r11 3:1 Stall=2, Loop start[2] interp.c:21776 + 260 andi. r0,r2,0x0001 1:1 interp.c:21777 + 264 bne $+280 <updatePointersInRangeFromto + 544> 1:1 interp.c:21777 + 268 lwz r0,0(r2) 3:1 Stall=2 interp.c:21777 + 272 cmpwi cr6,r0,0 1:1 interp.c:21777 + 276 bge+ cr6,$+268 <updatePointersInRangeFromto + 544> 1:1 interp.c:21777 Host Name: ap0.smalltalkconsulting.net Date/Time: 2005-04-26 21:34:29 -0700 OS Version: 10.3.9 (Build 7W98) Report Version: 2 Command: Squeak VM Opt Path: /Users/johnmci/Documents/Squeak3.8.0/build/Squeak 3.8.7Beta5.app/Contents/MacOS/Squeak VM Opt Version: Squeak VM 3.8.7b5 (3.8.7b5) PID: 12999 Thread: 2 Exception: EXC_BAD_ACCESS (0x0001) Codes: KERN_INVALID_ADDRESS (0x0001) at 0x3618bca2 Thread 0: 0 libSystem.B.dylib 0x90007878 mach_msg_trap + 0x8 1 libSystem.B.dylib 0x900073f8 mach_msg + 0x38 2 com.apple.CoreFoundation 0x901c16e0 __CFRunLoopRun + 0x350 3 com.apple.CoreFoundation 0x901c5e6c CFRunLoopRunSpecific + 0x148 4 com.apple.HIToolbox 0x92885f60 RunCurrentEventLoopInMode + 0xac 5 com.apple.HIToolbox 0x9288c6c8 ReceiveNextEventCommon + 0x17c 6 com.apple.HIToolbox 0x928ae1a0 AcquireNextEventInMode + 0x48 7 com.apple.HIToolbox 0x928ab670 RunApplicationEventLoop + 0x88 8 org.squeak.Squeak 0x0006dab0 main + 0x390 9 org.squeak.Squeak 0x00009118 _start + 0x188 (crt.c:267) 10 dyld 0x8fe1a278 _dyld_start + 0x64 Thread 1: 0 libSystem.B.dylib 0x90018e18 semaphore_timedwait_signal_trap + 0x8 1 libSystem.B.dylib 0x9000e9d4 _pthread_cond_wait + 0x268 2 ...ple.CoreServices.CarbonCore 0x902da570 TSWaitOnSemaphoreCommon + 0xb0 3 ...ple.CoreServices.CarbonCore 0x902ee4f0 TimerThread + 0x3c 4 libSystem.B.dylib 0x90024910 _pthread_body + 0x28 Thread 2 Crashed: 0 org.squeak.Squeak 0x000c3f9c updatePointersInRangeFromto + 0x10c 1 org.squeak.Squeak 0x000c19fc becomewithtwoWaycopyHash + 0xfc 2 org.squeak.Squeak 0x000c1b00 primitiveArrayBecome + 0x30 3 org.squeak.Squeak 0x000ca840 interpret + 0x1af0 4 libSystem.B.dylib 0x90024910 _pthread_body + 0x28 PPC Thread State: srr0: 0x000c3f9c srr1: 0x0000d030 vrsave: 0x00000000 cr: 0x24000244 xer: 0x00000004 lr: 0x000c19fc ctr: 0x0000004e r0: 0x00000000 r1: 0xf0203d90 r2: 0x3618bca2 r3: 0x0000005c r4: 0x07348b6c r5: 0x00000160 r6: 0x4d616369 r7: 0x00107fc0 r8: 0x3a2f2f2f r9: 0x00000059 r10: 0x071b8734 r11: 0x00000138 r12: 0x00000008 r13: 0x00000000 r14: 0x00000000 r15: 0x00000000 r16: 0x00000000 r17: 0x00000000 r18: 0x00000000 r19: 0x00000000 r20: 0x00000000 r21: 0x054c962c r22: 0x07348530 r23: 0x07348510 r24: 0x00107fc0 r25: 0x000f1b18 r26: 0x00000001 r27: 0x00000001 r28: 0x00107fc0 r29: 0x00000004 r30: 0x07348b60 r31: 0x07348b58 Binary Images Description: 0x1000 - 0xebfff org.squeak.Squeak Squeak VM 3.8.7b5 (3.8.7b5) /Users/johnmci/Documents/Squeak3.8.0/build/Squeak 3.8.7Beta5.app/Contents/MacOS/Squeak VM Opt Changes leading up to this VM are: 3.8.7b5 Fix logic to allow you to start VM from command line, changed sqMacFileLogic.c makeHFSFromPosixPath added to convert posix cmd line path names to HFS sqMacMain.c add logic to call cmd line parser sqMacUnixCommandLineInterface.c new file logic to parse cmd line sqMacUnixCommandLineInterface.h sqMacWindow.c minor changes to correct compiler errors 3.8.7b4/b3 Limited release for testing 3.8.7b2 sqUIEvents.c changed keyUp/keyDown to supply mac virtual keycode versus unicode, added new parm to keyChar to supply UTF-32 Unicode. As per Andreas request for Tweak. 3.8.7b1 sqUIEvents.c changed keyUp/keyDown logic to match Windows pattern, key repeat becomes keyDown/keyDown/keyUp 3.8.6b6 sqMacMIDI.c add #def so I can compile under code warrior os-9 again sqMacEncoding.c, sqMacEncoding.h Minor changes so I can compile user code warrior os-9 again sqMacUnixInterfaceSound.c Fix problem with empty buffer being full of junk under certain conditions, that causes a sharp sound bite. 3.8.6b5 Change to JMM & AR GC instrumentation patch, collect a few less statistics. Also change carbon VM to use Unix/Quartz sound driver which interfaces to Core Audio stuff. 3.8.6b4 Ensure VMMaker 3.8b2 - the VM code classes and tools for building a working vm for Squeak version > 3.8-6461. is incorporated. Add Andreas's GC improvement changes for * primitiveIsYoung: This answers the question whether an object currently lives in young or in old space. * primitiveIsRoot: Answers the question whether any given object is currently a root for young space. * primitiveRootTable: Answers a snapshot of the current root table. Useful to examine the roots table if the analysis requires complex other operations during which the root table might be modified itself. Note that since this primitive can cause GC there is a small chance that it will give an inaccurate answer. * primitiveRootTableAt: Answers a single element of the root table (by one-based index). This primitive can be used to quickly scan the root table for certain objects. * primitiveSetGCSemaphore: Indicates a semaphore (index) to be signaled whenever a garbage collection occurs. I can see at least two uses uses for the GC semaphore: running cleanup actions (for example after full GCs occured) and dynamic parameter tuning for the GC algorithm itself. Add John's changes for Smalltalk setGCBiasToGrowGCLimit: 16*1024*1024. "Set growth limit before full GC to 16MB" Smalltalk setGCBiasToGrow: 1. Set bias to grow upto GCLimit, this can avoid a problem where we attempt to avoid growing but results in thousands of incremental GC events as we approach a knee in a curve of space used versus the growth/compaction decision. 3.8.6b3 Add in logic for become: to fix issues with changes to become: logic. 3.8.6b2 sqMacWindow.c reduce number of lock/unlock pixel events and flush no more than 91 fps, with flush check every 1/50 or 1/60 and 1/5 of a second. this shaves another 50% off making drawing pixels 1.3% of the time originally taken. 3.8.6b1 sqMacWindow.c, sqUIEvents.c, changed logic to flush screen every 125 fps, this makes pixel level drawing complete in 2% of the time originally taken. 3.8.5b1 Fix issue with gnuify. The offical gnu header file used a different #ifdef for the foo register than an altered gnu header file I was using for many years. This resulted in slower performance, which is fixed in 3.8.5 3.8.4b3 Support for GC statistical data and change to how memory is allocated and tenured. Additional statistical information from the GC routines Additional primitives to set GC semaphore, and to force a tenure Change to make forward logic to look for "foo become: foo" and cause a prim failure. Earlier version of 3.8.4 where distributed to Ffenstri testers. sqMacHostWindow Change calls to move and size window to use GUILocker callbacks to avoid VM crash. sqMacWIndow.c Alter 1/2/4 bit support to pass in bit size of target window, then copy is more efficient. 3.8.3b3 Support for Host menus and windows. Also added back 1bit (B/W) and 2bit & 4bit color as requested by Dan Ingalls for os-x Earlier versions of 3.8.0/1/2/3 were distributed to the TK4 team, not the public. sq.h Added events, EventTypeMenu, EventTypeWindow to handle menu and window events generated by the Host OS and placed on the Squeak Event Queue to be read by EventSensor sqGnu.h Switch to the current file used by Ian for Unix, versus older modified copy. sqMacExternalPrims.c Cleanup of warning message sqMacImageIO.c When you set the image name, we set the title on window 1. sqMacMain.c Do not create the main window until we actually use it. sqMacNSPlugin.c Always refer to window 1, not sure what it means to have other windows in the plugin yet... sqMacUIEvents.c OS-9 Pass cmd-key menu activity up Pass update event bounding box window events up Pass window activate events up Pass menu interaction activity up Pass window events, resize,zoom,close,move,inconize up OS-X Same as above getUIToLock needs to ReleaseEvent on the passed in event for GC reasons sqMacUIMenuBar.c disable edit menu at start, if later we find it enabled, don't touch, otherwise handle os-9 DA issues. sqmacWindow.c sqShowWindow now gets a windowIndex parm. ioShowDisplayOnWindow is added, with logic to build main window at first draw time, also fix some boundary checks, and provide support (again) for 1/2/4 bit color under os-x ioHasDisplayDepth() will return true for 1/2/4/8/16/32 bit color -- ======================================================================== === John M. McIntosh <johnmci(a)smalltalkconsulting.com> 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===

2 1

Re: [Squeak-VMdev] GC improvements
by Andreas Raab 27 Apr '05

27 Apr '05

Just curious: Did we ever come to a conclusion re: GC improvements? I'm asking because I *will* require proper weak array support for Tweak in the not-too-far future and I'd like to see those changes rolled into standard VMs as soon as possible. Cheers, - Andreas

6 13

sq.h? sqConfig.h? interp.h? config.h? - and what goes where???
by Andreas Raab 23 Apr '05

23 Apr '05

Hi - Just tried to build a VM from the latest sources (the first time I tried 64bit) and promptly run into problems. The first ones: It seems that by now we not only have sq.h, sqConfig.h, and sqPlatformSpecific.h (sqGnu.h nonwithstanding) but also interp.h and config.h. Can someone please shed a light on what definitions need to go where? For example, if we have config.h shouldn't interp.h (which contains a single definition) be folded into that? Shouldn't sqConfig.h go away? Since config.h seems to be optional (at least so far Windows managed without it) why can't it be optional? This hole mix of strange header files has lost any sensibility to me. Cheers, - Andreas

3 2

[VM][ENH] LowSpaceAndInterruptHandler-dtl
by David T. Lewis 16 Apr '05

16 Apr '05

These change sets are updates to the VM and to the low space watcher that fix low space interrupt handling and user interrupt handling in Morphic. Under certain conditions the low space watcher was unable to determine the correct process to suspend following a low space signal. These changes permit the VM to remember the identity of the process that caused the low space condition, and to report it to the image through a primitive. The low space watcher changes are backward compatible for use with VMs that do not yet have the new primitive (i.e. broken the same way as was the case in Squeak 3.7). Caveats: - This is done with VMMaker 3-8b2, hence is not updated for 64 bit VM. - I added a numbered primitive, which is probably a bad thing (but I'm not sure where best to put it, possibly MiscPrimitivePlugin?). - There may be a simpler way to accomplish this, but at least this is a working implementation. - I have not tested the interpreter simulator, still need to make sure I did not break it. Cross posted to vm-dev and squeak-dev. Follow-ups should go to vm-dev (unless someone can think of a way to handle this properly without touching the VM). Dave

3 6

64-bit VMs
by Ian Piumarta 14 Apr '05

14 Apr '05

Folks, I've just released 64-bit compatible sources for 3.8, and checked-in both Unix and Cross changes to the TRUNK of the repository. To build a Unix VM from the repository, first check out the trunk as usual, then: mkdir bld cd bld ../platforms/unix/config/configure --with-src=src32 make will build a 32-bit VM. If you say ../platforms/unix/config/configure --with-src=src64 instead then it will build you a 64-bit VM. I have tested both 32- and 64-bit VMs on PowerPC (32-bit native), Intel Pentium and Intel Itanium 2. All appears to work. Note that I have made some changes to the platforms/Cross part of the repository. While I tried hard to keep things as backwardly compatible as possible (e.g., the Unix tree happily compiles pre-3.8 sources as-is) this might have temporarily broken the build for other platforms. My apologies if this is the case, OTOH it's a wonderful forcing function to have the maintainer of your favorite platform get a 64-bit VM out the door Real Soon Now. I've not had chance to update my usual download page to reflect the new 32/64 duality in the distribution, but I did have time to make some archives of source code, 32- and 64-bit images, and 32- and 64-bit VMs for the three architectures mentioned above. You can pick them up here: http://squeak.hpl.hp.com/squeak64/dist3/ wherein: Squeak-3.8a-2.src.tar.gz contains the source code, and a handy one-stop top-levelMakefile at which you can type 'make WIDTH=32' and 'make width=64'. Squeak32-3.8g-6548.image.tar.gz Squeak64-3.8g-6548.image.tar.gz contain 32- and 64-bit images, respectively. (Note that I had to condense the changes in the base 3.8g-6548 image because of reaching the limit on source position stamp.) Squeak32-3.8a-2.i686-pc-linux-gnu.tar.gz Squeak32-3.8a-2.ia64-linux-gnu.tar.gz Squeak32-3.8a-2.powerpc-apple-darwin7.8.0.tar.gz contain 32-bit VMs precompiled for Pentium, Itanium 2 and PowerPC. Squeak64-3.8a-2.i686-pc-linux-gnu.tar.gz Squeak64-3.8a-2.ia64-linux-gnu.tar.gz Squeak64-3.8a-2.powerpc-apple-darwin7.8.0.tar.gz contain 64-bit VMs precompiled for Pentium, Itanium 2 and PowerPC. Finally... ChangeSets is a directory containing everything that was used to create the above, relative to the base 3.8g-6548 image: VMMaker-tpr.14.mcz is Tim's latest VMMaker. VMM38b4-64bit-vm1-ikp.1.cs are Dan and my 64-bit changes, plus various changes Tim made to VMM since 3.7-5, all spiffed up and including a final couple of (nasty) new bugs (one original bug, and a few that crept in which merging VMM with the 64-bit changes) thoroughly fixed. This gets you enough to generate interp.c plus the three essential plugins: BitBlt, Ballon and File. VMM38b4-64bit-vm2-ikp.2.cs is new 64-bit support for plugins using the SmartSyntax thing. I made SocketPlugin (which uses it) 64-bit clean to make sure it was working reasonably. With both of the above loaded you should be able to regenerate the src32 and src64 trees included in the source tarball (and included in the unix part of the repository). System-Tracing.2.cs is the new SystemTracer from Anthony and Stephen, subclassed by Dan to make 64-bit images. This ONLY works on big-endian machines for now. (We know about this. Don't yell at us.) VMM38b4-64bit-image1-ikp.1.cs VMM38b4-64bit-image2-ikp.1.cs contain various changes to the class library that you probably need to have a healthy image running with 64-bit oops. (Tim, especially, will want to take the vm1 and vm2 changesets and file them into VMMaker _immediately_, before it becomes a 'moved target' [again]. No need to read these files: they are the changes Dan and I sent you, that you sent back last week with your recent VMM additions filed-in over the top, in which I then fixed some subtle 'merge' bugs that crept in during the process along with the WeakArray problem in primStringReplace posted to squeak-dev a couple of days ago. Trust me. Just hit 'install'. It's one huge chunk of goodness.) Enjoy! Ian (totally exhausted, and declaring the weekend already arrived ;-)

8 23

bug in svnx and rollback of /trunk/platforms/Cross/vm/sq.h needed (1172)
by John M McIntosh 14 Apr '05

14 Apr '05

Found a neat bug in the mac version of svnX 0.8. If you select one item in your list to commit, hit the commit button. Then say accidently hit the flat mode/smart mode check box, why then it cheerfully re-select *all* files that are pending, which then is processed by the pending/in progress commit, thus you commit a bogus version of trunk/platforms/Cross/vm/sq.h I'll note that the version of sq.h Ian posted in 1150 was missing some changes from last year. Now someone will need to repost the version from 1150 or rollback/copy? I'm not quite sure how to do this. I had sent a note earlier to Ian (10 minutes ago) asking him to check my copy of sq.h with his 64bit version. -- ======================================================================== === John M. McIntosh <johnmci(a)smalltalkconsulting.com> 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===

5 17

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev