Vm-dev August 2005

vm-dev@lists.squeakfoundation.org

8 participants
8 discussions

Primitive argument count
by Bert Freudenberg 03 Sep '05

03 Sep '05

I just noticed that many primitives do not check methodArgumentCount but just pop a fixed assumed number of arguments. Wouldn't it be better to always check this and fail if called with the wrong count? I guess stack corruption isn't as big a problem for us as it is for other systems, but still ... Another idea might be to "pop: self methodArgumentCount + 1" in the end so it doesn't matter how many arguments were supplied. - Bert -

2 1

[Fwd: Network performance on Mac and Linux]
by Andreas Raab 30 Aug '05

30 Aug '05

-------- Original Message -------- Subject: Network performance on Mac and Linux Date: Mon, 29 Aug 2005 15:15:41 -0400 From: David P. Reed <dpreed(a)reed.com> To: David A. Smith <davidasmith(a)bellsouth.net>, Andreas Raab <Andreas.Raab(a)SqueakLand.org> I'm sure you'd be as excited as I am to hear that I finally found the last piece of the network performance issue I've been chasing on the Mac and Linux. Over the weekend, I built a Mac VM with the same Socket improvements that I had done on Windows and Linux. It had the same flaws as the Linux code (not surprisingly, since the SocketPlugin source code is identical). The pieces of the puzzle that were Mac/Unix specific were: 1) the rate of polling for I/O events was once per 200 milliseconds, hard coded into checkForInterrupts part of the interpreter. (Windows Socket I/O didn't depend on this since it uses high-priority threads to wait for I/O). I changed it to once per 1 millisecond. This reduced the delay from well over 200 milliseconds to about 32 msec., averaged over a number of packets. 2) The unix driver didn't set TCP_NODELAY as a socket option by default, but Windows did. I changed the plugin to do this. This cut the measured end-to-end delay on packets from 32 msec. to < 1 msec. (Gaah - I had just assumed that this was done compatibly, and hadn't checked for it). So I will soon (tonight) have 3 VMs that have higher performance TCP network code. They should support the VP2PPN layer cross-platform. I'm wondering whether TCP_NODELAY should be default on or off for Squeak. The default should be the same on ALL platforms, in any case, and documented. For "real-time" stuff like Croquet and VoIP, TCP_NODELAY should be on (as it was for Windows). For batch file transfer, TCP_NODELAY should be off, because it helps by packing packets full of data.

3 2

flushExternalPrimitives usage?
by John M McIntosh 29 Aug '05

29 Aug '05

I was benchmarking the VM late last night and checking startup time and again noticed we invoke flushExternalPrimitives to scan over all object memory Interpreter>>flushExternalPrimitives "Flush the references to external functions from plugin primitives. This will force a reload of those primitives when accessed next. " On August 6, 2003 I had written a note "[ENH][VM] faster image startup" about this and had a change set to reset only the method contexts that we had noticed needed being reset in an earlier scan of memory when we were either adjusting t the oops offset or to address intel/powerpc issues. However the comments in initialCleanup imply we only should do this for earlier VM, not 3.8 or perhaps 3.6/3.7 I recall some discussion about changes to image saving so that new images wouldn't have this problem and then not require the scan. So did that fix get lost somewhere, or was an issue found and we should again look at how many times we scan oops memory at startup time? initialCleanup "Images written by VMs earlier than 3.6/3.7 will wrongly have the root bit set on the active context. Besides clearing the root bit, we treat this as a marker that these images also lack a cleanup of external primitives (which has been introduced at the same time when the root bit problem was fixed). In this case, we merely flush them from here." ((self longAt: activeContext) bitAnd: RootBit) = 0 ifTrue: [^nil]. "root bit is clean" "Clean root bit of activeContext" self longAt: activeContext put: ((self longAt: activeContext) bitAnd: AllButRootBit). "Clean external primitives" self flushExternalPrimitives. -- ======================================================================== === John M. McIntosh <johnmci(a)smalltalkconsulting.com> 1-800-477-2659 Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== ===

1 0

Re: internal or external, that is the question
by Andreas Raab 28 Aug '05

28 Aug '05

Brad Fuller wrote: > Andreas Raab wrote: > >> Different groups have different styles of work and we (the VM >> maintainers) have long adopted a style which does not assume that HEAD >> is always the latest stable version but rather provide explicit file >> releases. The main reasons being that there is no conceptual >> difference between checking out HEAD or using a stable file release >> and to allow us to move forward so that we know the order in which >> pending changes get integrated. This is in particular important since >> the main ports have maintainers with very different schedules and >> requiring everyone to get in sync before allowing to make a change to >> the trunk would probably deadlock the entire group. > > That's what branches are for, right? It depends on how you look at it. I see HEAD as the head of the main-line development. Branches are for trying out orthogonal stuff which will not necessarily go into main line. It's a different model from what you are describing where all development happens in branches and commits to the trunk happen rarely. I have seen both in practice and there is not much difference. >> For Windows, no. I still haven't gotten around to update the VM >> sources to 3.8 - in other words if we would follow your proposed model >> the trunk would be stuck at 3.7 which I would find an extremely >> unpleasant situation. > > Exactly, the trunk would be the stable version. I guess I don't see why > it's better to have the bleeding edge in the trunk. Ah, I think I see what you are missing: You assume that the different platforms are independent of each other, e.g., that the trunk of the SVN repository could have a stable 3.7 Windows *and* a stable 3.8 Mac set of source files. Unfortunately, that is not true. There are overlaps between the platforms (mostly in the Cross tree) and if there are conflicting changes (there typically are) you cannot have your cake and eat it, too. The only alternative is to either accept that some platforms may be broken at HEAD (the solution we adopted) or to delay any commits to the trunk until *all* platforms are updated. And the latter is probably impossible - maintaining a port is very specific stuff and I couldn't even dream of fixing an issue in the Mac or the Risc OS support code (the same probably goes the other way around). So unless you have a set of maintainers who can jump at the blink of an eye 24/7 you better deal with the fact that one of them might just be off for vacation. > If it sounds messy because of all the different platforms, maybe the > released should be located one more dir below for each platform. Or, > separate trunks for each platform. At least the source would be in one > place. This might be doable but it sounds like quite a bit of work and I suppose you aren't volunteering... >> Also, the set of maintainers has been working together in some form or >> other for almost ten years now and you should at least consider the >> possibility that there are reasons for why things are the way they are. > > Oh, I do. I just question it, that's all. Yes, I know. We see these questions every other year usually in the form of some demands that "you must organize your work like this" and of course, with absolutely no intent to help and actually improve things. I've gotten edgy about this over the years to the point that I've considered twice to just drop it and leave you guys doing whatever you think you can do better (who knows, maybe you can). >> They work for us. And If you think we're all morons who don't know >> Jack you are of course free to set up an alternative repository and >> show the superiority of your ways. We'll discuss the results in 2015 ;-) > > I understand and respect that they've been working together for a long > time. I'm indebted to their invaluable contributions. I just want to be > able to obtain those contributions and build it in a reasonable way. > And, I didn't mean to imply that the maintainers are morons. Sorry, that was meant as a joke about short-term vs. lasting impact. > Poking around, I see that there are indeed branches and tags, but they > seem a bit backward to me. There is a 3.7 branch for unix at: > http://squeak.hpl.hp.com/svn/squeak/branches/unix-3.7-7/. Would this be > the released 3.7 for unix? Good question. As you may or may not know, dealing with branches in CVS is a pain and so we essentially didn't use it. This might just be legacy from the CVS days or it might be something different. Only VM-dev could tell. > There are several places in the swiki that states to go to the trunk for > the sources. In one place, it does say "the latest sources". So, you got > me there. Ok, how about we clean up the swiki, so that it indicates > where to get the stable or released sources... to save headaches for the > next bloke. That's the problem with the Swiki - some people just edit something and there is absolutely no authority to what they say. The only place that counts for Unix is http://squeak.hpl.hp.com/unix. > Poking around some more, I see that there is a page that indicates where > to get the win sources. To me, it seems a bit backwards to have people > go to different places for sources. Isn't it a goal for squeak to > maintain consistency among all platforms? It's not possible completely, > of course, but that's the goal. So, it seems a waste (to me, no offense > intended) to not use SVN to it's full potential. And the only place that's authorative for windows is http://minnow.cc.gatech.edu/squeak/3272 (where it points back to squeak.hpl.hp.com). Cheers, - Andreas

1 0

Working copy 'unix/plugins' is missing or not locked
by Brad Fuller 23 Aug '05

23 Aug '05

How come I get this message with I try to 'svn update' This is a fresh svn co, so I don't know why this happens. If you could explain, I'd appreciate it. (didn't see anything in the archives) thanks, brad -- Brad Fuller (408) 799-6124 ** Sonaural Audio Studios ** (408) 799-6123 West San Jose (408) 799-6124 Cambrian ________________________________ Hear us online: www.Sonaural.com

1 0

[Fwd: Socket I/O Problems with UNIX Squeak 3.8a1 VM]
by Andreas Raab 20 Aug '05

20 Aug '05

-------- Original Message -------- Subject: Socket I/O Problems with UNIX Squeak 3.8a1 VM Date: Sat, 20 Aug 2005 11:33:09 -0400 (EDT) From: David Shaffer <cdshaffer(a)acm.org> Reply-To: The general-purpose Squeak developers list <squeak-dev(a)lists.squeakfoundation.org> Newsgroups: gmane.comp.lang.smalltalk.squeak.general I have been trying to figure out why Squeak performs so badly as a webserver. With some help from John McIntosh (thanks John!) I was able to track down a problem in the Unix VM's socket I/O support. The problem is the standard "missed signal" when using select(). When the Squeak VM enters aioPoll() it uses a select() to check for activity on various sockets (and other file descriptors -- but I'll stick to TCP sockets here). For server sockets (passive TCP sockets) who have received new connections it proceeds to call accept() once and then signal the appropriate semaphore. The problem is that more than one new connection may have arrived during (and before) that first select and standard POSIX practices say you have to call accept() until it returns -1 with errno=EWOULDBLOCK. I am quite certain of this...because should multiple connections arrive during the first select(), a subsequent select() will not detect them. Where I'm a little less certain, not having any books with me, is if you need to do similar things for read() and write()...I believe that you do. In any case, at the very least, this causes the SqueakVM to miss connections when they occur close together. I won't go into details but the problem cascades causing UNIX Squeak-based servers to behave very badly when receiving large numbers of closely spaced connections. I don't have a production-ready solution but right now I have a working image which uses an array like so: typedef struct privateSocketStruct { int s; /* Unix socket */ int connSema; /* connection io notification semaphore */ int readSema; /* read io notification semaphore */ int writeSema; /* write io notification semaphore */ int sockState; /* connection + data state */ int sockError; /* errno after socket error */ struct sockaddr_in peer; /* default send/recv address for UDP */ int multiListen; /* whether to listen for multiple connections */ int acceptedSock[1024]; /* a connection that has been accepted */ int acceptedSockCount; } privateSocketStruct; Anyway, I patched the code to loop over accept() and fill the acceptedSock array and repeatedly signal the semaphore. Of course I also added the logic on the other end of the accept to deal with decrementing this count. Clearly this is a stop-gap solution. I don't think we need to allocate 1024 ints for every socket. Anyway after a lot of pounding with httperf the net result seems to be a much more reliable Squeak/UNIX web server. I don't believe, though, that it performs much faster than it did before...just that it doesn't mysteriously drop connections. So, I'm only part way to my ultimate goal which is to improve performance. I thought this would be a good time to bring this up and get feedback (especially from Ian) about how you would like me to shape the actual fix to this problem. Add a single void * member to privateSocketStruct which, for server sockets points, to a "struct privateServerSocket" which holds the count and maybe a linked list of socket file descriptors. When I return to the U.S. I'll look into the details of the interactions of select() with non-blocking read's and write's. I suspect that this code might need a good looking-at too. After that I'd like to work with Ian to put together a patch against the 3.8g VM that Ian has provided since many of us use this VM in production Seaside images. Also, if someone wants to forward this on to the VM list, I'd appreciate it. I'm not a list member and have a very poor connection at the moment. BTW folks...ESUG was excellent! Thanks to everyone who put it together. David

1 0

Building on VM osx 10.4
by Alan Sibley 13 Aug '05

13 Aug '05

Hi folks, I'm a rookie VM builder -- I hope I've made an obvious and easy to spot error in my configuration. I am seeing a substantial performance issues with my build :( When it initially comes up... it pegs the CPU for 4 or 5 seconds... and clicking on windows is like watching paint dry. 1000000 timesRepeat: [] takes 3 times as long as 3.8.6Beta6. a) I'm building on OSX 10.4 Tiger using the developer tools b) I believe I have the latest source from the subversion server. c) I have two patches from Tim Rowledge concerning VMMaker d) I have all plugins built internally except (FFLPlugin, FileCopyPlugin, LocalePlugin, Mpeg3Plugin, TestOSAPlugin) which are disabled. e) I'm using the Squeak VM OPT target f) I've got optimization set at O3 g) I checked and don't have profiling (-pg) turned on ;) When I view the current Squeak 3.8.6Beta6 and my build under "Shark" I see fairly different picture (I can describe more in needed). Let me know what kinds of information to add to help track this down. Thanks in advance, Alan

2 1

Re: memory and VM issues (interp.c part 1 of 2 attached)
by David T. Lewis 05 Aug '05

05 Aug '05

On Fri, Jul 29, 2005 at 01:03:31PM -0700, Tim Rowledge wrote: > Could you try a 3.8-6665 image, SVN revision 1100, VMMaker38b4 combo? > That is the last pre-64bit-fiddling build and should work ok. One thing > we're a bit concerned with is the changes to GC/weak objects that got > put in later. OK, I spend some time hunting this down the hard way. I can't yet say exactly what's wrong, but I think I've got it pretty well localized. Executive summary: The problem was introduced in VMM code as opposed to platforms code. It was introduced in the change from VMMaker-tpr.21 to VMMaker-tpr.22. The VMMaker-tpr.22 change set contains "Merge in johnmci & ar's GC instrumentation and weak pointer changes". I don't know anything further at this point, but this should at least narrow the search. Here's my debugging log. A bit terse, but it will give you an idea of some of the combinations that I tried. - Dave (CC to vm-dev list: This pertains to building a VM on my 32 bit Intel Linux box from recent SVN and VMM code bases, following up on my previous message "Problem building from latest SVN & VMM".) ---------------------------- Key: OK -> does not exhibit the memory access bug (crash VM and/or debuggers in minimal image) NFG -> has the bug ? -> not sure, gave up due to other problems Test log (note: I closed in the the problem using SVN 1200 code base, varying VMM versions): Platforms version VMMaker version Result ----------------- --------------- ------ ./dist3 ./dist3 OK squeak-svn-source-SVN1100 VMMaker-tpr.15 ? needs sqMemoryAccess.h squeak-svn-source-SVN1140 VMMaker-tpr.15 ? mismatch, lots of missing stuff squeak-svn-source-SVN1140 VMMaker-tpr.20 ? parse error gnu-interp.c squeak-svn-source-SVN1145 VMMaker-tpr.15 ? mismatch, lots of missing stuff squeak-svn-source-SVN1145 VMMaker-tpr.20 ? parse error gnu-interp.c squeak-svn-source-SVN1150 VMMaker-tpr.15 ? needs interp.h, parse error gnu-interp.c squeak-svn-source-SVN1150 VMMaker-tpr.14 ? needs interp.h squeak-svn-source-SVN1150 VMMaker-tpr.20 ? needs interp.h squeak-svn-source-SVN1150 VMMaker-tpr.25 ? needs interp.h squeak-svn-source-SVN1150 VMMaker-tpr.34 ? needs interp.h squeak-svn-source-SVN1150 VMMaker-tpr.31 ? needs interp.h squeak-svn-source-SVN1158 VMMaker-tpr.15 OK WRT bug, but has FilePlugin problem squeak-svn-source-SVN1158 VMMaker-tpr.16 OK WRT bug, but has FilePlugin problem squeak-svn-source-SVN1158 VMMaker-tpr.17 ? parse error in #sufficientSpaceToInstantiate:indexableSize: squeak-svn-source-SVN1158 VMMaker-tpr.20 ? parse error in #sufficientSpaceToInstantiate:indexableSize: squeak-svn-source-SVN1158 VMMaker-tpr.31 NFG squeak-svn-source-SVN1158 VMMaker-tpr.33 ? undefined reference to `fetchWordofObject' squeak-svn-source-SVN1165 VMMaker-tpr.15 OK WRT bug, but has FilePlugin problem squeak-svn-source-SVN1165 VMMaker-tpr.16 OK WRT bug, but has FilePlugin problem squeak-svn-source-SVN1165 VMMaker-tpr.31 NFG squeak-svn-source-SVN1165 VMMaker-tpr.33 NFG squeak-svn-source-SVN1165 VMMaker-tpr.34 ? undefined reference to `fetchWordofObject' squeak-svn-source-SVN1180 VMMaker-tpr.35 ? undefined reference to `fetchWordofObject' squeak-svn-source-SVN1180 VMMaker-tpr.37 ? undefined reference to `fetchWordofObject' squeak-svn-source-SVN1200 VMMaker-tpr.16 OK WRT bug, but has FilePlugin problem squeak-svn-source-SVN1200 VMMaker-tpr.20 OK, but parse error in #sufficientSpaceToInstantiate:indexableSize: squeak-svn-source-SVN1200 VMMaker-tpr.21 ==> OK, but has FilePlugin problem plus parse error in #sufficientSpaceToInstantiate:indexableSize: squeak-svn-source-SVN1200 VMMaker-tpr.22 ==> NFG, plus has FilePlugin problem plus parse error in #sufficientSpaceToInstantiate:indexableSize: squeak-svn-source-SVN1200 VMMaker-tpr.23 NFG, plus has FilePlugin problem squeak-svn-source-SVN1200 VMMaker-tpr.25 ? parse error in #sufficientSpaceToInstantiate:indexableSize: squeak-svn-source-SVN1200 VMMaker-tpr.26 NFG squeak-svn-source-SVN1200 VMMaker-tpr.28 NFG squeak-svn-source-SVN1200 VMMaker-tpr.31 NFG squeak-svn-source-SVN1200 VMMaker-tpr.37 NFG squeak-svn-source-SVN1233 VMMaker-tpr.16 ? undefined reference to `fetchLong32ofObject' squeak-svn-source-SVN1233 VMMaker-tpr.37 NFG

4 12

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Vm-dev August 2005