Moving the discussion to squeak-dev. Follow-up related to the Squeak image should be on squeak-dev, discussion of the primitives can stay on vm-dev.

Dave

On Thu, Sep 20, 2012 at 02:11:35PM -0400, David T. Lewis wrote:

We have an upcoming image release, so I think that we should consider de-activating the new network support for that release. I think that can be done by just setting UseOldNetwork to true. Right now it is automatically set at image startup, depending on whether the IPv6 primitives are available in the image. We could turn it back on in trunk after the release, and deal with the issues then.

It would not be good to release with network issues that might affect the entire Windows user base.

Dave

On Thu, Sep 20, 2012 at 07:50:42PM +0200, Levente Uzonyi wrote:

...

On Wed, 19 Sep 2012, David T. Lewis wrote:

...

There seem to be some issues with the new networking code (in Squeak, for IPv6 support) when running on a Windows VM that has the IPv6 primitives in the SocketPlugin.

I booted up Windows and tried Levente's deadlock test:

...

...

I could reproduce a deadlock-like state by evaluating:

NetNameResolver addressesForName: 'amazon.com'

I tried this both with Ian's beta interpreter VM, and one of Eliot's recent Cog VMs. Both VMs have the IPv6 primitives now, and both of them show similar issues. I did not see actual deadlocks, but what I did see was extremely long primitive calls that make the image feel like it is deadlocked. The #primitiveResolverGetNameInfo call is a source of problems, and there may be others.

It looks to me like some of the new primitives are invoking some very slow system functions on Windows, and the Squeak network code updates cause these primitives to be called if they are available in the VM, so the newer VMs are having problems. I have not seen these issues on Linux, so it may reflect differences in the networking support for different operating systems.

Levente, is this consistent with what you were seeing?

It is, though I didn't check which primitive takes too long. I checked the implementation of #primitiveResolverGetNameInfo now on win32 and it seems to be okay, pretty much the same as on unix/linux. I see no reason why it would take so long to respond.

In the meantime I found another thing I don't like about the new primitives. The timeout for the namelookup is ignored, so now we have to wait till the primitive returns.

Actually I don't like the way how the name lookup is implemented. I think it might worth moving the DNS lookup to Squeak. The only thing the VM needs to provide then is a primitive which returns the IP addresses of the nameservers to be used (though the system can work without that too). Here are the pros and cons I came up with so far: Pros:

• the code is in Smalltalk, so it's platform independent
• concurrent name lookups become possible
• no more long waits on the VM side

Cons:

• extra complexity, since a DNS client has to be implemented
• the OS's DNS cache can't be used

Levente

...

Dave

On Mon, Sep 17, 2012 at 03:37:45PM -0400, David T. Lewis wrote:

...

Thanks Levente, this is very helpful.

It sounds like these are problems in the new network code on the image side (and I'm responsible for causing that). The updated VM is probably different only in that it provides the IPv6 primitives, which in turn expose the bugs on the image side. So I expect that the problems you describe should also happen with a unix interpreter VM (I'll check and find out as soon as I can).

I note for the record that Andreas is fully entitled to say "I told you so!" at this point ;)

To the extent that these are Squeak image problems, evaluating "NetNameResolver useOldNetwork: true" should make the symptoms go away.

Other comments in line below.

On Mon, Sep 17, 2012 at 07:01:47PM +0200, Levente Uzonyi wrote:

...

On Mon, 17 Sep 2012, David T. Lewis wrote:

...

Levente,

Can you say anything more about what weakness you found in the network code?

All issues I found are related to name lookup. To do a name lookup the new code requires multiple primitive calls (see SocketAddressInformation >> #forHost:service:flags:addressFamily:socketType:protocol:). The plugin uses static variables to store the result of the name lookup (hostNameInfo, servNameInfo and nameInfoValid). This means that only one name can be looked up at a time.

I noticed that, and attempted to provide some protection for it with a semaphore (ResolverMutex) in class NetNameResolver. Apparently I did not do a very good job of it though.

...

The image side code doesn't prevent simultaneous access to these static variables, so the can get into an unexpected state (see SocketAddress >> #hostName).

Another issue is that the plugin doesn't allocate objects (strings), so 2 primitive calls have to be done to fetch a string (see SocketAddress >> #hostName again). One requests the size of the string, the other copies the data to a string it receives as argument.

I could reproduce a deadlock-like state by evaluating:

NetNameResolver addressesForName: 'amazon.com'

It's sometimes possible to interrupt the process to get a debugger, but since the primitives are called by the debugger too (see SocketAddress >> #printOn:), the image will hang if you try to use it.

That seems likely to be a problem related to the semaphore in NetNameResolver.

If this turns out to be a problem with the network support in Squeak trunk, we can take the discussion back to squeak-dev for resolution.

Dave