I've now made remote up-and downloading of module work properly.
However, Squeak's underlying FTP code is so painfully inefficient that I recommend that you avoid using it, and instead keep the accessRemoteRepostories preference off, and upload from the local repository cache using some other ftp program.
An upload that takes say 5-15 seconds with another FTP program takes some *5 minutes* with Squeak according to the file time stamps.
I've made downloads avoid this by using HTTP access.
The problem is that Squeak's ftp code logs in and logs out between *every* CWD, load, store, etc. -- every single server action.
It is this same deficiency that makes project loading so slow: 0.25 seconds to load an image segment from disk *really quickly*, but eons spent on redundantly logging in and out of the ftp server.
It wouldn't be too hard to change the code to allow multiple commands within a single session: Put a socket instvar in ServerDirectory, and make the code either login and logout if that socket instvar is nil, or if not, just use it to send the command without logging in and out. Then wrap sessions in a login and ensured logout.
The other option is supporting HTTP PUT, Cees would be willing to make the server support this, but this may not be trivial. I am not planning to spend my time on this.
Henrik
Henrik Gedenryd wrote:
I've now made remote up-and downloading of module work properly.
However, Squeak's underlying FTP code is so painfully inefficient that I recommend that you avoid using it, and instead keep the accessRemoteRepostories preference off, and upload from the local repository cache using some other ftp program.
An upload that takes say 5-15 seconds with another FTP program takes some *5 minutes* with Squeak according to the file time stamps.
I've made downloads avoid this by using HTTP access.
The problem is that Squeak's ftp code logs in and logs out between *every* CWD, load, store, etc. -- every single server action.
It is this same deficiency that makes project loading so slow: 0.25 seconds to load an image segment from disk *really quickly*, but eons spent on redundantly logging in and out of the ftp server.
It wouldn't be too hard to change the code to allow multiple commands within a single session: Put a socket instvar in ServerDirectory, and make the code either login and logout if that socket instvar is nil, or if not, just use it to send the command without logging in and out. Then wrap sessions in a login and ensured logout.
ServerDirectory already has a instance variable "socket". I briefly looked at the code ... and I have to admit: I don't quite understand it. In ServerDirectory>>openNoDataFTP the storing of the socket is commented out. The last lines are:
"socket _ so". "If user wants to keep connnection open, he must store socket" ^so
After enabling the line, two putFile:named: messages share the same socket.
Highly confused :-)
Cheers,
felix
The other option is supporting HTTP PUT, Cees would be willing to make the server support this, but this may not be trivial. I am not planning to spend my time on this.
Henrik
However, Squeak's underlying FTP code is so painfully inefficient that I recommend that you avoid using it, and instead keep the accessRemoteRepostories preference off, and upload from the local repository cache using some other ftp program.
An upload that takes say 5-15 seconds with another FTP program takes some *5 minutes* with Squeak according to the file time stamps.
Yes. This is a constant problem to me when issuing updates. If the server is at all busy, I get timed out and have to start all over again.
I've made downloads avoid this by using HTTP access.
The problem is that Squeak's ftp code logs in and logs out between *every* CWD, load, store, etc. -- every single server action.
It is this same deficiency that makes project loading so slow: 0.25 seconds to load an image segment from disk *really quickly*, but eons spent on redundantly logging in and out of the ftp server.
It wouldn't be too hard to change the code to allow multiple commands within a single session: Put a socket instvar in ServerDirectory, and make the code either login and logout if that socket instvar is nil, or if not, just use it to send the command without logging in and out. Then wrap sessions in a login and ensured logout.
It would be great if someone who understands this would fix it. It would help us all around.
ServerDirectory already has a instance variable "socket". I briefly looked at the code ... and I have to admit: I don't quite understand it. In ServerDirectory>>openNoDataFTP the storing of the socket is commented out. The last lines are:
"socket _ so". "If user wants to keep connnection open, he must store socket" ^so
After enabling the line, two putFile:named: messages share the same socket.
Highly confused :-)
I'll check with Ted. he understood it all at one point.
- Dan
ServerDirectory already has a instance variable "socket". I briefly looked at the code ... and I have to admit: I don't quite understand it. In ServerDirectory>>openNoDataFTP the storing of the socket is commented out. The last lines are:
Umm, now that I looked *much* closer, there indeed some kind of support for it.
But there isn't any protocol for it, no one is using it, and it's not obvious from the code... There is *plenty* of room for refactoring code there. I think the same "QUIT" code alone is in some 10 different places.
I got it working. Woohoo! V e r y much faster! Now if only our proxy wouldn't return the wrong replies after an update...
I'll include some kind of rudimentary protocol for this with the imminent change set. It will go something like this:
openSessions _ repositories collect: [:rep | rep directory wakeUp; yourself]. [ <do your thing> ] ensure: [openSessions do: [:serverDir | serverDir sleep]].
I'm actually using the same session for multiple ServerDirectory instances on the same server.
Thanks to Felix for this discovery! I had been looking at that code many times without discovering this little gem.
Henrik
I got it working. Woohoo! V e r y much faster! Now if only our proxy wouldn't return the wrong replies after an update... ... I'm actually using the same session for multiple ServerDirectory instances on the same server.
Thanks to Felix for this discovery! I had been looking at that code many times without discovering this little gem.
Yayy. Thanks to both of you. It will be great to have FTP running right!
- Dan
Henrik (or anyone else...)
Did this fix ever go out? I can't find a ChangeSet.
I'd like to get it, use it, and put it out as an update. Slow FTP has been a problem for update management for ages. Basically, lots of stuff fails because the server gets bored, and you have to start over again.
- Dan ------------------
ServerDirectory already has a instance variable "socket". I briefly looked at the code ... and I have to admit: I don't quite understand it. In ServerDirectory>>openNoDataFTP the storing of the socket is commented out. The last lines are:
Umm, now that I looked *much* closer, there indeed some kind of support for it.
But there isn't any protocol for it, no one is using it, and it's not obvious from the code... There is *plenty* of room for refactoring code there. I think the same "QUIT" code alone is in some 10 different places.
I got it working. Woohoo! V e r y much faster! Now if only our proxy wouldn't return the wrong replies after an update...
I'll include some kind of rudimentary protocol for this with the imminent change set. It will go something like this:
openSessions _ repositories collect: [:rep | rep directory wakeUp; yourself]. [
<do your thing> ] ensure: [openSessions do: [:serverDir | serverDir sleep]].
I'm actually using the same session for multiple ServerDirectory instances on the same server.
Thanks to Felix for this discovery! I had been looking at that code many times without discovering this little gem.
Henrik
Dan Ingalls wrote:
I'd like to get it, use it, and put it out as an update. Slow FTP has been a problem for update management for ages. Basically, lots of stuff fails because the server gets bored, and you have to start over again.
It's out as 4776.
The clearest example of its use is probably in RemoteModuleStorageTests
verifyRepositoryContents or ModuleInstaller>> uploadStandaloneRepositories.
What you do is to create a ServerDirectory instance, send #wakeup, hold on to this instance, and all ftp commands invoked from this instance will share the same session. You can even have several instances, pointing to diffferent directories on the server, share the same session. Then send #sleep to close the session.
Since I use Repository objects to wrap the ServerDirectory instances, and to handle sharing the sessions between directories in a nice way, the uploading code has an extra layer of indirection making it a little harder to see what's going on (downloading uses http currently).
Henrik
squeak-dev@lists.squeakfoundation.org
-
Dan Ingalls -
Felix Franz -
Henrik Gedenryd