Cross your fingers, with luck we should be good now.
For over a month we've had trouble with the community server locking up and requiring a reboot. On Friday Hetzner did some testing on the server and, as I understand, traded out the RAM at least once (I believe this has been done before as well). From then on the server was up in some sense, but not working properly.
Unfortunately we don't have weekend support so there was a delay. Additionally there was some confusion that resulted in further delays. On Tuesday Marcus Denker, our contact with Hetzner, convinced them to hook up a 'virtual console' to the system which allowed us to see the console output. Cees de Groot and Goran Krampe were able to run some tests that indicated that the RAM was bad. This convinced Hetzner to change the RAM again. After this the server booted up and ran fine for a few hours.
Then it went down again as it has for weeks now. After the latest trouble Hetzner was convinced to do more and took the hard drive from the server and put it in another server with identical hardware. And we are up again.
Hopefully this takes care of us for a while. But I think there are some things we can learn from this event and I believe some changes need to be made to how we manage the server. I'm not sure what to do right now. But I'm thinking about it and sometime soon I would like to start a discussion of the future of the server.
Ken
@Ken, Goran, Cees, Marcus, ... Thanks to all of you for taking care of and fixing the server.
It had been a rough weekend without the mailing lists ;-)
Cheers, Lukas
On Thu, Oct 23, 2008 at 08:38:59AM +0200, Lukas Renggli wrote:
@Ken, Goran, Cees, Marcus, ... Thanks to all of you for taking care of and fixing the server.
It had been a rough weekend without the mailing lists ;-)
Indeed, thank you very much for your work. Not just for fixing things, but also for all of the time that the servers hum along uneventfully and we all take it for granted and forget to say "thanks".
Dave
On Thu, Oct 23, 2008 at 8:07 AM, Ken Causey ken@kencausey.com wrote:
Cross your fingers, with luck we should be good now.
Hopefully this takes care of us for a while. But I think there are some things we can learn from this event and I believe some changes need to be made to how we manage the server. I'm not sure what to do right now. But I'm thinking about it and sometime soon I would like to start a discussion of the future of the server.
Many thanks to all that got things running again!
May I recommend that we host the mailing list on another machine than our main server? A cheap VPS server would manage this fine. Most services we can afford to lose for a while, but it was a bit unnerving not being able to see what was happening.
Gulik.
Michael van der Gulik schrieb:
On Thu, Oct 23, 2008 at 8:07 AM, Ken Causey ken@kencausey.com wrote:
Cross your fingers, with luck we should be good now.
Hopefully this takes care of us for a while. But I think there are some things we can learn from this event and I believe some changes need to be made to how we manage the server. I'm not sure what to do right now. But I'm thinking about it and sometime soon I would like to start a discussion of the future of the server.
Many thanks to all that got things running again!
+1
May I recommend that we host the mailing list on another machine than our main server? A cheap VPS server would manage this fine. Most services we can afford to lose for a while, but it was a bit unnerving not being able to see what was happening.
Well it was possible to get informed about the "rescue mission" on irc.
So maybe a message like: in case of the mailimg list beeing down, try to get uptodate information on irc.freenode.net: #squeak
can be added to the welcome message of the mailinglist.
Rudolph --
Gulik.
Hi!
rudolf-Rednose wrote:
Michael van der Gulik schrieb:
May I recommend that we host the mailing list on another machine than our main server? A cheap VPS server would manage this fine. Most services we can afford to lose for a while, but it was a bit unnerving not being able to see what was happening.
Well it was possible to get informed about the "rescue mission" on irc.
One thing is that we are most probably moving DNS to zoneedit or similar so that we can easily point www.squeak.org to another ip during an event like this. That should easily fix the "we don't know what is going on"-problem.
Now we couldn't easily do that unless involving Dan etc.
regards, Göran
On Mon, 2008-10-27 at 11:15 +0100, Göran Krampe wrote:
Hi!
rudolf-Rednose wrote:
Michael van der Gulik schrieb:
May I recommend that we host the mailing list on another machine than our main server? A cheap VPS server would manage this fine. Most services we can afford to lose for a while, but it was a bit unnerving not being able to see what was happening.
Well it was possible to get informed about the "rescue mission" on irc.
One thing is that we are most probably moving DNS to zoneedit or similar so that we can easily point www.squeak.org to another ip during an event like this. That should easily fix the "we don't know what is going on"-problem.
Just a little correction. We are adding zoneedit.com as additional secondary DNS servers, not moving it. The community server will still be the primary DNS server. But if it is down for an extended time the zoneedit configuration can be manually modified to point to a different server temporarily.
Now we couldn't easily do that unless involving Dan etc.
I've emailed Dan. We just have to be patient.
Ken
regards, Göran
squeak-dev@lists.squeakfoundation.org
-
David T. Lewis -
Göran Krampe -
Ken Causey -
Lukas Renggli -
Michael van der Gulik -
rudolf-Rednose