The Kos Proto-server Has Died

Long live the server.

Plans had been in place to move mail and search to a new server for a while, but such a major move requires planning and thought. On Wednesday, all those plans went out the window. The old server froze up again for the second time in two days, so I went and hard rebooted it since nothing else would bring it back in that situation. I waited and waited for it to come back up, but it never did. Growing a little concerned, I connected to its serial console and saw that it was waiting at a single user prompt - /var was corrupted. I ran fsck to fix the errors as suggested, but it was bad. Very bad. Scores of hard read errors scrolled by while I watched helpessly, the superblock was completely hosed, and the "lost+found" directory was gone. I was also given the impression by the folks at the data center that that server was making a loud racket while I was trying to resuscitate it. I ran it a second time, but all the same errors went passing by while I sat and waited for it to get fixed. It wanted me to fix the filesystem again, but I could tell it was never going to get properly fixed, so I tried rebooting. Miraculously, it came back. Why it managed to come back I have no idea, but it gave jotter and I the time we needed to get the important stuff off of it before it failed for good. It even kept running until I pulled the plug on it once the switch was complete, still working away. - DailyKos

For the geeks who read, you know the emotional attachment that accrues to some equipment. My company had a server - "gizmo" - which was shut down a couple years ago. Gizmo not only served my current company, it came with us from a PREVIOUS company.

Serve on Kos server, your time was well spent.