So everytime I pull a server to do any installation work, it goes slightly wonky. Well, that's not really true, but the volume of cables supported tends to slop up things even with good cable-management arms.
Today I was supposed to pull a pair of major servers to plug in shiny new FiberChannel Host Bus Adapters. One got done. It's still rebuilding itself after about 4 hours.
One of the internal drives failed, and munched on the boot partition. After a bit of questionable advice from a support technician out on another call, we tried to force the drive back online. That just dragged the problem out, and resulted in having to reinstall Windows 2003 server.
Here come's the "I hate Microsoft part"
The Win'03 server install is the clunkiest, most unforgiving piece of software I've seen in ages. Where older versions gave you a fighting chance to install vendor-supplied HD drivers, on this install you've got 5 seconds to press <F6> before it just goes on. My servers require vendor-supplied drivers. If I miss that, I've got to reboot the whole setup process.
The crash-generated error message and repair instructions are bad. If you follow the instructions it takes you to the "Windows Recovery Console." I actually know how to use the recovery console, but it requires the local administrator password, and we always log in with personal IDs, so that was worthless.
There's an "automated server recovery" process but it's far clunkier than the previous versions of windows; rather than creating one ASR disk with vital configuration information, it generates dozens of floppies. I hate floppies.
It is possible to do an overlay install that will actually repair the hosed up OS, but only at a later point, not by selecting "Repair" at the beginning (like the error message says). Surprisingly, it works. Mostly. It works best if the boot partition isn't continuing to re-corrupt itself, so we pulled the suspect drive and limped along in "compromised" mode for one final reinstall that got us logged in.
About this time I got hold of a vendor telephone service rep who told me that the suggested "force online" was a bad idea. It probably wasn't a physical error, it was probably some data corruption that got out of hand. I popped the drive back in and set the array to rebuilding the final drive. It's still going.
At least I can be confident in my backup software. Once the disks are happy again, I can just restore the configuration and apps back to where they were this morning.