Challenges of Upgrading Linux

I’ve been running a web and email server out of my house for three years. When I did the initial install I had experience with configuring Windows as a web and email server but Exchange was unstable and I hated having to reboot Windows servers to apply security patches. Sasser and Blaster had just made their rounds on the world and I, like thousands of System Administrators, was affected by the worms. As a result of my experiences I chose to increase my Linux experience and the best way to do that was install and set up Linux as a web and email server. Three years ago I chose to do that with Fedora, Apache2, and Sendmail.  The initial configuration wasn’t simple. I had to learn how to compile source code. I spent hours digging through forums and how-to articles to get my configurations correct. But after a couple of weeks working in my free time I had a stable, secure, Linux system with web access to my email. In the course of three years I never had to reboot the server and when I found security holes I did a quick up2date and it applied the upgrade and installed a patch. I was very happy with my new Fedora server and it ran perfectly until last Tuesday…

Redhat moved from Up2date to yum and as far as I could find Fedora Core 2 stopped being supported late last year. As a result I couldn’t use up2date to patch Apache2 when security vulnerabilities were found for Apache 2.0.51, instead I would have to download source and compile it then reconfigure it to work with Sendmail and Squirrelmail. When faced with this challenge I decided it would probably be best just to update to a new version of Fedora that supported Yum and Redhat still offered updates for. The install from CD seemed to go smoothly. I inserted disk one of five, rebooted, clicked next a few times, and then picked the radial button for upgrade. The Fedora Core installer (Anaconda) found my FC2 partition and I chose to upgrade hdb1. I clicked next a couple more times and the installer searched for my software then started the install. After inserting the five disks and waiting around two hours the install finished and rebooted. Upon rebooting the system started the kernel promptly returned the error: “mkrootdev: expected fs options; mount: missing mount point; kernel panic.”  This message means it was expecting a specific file system or trying to mount that file system from a specific device but couldn’t find it. Basically, the system knew to look for the “/” directory but couldn’t find “/.” Without that directory the kernel couldn’t find the files it needs to boot and instead panicked and crashed.

I (like any decent System Administrator finding himself in an unfamiliar area) turned to Google in my time of need. I found that dozens of people have had the same problem with a FC 5 upgrade from FC 2. This is basically a known issue and the inevitable answer on each forum I found is “format and install FC 5 clean.” This confounded me. This sounded like the answer a bad MCSE would give me. I called Compaq one time and they told me “Format and Quick Restore.” Nobody asking my advice has every bought a Compaq since then. But, Linux guys are supposed to be the geeks who find a work around when there isn’t a known solution. Linux is an idea and a community where people come together and help each other find answers when a corporation hasn’t defined what you should do. I turned to my friends on IRC, they know how to work together and find answers… But, though IRC has helped me numerous times in the past, each suggestion I received led to a dead end.

I was digging through fstab and mtab, editing my grub menu and boot options, but each step led only to different errors. Nothing I did helped my kernel find “/.” The final answer I got from one of my colleagues at NetStandard Inc. was to upgrade from Fedora to Debian (which I like better anyways and I will eventually do) but I’m a true geek at heart and I hadn’t given up yet.

I knew Fedora 5 wasn’t the newest version of Fedora when I upgraded, I just didn’t want to waste 5 more disks and I had 5 on another machine at home. Fecora Core 6 was available off the mirrors on Redhat’s web site. I burned disk 1 and started through the process outlined in paragraph 2 only this time the upgrade program found Fedora Core 5 on hdb1 and received an error that the file system was in an unbootable state. The Fedora Core 6 disk told me to boot to rescue mode and repair the file system. I booted to rescue mode and ran fsck on device hdb1, hdb2, hdba1 and hdba2. Then booted back to Fedora Core 6 and installed from the five disks. The install went fine and I was able to boot past the error I had been getting, for a second I thought I was in the clear, and then an error that SeLinux could not permit a process (I don’t remember which) to run followed, inevitably, by my favorite error “Kernel Panic.” I was up a creek again but this time I had a paddle. I hadn’t been running SeLinux on Fedora Core 2 so I thought if I disabled it I’d be able to at least boot to a different error. I booted to disk 1 and exited to System Rescue mode then found the file that starts SeLinux and edited the file to disable SeLinux. Upon reboot Fedora Core 6 loaded up and X Windows even started. My config files for Apache, Sendmail, IMAP, and Squirrelmail were overwritten during one of my upgrades but I was in the clear, I still had my data! After a few quick Google searches all my services were running again, and with the newest, most secure, version.

This experience taught me a lot. I’ve learned more about how Linux works and how different files (fstab and mtab) affect the OS in different ways. I’m very comfortable with booting to various Linux recovery disks (over the last week I’ve used Helix, Fedora Core 5 and 6, Knoppix, and Knoppix STD) as resources. I also gained experience with some of the tools (fdisk, fsck) I had to use to get the OS to boot again. But, more than ever I’m comfortable with the ability of Linux to preserve data regardless of changes to the OS. At no point during this experience did I feel like my data was unrecoverable or that I’d have to format and reinstall the OS.

A couple of follow up notes:

1.  With Linux always put your /home directory on a second partition. Even if I had formatted and installed Debian I would have preserved that data on a different partition.

2.  There are probably better and different ways this could have been fixed. But I’m comfortable with the success I had.