I’ve been running a web and email server out of my house for three years. When I did the initial install I had experience with configuring Windows as a web and email server but Exchange was unstable and I hated having to reboot Windows servers to apply security patches. Sasser and Blaster had just made their rounds on the world and I, like thousands of System Administrators, was affected by the worms. As a result of my experiences I chose to increase my Linux experience and the best way to do that was install and set up Linux as a web and email server. Three years ago I chose to do that with Fedora, Apache2, and Sendmail. The initial configuration wasn’t simple. I had to learn how to compile source code. I spent hours digging through forums and how-to articles to get my configurations correct. But after a couple of weeks working in my free time I had a stable, secure, Linux system with web access to my email. In the course of three years I never had to reboot the server and when I found security holes I did a quick up2date and it applied the upgrade and installed a patch. I was very happy with my new Fedora server and it ran perfectly until last Tuesday…
Redhat moved from Up2date to yum and as far as I could find Fedora Core 2 stopped being supported late last year. As a result I couldn’t use up2date to patch Apache2 when security vulnerabilities were found for Apache 2.0.51, instead I would have to download source and compile it then reconfigure it to work with Sendmail and Squirrelmail. When faced with this challenge I decided it would probably be best just to update to a new version of Fedora that supported Yum and Redhat still offered updates for. The install from CD seemed to go smoothly. I inserted disk one of five, rebooted, clicked next a few times, and then picked the radial button for upgrade. The Fedora Core installer (Anaconda) found my FC2 partition and I chose to upgrade hdb1. I clicked next a couple more times and the installer searched for my software then started the install. After inserting the five disks and waiting around two hours the install finished and rebooted. Upon rebooting the system started the kernel promptly returned the error: “mkrootdev: expected fs options; mount: missing mount point; kernel panic.” This message means it was expecting a specific file system or trying to mount that file system from a specific device but couldn’t find it. Basically, the system knew to look for the “/” directory but couldn’t find “/.” Without that directory the kernel couldn’t find the files it needs to boot and instead panicked and crashed.
I (like any decent System Administrator finding himself in an unfamiliar area) turned to Google in my time of need. I found that dozens of people have had the same problem with a FC 5 upgrade from FC 2. This is basically a known issue and the inevitable answer on each forum I found is “format and install FC 5 clean.” This confounded me. This sounded like the answer a bad MCSE would give me. I called Compaq one time and they told me “Format and Quick Restore.” Nobody asking my advice has every bought a Compaq since then. But, Linux guys are supposed to be the geeks who find a work around when there isn’t a known solution. Linux is an idea and a community where people come together and help each other find answers when a corporation hasn’t defined what you should do. I turned to my friends on IRC, they know how to work together and find answers… But, though IRC has helped me numerous times in the past, each suggestion I received led to a dead end.
I was digging through fstab and mtab, editing my grub menu and boot options, but each step led only to different errors. Nothing I did helped my kernel find “/.” The final answer I got from one of my colleagues at NetStandard Inc. was to upgrade from Fedora to Debian (which I like better anyways and I will eventually do) but I’m a true geek at heart and I hadn’t given up yet.
I knew Fedora 5 wasn’t the newest version of Fedora when I upgraded, I just didn’t want to waste 5 more disks and I had 5 on another machine at home. Fecora Core 6 was available off the mirrors on Redhat’s web site. I burned disk 1 and started through the process outlined in paragraph 2 only this time the upgrade program found Fedora Core 5 on hdb1 and received an error that the file system was in an unbootable state. The Fedora Core 6 disk told me to boot to rescue mode and repair the file system. I booted to rescue mode and ran fsck on device hdb1, hdb2, hdba1 and hdba2. Then booted back to Fedora Core 6 and installed from the five disks. The install went fine and I was able to boot past the error I had been getting, for a second I thought I was in the clear, and then an error that SeLinux could not permit a process (I don’t remember which) to run followed, inevitably, by my favorite error “Kernel Panic.” I was up a creek again but this time I had a paddle. I hadn’t been running SeLinux on Fedora Core 2 so I thought if I disabled it I’d be able to at least boot to a different error. I booted to disk 1 and exited to System Rescue mode then found the file that starts SeLinux and edited the file to disable SeLinux. Upon reboot Fedora Core 6 loaded up and X Windows even started. My config files for Apache, Sendmail, IMAP, and Squirrelmail were overwritten during one of my upgrades but I was in the clear, I still had my data! After a few quick Google searches all my services were running again, and with the newest, most secure, version.
This experience taught me a lot. I’ve learned more about how Linux works and how different files (fstab and mtab) affect the OS in different ways. I’m very comfortable with booting to various Linux recovery disks (over the last week I’ve used Helix, Fedora Core 5 and 6, Knoppix, and Knoppix STD) as resources. I also gained experience with some of the tools (fdisk, fsck) I had to use to get the OS to boot again. But, more than ever I’m comfortable with the ability of Linux to preserve data regardless of changes to the OS. At no point during this experience did I feel like my data was unrecoverable or that I’d have to format and reinstall the OS.
A couple of follow up notes:
1. With Linux always put your /home directory on a second partition. Even if I had formatted and installed Debian I would have preserved that data on a different partition.
2. There are probably better and different ways this could have been fixed. But I’m comfortable with the success I had.
The PCMech.com weekly newsletter has been running strong for over 8 years. Sign up to get tech news, updates and exclusive content - right in your inbox. Also get (several) free gifts.
Upgrading from FC2 to FC5/6 … You’re a braver man than I am. Nice article, Branham.
Personally, I think you’re crazy for trusting the upgrade process for *any* OS. I always re-install.
You know, if you simply set up two partitions on your system – one for the OS and one for your custom stuff – you can, in fact, wipe the OS partition and install cleanly. After that, just make changes to a few configuration files (mostly in “/etc”) and you’re done.
Nice article. I’ll assume you had backups. If was a newbbie from reading, you almost infer it’s not needed with Linux. Sometimes a clean install is the best option, if not the most manly. I could buy a car and continually keep riveting sheet metal, and buying parts from junk yards for 25 years, instead of changing the car. I guess it depends on how much the time is worth.
I always back up my critical stuff and anything else I deem useful, either to another hard drive – internal or usb attached – or to another box.
When upgrade problems have reared their ugly heads on occasions, I’ve gone ahead and done a fresh install without reformatting the partitions. On the SuSE’s, this has left my /home totally intact on drives that have only / and swap partitions.
One to remember, there has always been some way for me to get out of trouble with Linux. Solaris is another matter, change to new hardware and move your disks over and there is no way it will work on the new platform, a colleague and I spent a day trying to do that on a customer’s site and in the end we had to do a fresh install and NFS copy his data across from the old box overnight.
I had the same problem, repairing a RedHat server of another person.
The solution is just, to type #tune2fs -L / /dev/hda1 . For coming in, use any live CD, mount the installed partition, and chroot.
The reason for this is the stupidness that in the last time more and more distros mount the partition by the LABEL option.
Dude, you’re dumb!
Backup your data & do a clean install!
**Microsoft suggests this–the experts tell you this.
**Sun Microsystems/Solaris tells you this–their OS “update” is to a new partition according to their system engineers
**From FC2 to FC6? Nobody suggest a jump of 4 version levels! That’s like going from Windows ME to Vista on the same disk! Microsoft doesn’t even approve of that!
**Red Hat suggests backuping your data and configuration files before attempting an on-disk upgrade.
If you do something STUPID, don’t tell EVERYBODY!!! Learn from it and go on!!!
**You blew-up a working system.
**You didn’t research before the upgrade. (DECENT system administrators research BEFORE–it’s in the rule book–don’t you have one?) For instance, Apache 1.x is different than Apache 2.x in configuration syntax.
**You didn’t make backups–it’s in the rule book, too!
**You insult “people”; names are important: RED Hat, not Redhat.
**You have no way to restore to you older configuration! Hard drive are cheap–no need for you to be also!
**You think you are better than you are: (dumb != decent)!
hike,
He says as a follow up note that he’s got his data on a second partition. Learn to read before you insult people.
Hike,
“If you do something STUPID, don’t tell EVERYBODY!!! Learn from it and go on!!!”
No DO tell everyone! It’s inevitable that someone else is going to do the same thing. If you’ve found a solution post it! Saves everyone else from struggling like you did for hours.
For instance, as the author says “inevitable answer on each forum I found is “format and install FC 5 clean.”” but now for someone going from FC2 to FC5or6 we now have a possible work around no matter how sloppy a work around it is.
Lachild
Good posts Everyone. I appreciate the feedback.
As many of you said I didn’t stress the importance of backups as I should have in this article (didn’t mention them at all actually). Before you make any serious changes to a system make sure you have a good backup you can restore from.
In all my production environments I use a product that NetStandard resells called DataSafe. It allows me to perform all my backups over the web and its so intuitive I almost forget about it sometimes.
Thanks again for the comments!
It’s a tricky position to be in. A lot of RPM distributions seem to recommend the ‘format and reinstall’ variety of upgrading. I’ve been a Debian and Ubuntu user of many years, and quite often track the Debian Testing branch. This ensures that all packages are always up to date, and I don’t really encounter any of the large release updates that plague other distributions. Even tracking Debian Stable avoids this. Although it will have a major release every few years, you can selectively upgrade the system a section at a time. Apache first, Exim second, kernel last, or whatever you choose. It will also save all your config files and data
Enjoy – it’s a learning experience every day
Back in 1999 I built a Red Hat 7.0 machine for email and web services. I have since upgraded it remotely from 7.0 to 7.3 to 8.0 to 9.0 to Fedora Core 1 to FC2 to FC3 to FC4 to FC5. Every upgrade was done remotely.
For the first few upgrades, I used apt-get and for the last few I used yum. The only time that I had to visit the machine was when I switched from the lilo bootloader to the grub bootloader. I had made a mistake in my grub.conf file. I created a grub boot disk and used it to boot and correct my mistake.
I’ve had various other issues that I was always able to correct remotely.
Backups are always a good idea if you care about your data.
To upgrade from Fedora Core 5 to Fedora Core 6 with yum. Go find a mirror that contains the FC6 distribution and find the fedora-release package for FC6. Download it and install it with the “rpm -Uvh fedora-release*.rpm” command. It may require another package, download the required package and install them together “rpm -Uvh *.rpm”
Then use “yum update” to upgrade the OS. You might have some dependency problems, I usually remove packages that complain about dependency issues and then install the new version after the upgrade.
yum install packagename
Good luck, google and patience are required.
No, I don’t guarantee that this will work for anyone. You have to be a bit of a geek. If you screw up your box and can’t find a way out of it, you will have to rebuild.
Let’s not flame my author without reason here – we are here to do you a service by putting content online that will help you at some point. Its a blog, folks – its informational. He had a hell of a time upgrading… that’s it.
The fact that he didn’t mention backups has nothing to do with the article itself. The purpose was to talk about the “Challenges of Upgrading Linux” that our author ran into. Someone who types that into Google will get this site as the second hit, so it may be useful to them if they are curious about it.
As far as upgrades are concerned, why wouldn’t you try an upgrade? Sure, Windows upgrades suck and should never be done because of significant changes to the kernal, but Linux? Even with a change in kernal versions, the operating system has not changed how it functions significantly for years. I would consider an upgrade compared to setting everything up all over again, personally.
Our author is by NO means dumb or incompetent – and if you think so, perhaps you should look in a mirror. They don’t hand out MCP/CCNA/CCSP out to just anyone.
Have you thought of trying FreeBSD? The upgrades are quite easy, especially upgrading minor versions.