File Systems – Which Need Defragmenting?

So, I was curious about, of all things, file systems. I am aware this is very nerdy of me, but since I got the Mac I was curious if I had to worry about such things as defragmentation. Windows users know that you need to defrag your hard drive every so often in order to keep your computer running smoothly. In the world of Mac, it is said that you don’t need to defrag. Why is this?

Defrag So, I decided to look into the different types of file systems and look for an answer. Is the file system Windows uses responsible for the difficulties?

The Different File Systems

File Allocation Table (FAT). This is a file system developed by Microsoft for MS-DOS and was used up to Windows ME. A disc formatted using FAT consists of a boot sector, file allocation tables, and the data. The boot sector contains the code necessary for your computer to boot up. The file allocation tables are a mapping of where on the disc certain files and directories reside. Then you have your data itself. The problem with the FAT file system is that when a file is deleted or a new file is removed, that blank space on the drive can be written to by something else. FAT does not take care of the location of new files when this happens, and that leads to file fragments being written all over the disk. The file allocation tables do allow the data to be found, but the read/write head on the hard drive would need to gather your data from disparate parts of the disk. This is why you see a performance decrease. The FAT file system is particularly susceptible to fragmentation by its very design. More information about FAT can be found on Wikipedia.

NTFS is short for New Technology File System. It was, again, developed by Microsoft for its Windows NT line of operating systems. That means NTFS is used in any version of Windows based on the NT kernel, including Windows 2000, XP, Server 2003, Server 2008 and the venerable Windows Vista. The major difference with NTFS is that it is based around metadata. Metadata is “data about data”, according to Wikipedia. In other words, the metadata is like a little mini-database on the drive which stores all kinds of information about the files and directories on the hard drive. NTFS supports things like compression, file-level security and other things useful to enterprise and those attributes are stored in the metadata. The cool thing about this way of dealing with files is that it is extendable to support other features. In fact, Microsoft has released five different versions of NTFS, each successive update providing more features. For information on NTFS can be found on Wikipedia.

As for fragmentation, NTFS is much more improved than FAT and is more efficient in how it handles data location on the hard drive. But, NTFS is subject to fragmentation. There was a myth for awhile that NTFS was not subject to fragmentation, but again, it was a myth. The NTFS file system is a very flexible one. As new attributes or capacity is called for from NTFS, it makes room and stores that information in the Master File Table. If a certain amount of space was set aside for a small file, and then that file becomes very large, portions of that file will need to be stored in other areas of the drive as NTFS creates new data storage areas are created. Also, the NTFS file system used clusters, just like FAT. So, yes, NTFS might have a reduced capacity for fragmentation, but it is still susceptible.

Ext3 is the file system used by Linux. What makes ext3 most notable in contrast to any Windows file system is that it is a journaled file system. A journaled file system is one where any and all changes to any file are logged into a journal before actually being written to the drive. The journal is stored in a designated area of the drive. By its nature, a journaled file system is much less likely to be corrupted. The journal is a running record of ALL actions to be performed on a file. So, in the event of interruption (such as a power failure), the events in the journal can simply be “replayed” to recreate the consistency between the journal and the files on the drive.

The nature of ext3 makes fragmentation all but non-existent. In fact, Wikipedia says that the Linux System Administrator Guide states, “Modern Linux filesystem(s) keep fragmentation at a minimum by keeping all blocks in a file close together, even if they can’t be stored in consecutive sectors. Some filesystems, like ext3, effectively allocate the free block that is nearest to other blocks in a file. Therefore it is not necessary to worry about fragmentation in a Linux system.”

However, when you look into OS X’s file system, which is also journaled, you begin to see why Linux is subject to fragmentation as well…

Hierarchical File System (HFS) is the file system used by Mac OS X. It was developed by Apple itself. We have the original HFS file system (often called Mac OS Standard) and the more recent revision HFS Plus (referred to as Max OS Extended). HFS has been through many revisions. HFS is hardly used anymore. HFS Plus was introduced with Mac OS 8.1. Most notable to our discussion is that Apple introduced journaling to their file system with Mac OS 10.3, along with several other features key to the way OS X operates.

There seems to be two schools of thought when it comes to defragmentation in OS X. Some say that it is not necessary because it is using a journaled file system. Other say that it is unnecessary, just not in the same way as Windows. OS X has the built-in ability to take care of file fragmentation and it will do this on it’s own. However, what can occur is drive fragmentation – small bits of free space between files. Performance-wise, it is almost a non issue and you will rarely gain any performance gain by doing a traditional defrag of an OS X machine. Drive fragmentation really only becomes an issue if you begin to fill your drive up to near full capacity. This is because OS X will run out of room for its own system files.

So, in short, there is no need to defragment in OS X unless you are beginning to fill up your hard drive. As this proceeds, you may begin to experience random OS X “weirdness” due to the OS running out of space for it’s temp files. When this occurs (or preferably before), a defrag of your hard drive will get rid of any slack space between files on the drive and recover space for the use of OS X.

I believe this will also be the case with Linux.

Other reference content:

So, In Short

If you are running Windows, you are subject to fragmentation. FAT32 systems are VERY prone to it. NTFS is less prone, but still enough so. Linux and Mac users are subject to fragmentation, but unlike Windows, it does not cause any performance drag on the machine. Also, Mac users really only need to worry about fragmentation as their drives near capacity. The issue with Linux and Mac is not file fragmentation (as with Windows), but drive fragmentation.

Hope that helps. And, as always, I welcome any people with knowledge of this area to comment. I did the best research I could on this, but it is certainly possible I got something wrong. And when you throw the Linux and Mac “defrag debates” into the mix, there are certain to be opinions on both sides.

Comments

  1. NTFS is a journaled file system, and journaling isn’t the reason that EXT3 doesn’t require defrag

  2. Thanks for the article. Very well done.

  3. With very good automatic background defragmenters available for NTFS currently, XP and Vista don’t need to manually defragged or scheduled periodically to keep them running smoothly. These auto defraggers run in the background, taking up barely any memory and monitor fragmentation. If necessary, defragmentation occurs quietly in the background when there are free system resources, so other, primary computing activities are not affected. It’s an elegant and easy solution to the fragmentation problem.

    I think Macs do something roughly similar, but i could be wrong, since I haven’t used OSX for a while now.

  4. Vista and Server 2008 use Transactional NTFS this is what makes both more secure than Windows 2000/XP. Vista is designed to defragment the hard drive on a shedule. Most probably Server 2008 will also perform the task automatically. Unlike Windows 2000/XP there is hardly any speed boost from running defrag with Windows Vista, at least I haven’t gained anything from doing so as of yet.

  5. Whether files are fragmented or not is not the only reason to use a defragmenter. By default most defrag programs also arrange files in order to speed up access. (The built-in Windows defrag only arranges executables based on prefetch data.) Different programs do it differently. Some will move recently modified files to the faster part of the disk/partition on the assumption that you want faster access to those. Others move the recently modified files to reside after all other files, so that when they’re modified again and may become larger, the drive heads don’t have to travel from a location at the beginning of the disk to farther inward in order to write the extra data. Some let you specify where certain types of files get placed. Most will also move and defrag system files which the Windows defrag will not, resulting in better performance when loading those files, since installing updates will result in some files changing a lot.

  6. it’s very good document

  7. Correction: Both NTFS and ext3 use metadata journaling. The actual data writes aren’t journaled, but any changes to the structure of the filesysem are. This means that you won’t get a corrupt filesystem if power fails, but you may get corrupt data.

  8. Lew Miller says:

    The argument that the HFS+ system never needs to be de-fragmented is not quite right.

    As this article, and others, explain, the jounalling and automatic file de-fragmentation features in OS X work to keep files in single pieces. To do so, the OS uses up segments of the “free” disk space. Eventually the “free” disk space becomes highly fragmented. Empirically, when that happens, the OS has little or no space in which to re-write the fragmented files.

    Tech Tool Platinum illustrates this problem very graphically. When I run its optimization utility, there are usually relatively few fragmented files and file fragments. However, the number of “free-space” disk fragments are almost always enormous. Typical numbers might be:
    98 fragmented files
    124 file fragments
    1,281 disk fragments

    The graphical display will show many discontinuities in the blue blocks representing the data, which in extreme cases are distributed throughout the disk. It is not uncommon for half or more of the files to have been migrated to the “end” of the disk.

    Consequently, Tech Tool’s optimizer disposes of the fragmented files very quickly, then spends considerable time clustering the files together at the “front” of the disk to create a single “free” fragment, which the OS then uses to de-fragment files again.

    I have a 12″ PowerBook G4 and an iMac 20. Almost every time I run Disk Warrior and Tech Tool, the start-up and operating performance of the computers improves noticeably.

    Therefore, I would argue that de-fragmentation is logical and justifiable, even in OS X.

  9. Nice explanation of the differences in FAT, NTFS and HFS used by MAC. Personally, I have never used a MAC so have always had to manually defrag my system. That is until now.

    As Jerico mentioned, there are some NICE behind the scene optimization tools for Windows. Also there are much better defrag tools than the default one that Windows gives you.

    I have used JKDefrag (which is free) and I feel it does a much better job at defragmenting the hard drive than the default one.

  10. Larry Miller says:

    This article contains a number of errors.
    The ext3 file system is not unique in being a journaled file system. NTFS has had this since it was introduced about 1993. In any event this feature has nothing to do with file fragmentation. The purpose of journaling is to help prevent the file system, but not necessarily individual files, from becomming corrupt. This was a common problem with FAT, FAT32, and other early file systems.

    ALL modern file systems are subject to fragmentation to some degree. Some early systems avoided the issue by requiring that there be adequate contiguous space before a file could be saved but this caused other serious problems. Some systems incorporate features to minimize fragmentation but it cannot be entirely avoided. A fragmented file will take longer to read because of the nature of the storage medium, regardless of the file system in use. In some early systems fragmentation was not a serious problem because these effects were masked by more serious problems.

    Larry Miller
    Microsoft MCSA

Leave a Reply