That takes me back. In the WinNT 4.0 days my NTFS hard drive would become so heavily fragmented that it would take Explorer over a minute to display folder listing as it hunted around for file icons and other metadata. And then Explorer would randomly decide to rebuild its icon cache a freeze up for another minute.
The built-in defragmenter did nothing to improve things.
Our idiot IT guy said I should compile less often.
I had to resort to WINFILE to do anything with the file system, at least until I convinced my boss to get me a Norton Speed Disk license.
And yet when Win2K came around I refused to upgrade. It'll be too slow and buggy, I figured.
Explorer performance issues have become one of my top annoyances in Windows 10. There's so many apps what want to hook themselves to provide context menu items, icon overlays or something else. If these apps are not responding in timely manner, Explorer gets really slow and figuring out the cause is not straightforward.
I always thought NTFS was declared to be relatively immune from fragmentation problems. That’s why there was no in-built defragmenting tool for NTFS-formatted drives until much later on.
> thought NTFS was declared to be relatively immune from fragmentation problems
The operative word there there is relatively. In terms of main file content storage it did (and still does) a very good job, especially relative to FAT's naive first-available-block allocation strategy, until the volume gets near-full. How much is "near full" very much depends on your local use cases (rules of thumb like "the larger of 10% or 20Gb" are banded about, but they are just myth-like guesswork at an old-wive's-tale level IMO as they are usually traceable back to times when volumes were much smaller and common data patterns vastly different).
Once volumes are near-full not much helps - this is still true for modern variants of NTFS, ext, and any other filesystem (I'm not counting filesystems with "active defragmentation" as that is essentially a filesystem plus a tool). Some have tweaks that help reduce the effects (NTFS being able to store small files in directory blocks directly for instance) but it is not practical to completely rid yourself of them especially in pathological cases. Furthermore once a file is fragmented it will stay that way until removed & replaced or acted upon by a defragment process on the filesystem, and deleting fragmented files may leave the free space around them a mess of small blocks so delete & reallocate may not help on a near-full filesystem.
Any filesystem that claims to be fragmentation free without a list of caveats attached to that claim is snake-oil.
The allocation strategy in use by the OS doesn't necessarily have any relation to the filesystem it is being used with. IIRC, NTFS being considered "immune" to fragmentation concerns had a lot more to do with a shift in the Windows allocation strategy at about the same time than anything in the filesystem itself.
The allocation strategy is generally part of the filesystem design, not the OS design. NTFS was properly designed with real use cases in mind to avoid fragmentation where possible. Earlier Windows filesystems (FAT) were designed to just do the job simply.
You can* use a more effective strategy with FAT* but you need to jump through some hoops as it doesn't have any built-in structure to help. It is far more practical to just ask your users to use a more modern filesystem, except for smaller removable devices where the issue is far less significant anyway.
>NTFS was properly designed with real use cases in mind to avoid fragmentation where possible.
Once said that NTFS was very properly designed (and still works fine after some 25 years with only a very few modifications and on relatively huge volumes) it does have a few "strange" characteristics.
JFYI, a strange case when there was apparently no solution (due to some queer placement of the NTFS filesstem structures), a 6 Gb file couldn't be stored contiguous on a 8 Gb USB stick (and of course FAT32 wouldn't have allowed a bigger than 4 Gb file):
> due to some queer placement of the NTFS filesstem structures
It isn't really queer: some core, regularly accessed, structures are deliberately in the middle of the filesystem to reduce head movements in the common use case of it being a single volume (or the most significant volume for the currently IO tasks) on a spinning-disk-and-floating-heads based drive. This essentially splits free space in two from the very start so an 8Gb volume is going to start with two ~4Gb ranges of unallocated space not one single 8Gb range.
ext* filesystems distribute group descriptors, superblock backups, and other structures, around volumes instead of keeping them at one end, for simialar reasons. See https://www.slashroot.in/understanding-file-system-superbloc... and much other online documentation. (though spreading the superblocks around isn't a performance thing, it is to reduce the damage potentially caused by localised physical disk corruption)
The queer part is software coupling itself so tightly with the storage layer such that it can't cope with a file not being physically 100% contiguous. The example in that link sounds like a virtualisation method trying to be very aggressive about optimising the mapping between guest disk access and physical storage (which might convey a small performance benefit) - if you are optimising like that then you are probably better off trying to use raw block devices rather than files on a host filesystem.
Well, the point is that after the 2000's some new hard disk like devices (including USB sticks and SSD's) have been available and became common in use, making most if not all provisions that were "smart" for rotating media largely meaningless, and the size of rotating media increased so much that the "around the middle of the filesystem" has been a largely moving target.
The "default" settings (including the placement of the $MFT on cluster 786432 or 0xC0000 if you prefer) is clearly something that made a lot of sense on a given size of volume, but having it "hardcoded" as default for any volume larger than around 5/6 GB makes not any real sense, hence I stand by my "queer".
As seen in the given reference, it is possible (and relatively easy) to "move" the $MFT (and other NTFS filesystem structures) to an "earlier" address on the volume, and the NTFS (which as said has been very, very well designed) works and performs just fine.
On the other hand, moving it near the actual "around middle" (on rotating disks) is far from being easy, even nowadays.
What I was pointing out is that after all these years the "format" tool has not evolved allowing directly these kind of changes/offering the user the possibility to finer tune the file system.
Either you never used FAT* or were very very lucky! It (and other old, naive, filesystems) used a pure first-available-block strategy that almost invites pathological fragmentation especially in a multi tasking (or worse, multi-user) environment.
Also, I've seen ext* filesystems just as badly fragmented as NTFS ones in similar environments.
One key problem I've seen on NTFS volumes in the past is with those that were converted from FAT (IIRC old Windows NT installations would do this even for fresh installs: install on FAT* for the first stage then convert to NTFS before the second) - the conversion process didn't move files around when converting to 4KByte blocks from larger ones so left a small chunk of free space after every file.
> rewritten terabytes of data over the same blocks
It isn't about the amount of data throughput or the reuse of blocks: it is how files are allocated space as they grow, particularly when there is concurrent activity, and how free space fragments as files shrink or, more likely, are removed.
> What kind of environment is that?
Busy near-full write-heavy multi-user file servers particularly. Multitasking exacerbates the "how files grow" issue, multiple users exacerbates both, and multiple concurrent users multiplies the issue further, being near full means that data fragmentation countermeasures are circumvented because due to worsened free-space fragmentation (all the space left is in small units).
Ignoring a few pathological cases (that are usually unrealistic thought experiments), if the filesystem never gets particularly full then neither NTFS nor ext* are likely to fragment significantly. But in the circumstances mentioned above both are just as likely to. Their key mechanic for avoiding data fragmentation (keeping some free space after files where possible to allow for growth) is very similar, other technicalities where they vary more widely (like storing small objects in directory structures directly or the MFT) have an effect, but a much smaller effect than that attributable to the methods they share.
Of course even some near-full multi-use volumes aren't going to fragment significantly at the filesystem level. If a volume is hosting large database files then they are likely to be reasonably contiguous from the filesystem's point of view and fragmentation issues within the data files is massively more significant. Also if most of your concurrent users are only reading (perhaps a media archive system maintained my a small team, perhaps one person, or just at most one person at any given time usually, but accessed by many?) they can be ignored from a fragmentation PoV (readers aren't going to affect data layout unless you have active writers and some exotic MVCC arrangement going on at the filesystem level).
The built-in defragmenter did nothing to improve things.
Our idiot IT guy said I should compile less often.
I had to resort to WINFILE to do anything with the file system, at least until I convinced my boss to get me a Norton Speed Disk license.
And yet when Win2K came around I refused to upgrade. It'll be too slow and buggy, I figured.