The neat thing about inline dedupe is that if the block hash already exists than the block doesn't have to be written. This can save a LOT of write IO in many situations. There are even extensions where a file copy between to VMs on a dedupe storage array will not actually copy any data but just increment the original blocks reference counter. You will see absurd TB/s write speeds in the OS, it is pretty cool.
This is only a win if the dedupe table fits in RAM; otherwise you pay for it in a LOT of read IO. I have a storage array where dedupe would give me about a 2.2x reduction in disk usage, but there isn't nearly enough RAM for it.
This array is a bit long-in-the-tooth and only has 192GB of RAM, but a bit over 40TB of net storage, which would be a 200GB dedup table size using the back-of-the-envelope estimate of 5GB/TB.
A more precise calculation on my actual data shows that today's data would allow the dedup table to fit in RAM, but if I ever want to actually use most of the 40TB of storage, I'd need more RAM. I've had a ZFS system swap dedup to disk before, and the performance dropped to approximately zero; fixing it was a PITA, so I'm not doing that anytime soon.
Be aware that ZFS performance rapidly drops off north of 80% utilization, when you head into 90%, you will want to buy a bigger array just to escape the pain.
I think that is well known amongst storage experts, though maybe not everyone who might be interested in using ZFS for storage in a professional or personal application. What I’m curious about is how ZFS’s full-disk performance (what is the best term for this?) compares to btrfs, WAFL, and so on. Is ZFS abnormally sensitive to this condition, or is it a normal property?
In any case it doesn’t stick out to me as a problem that needs to be fixed. You can’t fill a propane tank to 100% either.
ZFS has gotten significantly better at 80%, but 90% is painful enough that I almost wish it would reserve 10% a bit more explicitly (maybe like the old Unix systems that would prevent non-root users from using the last 5% of the root partition).
All my arrays send me nightly e-mails at 80% so I'm aware of when I hit there, but on a desktop system that's typically not the case.
