I don't particularly like dumping everything in home, but XDG is worse. Now, if I want to start over with a clean profile, there's a whole list of locations I have to zap. Spraying shit in three directories is worse than one.
my problem with that "one" directory is that i have to take care specifically to back it up. applications that at least use ~/.cache means i don't have to worry about backing up a bunch of useless stuff.
Consider a browser cache. /tmp is not guaranteed to be preserved between program invocations[1]. /var/tmp [2] might be a better place. /var/cache [3] is intended for exactly this kind of thing.
Unfortunately, it can be far more useful for an administrator in a multi-user setup to want user-specific caches in home directories. That way, you get all of the infrastructure for managing user data for free, like quotas.
Also note: if you're considering putting data into /tmp or /var/tmp, please honor the TMPDIR environment variable, if it is set[4].