Most places I worked have a blame.ignoreRevsFile[0] somewhere on the top level to inhibit this. It's a bit awkward because first you need to commit, then you need commit again to update the commit hash in the ignore revs file, but it's great for filtering out pure refactor churn.

[0] https://git-scm.com/docs/git-blame#Documentation/git-blame.t...