I've definitely done that as well. I generally don't have a problem with it, but it makes for an extra conversation the first time someone sees the pattern.
Given how some of the other ergonomic changes in go have gone (closures capturing loop variables, for instance), I'd support a change to lexical scoped defers if it were on the table.