My biggest complaint about velocity is that most people I've worked with don't actually think of it as velocity, they think of it as speed. The vector aspect of it is often ignored, and of course if you're going fast in the wrong direction that isn't particularly helpful.

Trying to model velocity as a "metric" also bakes in A LOT of assumptions. If the right cards aren't being created, or if the backlog isn't being prioritized correctly then what you are attempting to measure isn't even velocity (or at least not the specific velocity in the direction you ought to care about).