With any Pigouvian tax[1] you encounter the problem of progressivity. The answer is to make sure that the tax revenues are directed towards making it easier for low income people to have better transit options. Whether that means improved transit systems or income based toll exceptions, it's relatively easy to offset the detrimental impacts that tolling may put in place.
I just think A) we have to be honest that is only a tax on the poor B) then have the rich pay in a different way, maybe it should be a limit on duration of travel or miles by humans, ie. no matter what car you drive or who drives you you're not allowed to travel more than x number of miles/day. (this is beginning to sound really odd)
[1] https://en.wikipedia.org/wiki/Pigovian_tax