It's always been -4 for me. At times it's felt like more when I've doubled down on my unpopular comments with more downvoted comments (and because usually every down vote hurts).

I think the limit probably discourages long winded "all these downvotes are a badge of honor" comments. People who crave downvotes only get four. People hurt by downvotes only get four, too.