Last I checked, the votes still go against your karma. It just doesn't display anything below -4. I think this was intended to prevent hive mind downvoting.