There should be an auto block feature. If you have it on and they downvote you, they get blocked and can't see your posts anymore.

Second thought about gamification. A downvote should cost the person a point or two of their own karma.