It is correct. What is incorrect is to thing that symmetrical interface elements can have asymmetrical actions.
How would you like if turning steering wheel to the left would give left turn, but turning it to the right would stop the car?
If you want asymmetrical meaning, please, use different elements.
By the time someone aquires 500 points, they will likely have spent enough time on the site to learn the distinction between the two. While using text labels would be fine, I do not think that would be better than using the symbols.
The reasons for clicking may not be symmetrical, but the actions when clicking are symmetrical. The points go down with the down arrow and the points go up with the up arrow. This is symmetry even if the reason (agree, disagree, quality, spam, etc.) for the increase and decline of the points is asymmetrical.
When the site was initially made that was what the meaning was: Agree/Disagree.
However that norm changed when people realized that was a bad idea.
The UI didn't change with the norm change.
At one point the norm was you could downvote to disagree if the comment had a score greater than 1. But now that comment scores are no longer displayed, that norm changed too.
Now the downarrow basically means spam/stupid/useless, etc. Actually for spam you also need to click on flag.
