I didn't notice the problem until you mentioned it. For me it was obvious that it points up/down, but now that you mention it I see that it could point in other directions as well.
I guess the question is: Do users get confused about the arrows?
I am certain the users do not get confused about the arrows. However, if you can wrap your head around it, it might actually look like it is pointing at the message itself, but that would have no actual meaning to a person.
It might be mistaken for a collapse/expand option, but people have been conditioned that + and - or the down arrow and right arrows are for that.
If it had a stick indicating the bottom protruding from it, it would appear to be a direction rather than an action link.
I guess the question is: Do users get confused about the arrows?