Of particular note, sometimes expressing an unpopular opinion on a sensitive subject (or a subject on which some people have very strong feelings, even if it's not sensitive to most people) can trigger a cascade of "revenge downvotes". So while the original comment might get upvoted back to positive, comments on completely unrelated topics might pick up undeserved downvotes.

I once had a few week period where I noticed the majority of my comments drop by exactly 2 karma, usually around the same time of day, regardless of how many upvotes they'd received or how heavily trafficked the thread was. My best guess is that I'd upset two people (or one person with two accounts) and they kept clicking back on my profile until they got bored with it.