I'm not sure simply saying "I'm not being an asshole here" precludes you from being an asshole, in the same way that "I'm not a racist but..." is no guarantee that what follows isn't going to be racist.