Everybody has their own way of rationalizing why something is not 'their problem'.

You could buy some food and give that instead. I've had a guy panhandling outside a supermarket here refuse food that I'd just bought because it wasn't money. Clearly he wasn't hungry but to me that doesn't make me look at all homeless people in the same way.