Some are bothered by his positions that explicitly want to roll back rights for people while others are bothered by his current "government aid is bad" position given how much such aid he has received.

I'm guessing if he actually lived the beliefs he said he used to have more people with support him.