I did this a few years ago and haven't looked back.

I had the issue where I was suddenly following everyone again on one occasion (years ago), but since I've unfollowed everyone again, I haven't had any issues.