It does have something to do with guilt, because the ultimate concession of acknowledging privilege is that you didn't truly earn your advantages, which is a root cause of perpetuated injustice. That's where the guilt lives: being the beneficiary of systematic injustice. This doesn't make you a bad person, of course; pretty much anyone in a position of privilege will capatalize on it, as that is human nature.
Acknowledging privilege doesn’t imply you didn’t earn your advantages; it implies there were some that were handed to you. Plenty of others need to be earned.
It is rare, extremely rare, to meet someone with absolutely no privilege, and that isn’t a place to aspire to.