Let's see if I understand the argument. Many people's ancestors didn't own slaves. But, because of social construct, they have a leg up (+advantage) in the present because of skin color in the present society of the US (not because their ancestors were abolitionists or slave owners). Which is why people argue against current racism, not solely historical racism.

So even if you didn't benefit from past injustices directly, the ongoing inertia in society results in your net benefit or detriment as an immigrant on average due to skin color.

(Note, I'm trying put the argument into perspective with a steelman, not dismiss your experience as you've described it.)

Or worse yet you came from someplace like South Africa, where you were abused for racial reasons and then after coming to the US you get blamed for historical action of some plantation owner physically abusing others. Double points if you're Palestinian or something and barely made it here with the skin on your back.