Let's see if I understand the argument. Many people's ancestors didn't own slaves. But, because of social construct, they have a leg up (+advantage) in the present because of skin color in the present society of the US (not because their ancestors were abolitionists or slave owners). Which is why people argue against current racism, not solely historical racism.

So even if you didn't benefit from past injustices directly, the ongoing inertia in society results in your net benefit or detriment as an immigrant on average due to skin color.

(Note, I'm trying put the argument into perspective with a steelman, not dismiss your experience as you've described it.)