That's not really true. You know about the Elgin marbles precisely because they're vigorously contested - unlike 90% of artefacts that left Greece, Egypt, Italy, etc in perfectly legal circumstances.
It's true that colonial powers (chiefly Britain and France) have a particularly bad record, but not "all museums" contain looted stuff.
but a lot of the artefacts left their countries in circumstances which sounded legal but were not.
Consider the example of Euphronios Krater[0] which the Metropolitan Museum of New York "legally" acquired, and it took 40 years to convince the museum that it was not actually legal.
Or the Ishtar Gate[1] parts which were smuggled out of Egypt by the germans and then bits were sent off to museums in the US, Denmark, Canada etc.
Or the parthenon marbles owned by the Vatican and recently "gifted back".
And that's big stuff! The smaller stuff is just less noticeable, but it's obvious if you think about it that a lot of stuff acquired before 1900 has not being acquired in a way that would be considered "proper" today.
I don't think _anyone_ other than the owners who suffered the theft cared about "legit origins" until very recent times.