How am I missing what matters? The glycemic index of most fruits is higher than, for example, the glycemic index of sweetened yoghurt. So it makes little sense to eat fruits instead of sweetened yoghurt (but it would be a lot better then, say, sweet cereals).

What matters is added sugar anything is higher in glycemic index compared to the same thing with no added sugar.

You're correct in that adding fibre would alleviate some of it but might as well eat the real thing without added sugar and take a piece of fruit. It's bound to be easier and better absorbed (containing various cofactors) compared to trying to emulate the cocktail with added vitamins and fibre.

In addition dairy is notoriously insulinogenic so not necessarily a good comparison (although non added sugar full fat yoghurt is probably okay).