>> Fruit almost always has plenty of fiber in it to modulate sugar absorption
> but do we have sufficient sources to back it?
It's not an assumption, it's a well known fact that adding fat or fiber[2] to a high GI food will decrease it's GI. Conversely removing fiber increases it. For example brown rice(50) has a lower GI than white rice(72). The spike/crash in blood sugar from high GI foods is likely to cause carb craving and is linked to obesity/diabetes.
I have been unable to find out whether wild ancestors of grapes (if there are any left) have more fiber than domestic varieties. And, I don't know about dates.
But I am assured by numerous online sources that domesticated grapes have vanishingly little fiber, however that came about.
I have encountered a claim that sugar in mango is, uniquely, mostly fructose. But I have been unable to confirm that.
I'd assume at least certain fruit would still be dangerous, like grapes, figs, maybe dates. And of course candied fruit too.