You might also have some level of intolerance to the so-called FODMAPs[0] (Fermentable Oligosaccharides, Disaccharides, Monosaccharides and Polyols). Speaking from personal experience, excess consumption of foods containing one or more (not necessarily all) of these carbohydrates can produce some quite unpleasant effects. None of these issues will present as an allergy because they aren't that. Dietary intolerances can be tricky things to pin down.

Not to discount the mold sensitivity suggestion either; it seems quite plausible to me as well.

[0]https://www.monashfodmap.com/about-fodmap-and-ibs/