The sickness is due to how the particles interact with the lining of your lungs, and breathing in enough to get ill effects will be much more than you consume through food over a short period.
There may be longer term effects, but I'd imagine we would have identified more of them by now, considering how severely dangerous silicosis is.
My property is built on the silt plume of a river that now empties out a few miles away. It does become a dust bowl when grass is removed and everything's dry. It otherwise stays in its place.