"Seed oils" are a nonsensical category because the main example is canola oil which doesn't actually have the problems associated with them (bad omega-3:6 ratios).
I thought the problem with seed oils is that seeds don’t want you to eat them and their chemistry may reflect that. Fruit bodies such as olives on the other hand are “designed” to be eaten and so aren’t likely to have such defenses.
Olives are a pretty weird example of something that "wants" to be eaten by us, given the insane amount of processing it takes to make anything remotely palatable from them.
Contrast with, say, sesame or sunflower seeds which can be eaten straight from the plant raw, or pumpkin seeds which just need a simple roast and peel, I'm not sure that your categorical assertion really holds up, as intuitive as it may seem.
It just has to be mammals in general. If e.g. a rape seed doesn’t want to be eaten by squirrels, to take a common seed eating mammal, there’s a decent chance that as mammals we share enough in common that whatever surface is being targeted in squirrels would affect us as well.
If something was specifically targeting birds or reptiles then it may not affect humans, but are the seeds in question in environments without mammals? I don’t think so?