Evolution's indifference is sometimes indistinguishable from cruelty. Presumably we're all descendants of the human and pre-human babies that freaked out at the flimsiest sign of abandonment. Presuming further, the ones who didn't may have had better outcomes for themselves and their parents 99.9% of the time, until e.g. they got lost in the tall Savannah grass just one time, and because of that never got to grow up and have children of their own.

Even if it only happened to, say, 20% of the quiet babies, it's enough to surpass the costs of paranoia.