I think the issue is that in a calm, rational state you would think "let me just slowly visually examine the door area and where a typical lever or latch would be" but in a state of panic humans get tunnel vision and lose their slow methodical approach. That's why emergency planning and evacuation type designers will create UX/UI affordances that are multi-factor and unmistakable (flashing lights, alarm sounds, bright colors, capital letters with simple instructions). Think about the neon red EXIT signs above doors or the horizontal panic bar door mechanism (which to be fair has other design considerations like the fact that in a crowd crush situation even people unable to move their hands and arms may push the door open using the weight of their body and torso itself against the panic bar).