You're right.

Even if we could see it, it'd be more of an impossibly-brief flash of light that dumps all of its energy in one place.

It wouldn't be a weird fireball that moves across the sky.