Indeed, verifying a signature does not rely on mathematical complexity like hashing does. However, they do usually involve finite fields and large numbers that require a whole bunch of complex mathematical operations. Then there's also the fact that optimizations are often not available as they would allow for timing attacks.

Compared to something like simple a CRC32 checksum to verify that the data was transmitted correctly, these operations will always be complex and more time intensive than you'd prefer them to be.