It's also because a filesystem is much more likely to have consecutive parts of a file stored consecutively on disc, whereas swap is going to just randomly scatter 4kB blocks everywhere, so you'll be dealing with random access read speed instead of throughput read speed.
ext4 is irrelevant to what happens when a file is backed by swap; even with swapfiles, the mm subsystem more or less goes behind the back of the filesystem to access the disk corresponding to the swapfile.
The overhead of making (size-of-read / 4kb) requests (potentially stalling the reading process for every page) is relevant even on an ssd; there are costs to random access beyond moving a disk head and waiting for a platter to spin into position, and those costs are still relevant with solid-state storage.
You wrote your comment like it was a rebuttal of the person above you, but the text supports what they said: A filesystem is faster than swap for this.
