That's the exact opposite of my experience. I grew up on a farm, and as you might imagine it involved hard physical labor daily. I would 100x rather do my current job as a systems engineer. It's so much easier than working on the farm that I feel guilty sometimes, because I know that I work 1/10 as hard as people making 1/5 of what I do.
They're not saying it's harder, just that it leaves you not wanting to do other things when you're done, and in a way that doesn't map to traditional physical fatigue, which is why it's "weirdly" taxing and not just taxing.
Mental and emotional fatigue accrue and exhibit themselves differently sometimes, but often matter quite a bit in the end. Plenty of people choose to leave well paid office jobs for more physical jobs that pay less. Unless you think they have a mental illness or are stupid, presumably they did it for a good reason.
I used to be a high school teacher and, though not as physically taxing as working on a farm, I feel the same about how much easier my current tech job is.
It depends upon the person and the exact situation, but it's a fundamentally different kind of tired. My students marvel at this.
They can tinker with mechanical things for a 12 hour crunch session before a robotics competition, and end up achy, hangry, and physically tired. But it might not be as exhausting as as an intense 1.5 hour math class or chasing a bug staring at screens for a couple of hours.
