The confusion lies in why we would measure the efficiency of human labor in joules per unit of work instead of hours of human effort per unit of work.
In particular, if we can make a machine that spends more joules than a human, but reduces the human effort by orders of magnitude, why would that be "horribly unproductive"? Most people would call that amazingly productive. And when they want to broaden the view to consider the inputs too, they're worried about the labor that goes into the inputs, not the joules.
(And if the worry is the limited amount of fossil fuels in particular, we can do the same with renewable energy.)
Joules are just a measure of work, and this all started by an attempt to say how productive we are because we need fewer farmers today. My argument is that we only need fewer farmers because we found a cheap source of energy and have been using that to replace farmers.
When looking at joules its an attempt to compare something like a human cutting a field with a scythe and a tractor cutting it with an implement. The tractor is way more efficient at cutting it when considering only the human hours of labor cutting the field. But of course it is, a single barrel of oil has way more energy potential and even a small tractor will be run with fuel milage tracked by gallons per hour.
Farmers were already using the cheapest source of energy, sunlight. Oil came late to the party here after plant matter and coal because it was slightly more convenient not because it has more energy.
Back in the day, wood powered tractors beat the fuck out of horses, and horses beat the fuck out of human labor because they could digest cellulose. Oil is just very slightly cheaper. Even today people heat their homes with both wood pellets and oil, meanwhile there’s cheaper alternatives.
In particular, if we can make a machine that spends more joules than a human, but reduces the human effort by orders of magnitude, why would that be "horribly unproductive"? Most people would call that amazingly productive. And when they want to broaden the view to consider the inputs too, they're worried about the labor that goes into the inputs, not the joules.
(And if the worry is the limited amount of fossil fuels in particular, we can do the same with renewable energy.)