You miss the important point, which is the productivity gain means the average living standard of society as a whole increases. A chunk of what is now regarded as 'toil' work disappears, and the time freed up is able to be deployed more productively in other areas.

Of course, this change is dislocating for the particular people whose toil disappeared. They need support to retrain to new occupations.

The alternative is to cling to a past where everyone - on average - is poorer, less healthy, and works in more dangerous jobs.