It's an interesting argument, but is it based on empirical evidence, or merely supposition?
(More recently, you can observe a seeming correlation between the decline in hours worked and the increase in labor productivity in the US in the 20th century--and I think in the rest of the developed world--but I don't know if this is causal or not.)
Smith specifically cites a reference, Remuzzini, in the passage.
I've the hardcopy version by Edwin Cannan with his notes, that doesn't specifically indicate additional sources, and I'm not aware of any further scholarship on this.
If you read Smith, he generally writes from a mix of scholarship and experience, though generally doesn't cite his sources with the rigour you'd expect from a current academic text. That's somewhat disappointing. He does occasionally go well wide the mark -- his accounts of barter and recommendations to the American colonies to avoid industrialisation are among his more telling blunders. I recommend him as a) informative, b) highly misrepresented (most particularly by the Mont Pelerin / free-market fundamentalist / libertarian camp), and c) imperfect but useful.
Generally, Smith is a bit of a cipher: he ordered his unpublished correspondence and notes burned on his death, and there are relatively few biographies of him.
The one of which I'm aware that was near-contemporary, by Dugald Stewart written a few years after Smith's death, I've marked up and published as PDF and ePub, links available here:
There's a great deal of highly ideological treatment of Smith, of course. I try to avoid that where possible.
Edwin Cannan and Francis Wrigley Hirst (ideological: free-market advocate) wrote biographies in the late 19th / early 20th century. I've not read them.
(More recently, you can observe a seeming correlation between the decline in hours worked and the increase in labor productivity in the US in the 20th century--and I think in the rest of the developed world--but I don't know if this is causal or not.)