I almost never buy fancy clothes with materials like linen. I do buy fancy sheets :)
With work-at-home, most of my work day is spent in fluffy cotton pyjamas in the winter and simple underpants in the summer from basic-but-large companies like Hanes, Fruit-of-the-Loom, and similar. I'll put on a plain cotton t-shirt and a pair of jeans to go out to the store or whatever. If it's cold, I'll toss on a worn-out long-sleeved t-shirt (the good ones see very little use) and/or a jacket of some kind.
I do have fancy clothing for business meetings, as well as for specific types of exercise, but the total number of hours of usage on those is quite low.
Happens with clothes too, had to return several linen shirts for this reason.