It depends on the nature of the harm. If sedentary life reduced the typical lifespan from 70 to 50 (which would be dramatic) but didn't significantly affect your health during your 20s and 30s, it would have little to no effect on reproduction and thus would not be genetically selected.