Operating the controls of your vehicle to respond to environmental conditions increases your safety more than it decreases it. Turning on my high beams does take a few brain cycles to flip the switch, but it also lets me spot the deer on the side of the road.
It's almost as if there are legitimate reasons not to have 100% of your attention focused on the road, that aren't simply attempts to rationalize unsafe behavior.