I've done such things. It is surprisingly not bad if you spend ~$500 on fluorescent breathable rain gear and lights.

The biggest problems are leaves on the stupid green paint they coat the bike lanes with, and increased stopping distance / driver fatigue for the cars.

Storing the rain gear at your destination in a way that lets it dry out is also a big concern.

Being cold is not an issue at all. If anything, you will be too hot while rising.

Realistically speaking, what percent of the commuting population could do this on a daily basis? Many people have additional limitations that also need to be considered. E.g. Have to pick up the kids after work, have to shop after work, age/health constraints, etc.