There is no abstract "government" doing it. It is the work of people who do not want new housing to be built for some reasons. They probably have lots of influence in local governments.
Local governments are small and arguably inefficient, it is easy for special interests such as homeowners to influence them. But there is a long tradition of local government in US and they have real power. The solution here would be for higher-level (state) government to intervene, keeping in mind interests of all state residents and not only homeowners.
Building a house is cheap, having the local infrastructure to support houses is expensive. Letting people build unlimited housing without restriction is a terrible idea.
What we need is a tax on new houses that covers local infrastructure needs. We don't do this because developers have long fed on subsidies of there external costs by local governments.
Property taxes are a poor approximation of external costs even excluding prop 30. Most obviously there paid after the fact and the county needs infrastructure up front.
Consider you add a new subway stop in an area and suddenly people want to replace single family dwellings near it with apartment buildings. The region paid for that subway expansion, but a few well connected developers stand to make a killing. When you get down to it Zoning is really an ideal way for local politicians to shift the benefits of public infrastructure into well connected private hands.
It depends on how the project is funded. Some improvements (such as bringing city water/sewer to an area on the outskirts of town or paving a dirt road) are typically funded via special assessments on the affected property. The local government can then sell bonds secured by the future revenue stream and pass along the financing cost to those who see the benefit.
Subway extensions typically distribute costs over a wider population (the regional/state/federal government paying for large chunks of it). On some level this makes sense. A special assessment on the areas within X feet of a subway station would likely be prohibitively costly for property owners, and the entire region benefits from better transportation infrastructure with more benefits. On the other hand, the person who owns the property across the street from the new subway station is benefiting much more than someone who lives on the other side of town. Perhaps a combination of special assessments and general revenue funding is more appropriate for this scenario? You're certainly correct that the current system transfers a lot of public value into private hands. The Central Subway in SF is going to make some property owners in Chinatown very wealthy.
