The cost of the building is minor compared to the cost of the wafer equipment. It is not uncommon to build a Fab but only fill it with a fraction of the equipment capacity. Fabs can be scaled up when demand increases.
The equipment is costly (lithography equipment especially) but everything around them is also very expensive, including building. Huge Class 4 clean rooms, water purification, etc. There are lots of chemicals, acids and their processing involved. There are pipes moving liquid helium, nitrogen etc. Semiconductor fab is essentially a chemical plant.
