It's not just about insulation. It's also about thermal mass, a.k.a. thermal capacitance a.k.a. heat capacity - how much heat to the falls retain.
In a desert, buildings with thick stone or earthen walls will get cool at night thanks to the cool desert night air. They will stay cool throughout the day and the building's interior will remain cool(er) as a result even in the afternoon. (Conversely, the buildings will remain warm at night for the same reason)
Of course, this is most effective in an environment like a desert where you have a large diurnal temperature swing: scorching days and chilly nights.
But many/all of the same principles can be employed in temperate climates in a modified form, such as with earth-sheltered homes https://en.wikipedia.org/wiki/Earth_shelter
In temperate climates mold is a bigger problem than cooling. Especially when building underground.
Air conditioning mitigates mold by removing moisture from the air…which doesn’t mean that ac ducts won’t hold mold if there’s improper operation or bad design.
Earth homes are uncommon because building science is empirical not theoretical. Because the construction market is efficient even though its efficiency doesn’t mean you can afford a house in a desirable location.
Mold is problem for modern construction techniques. Mold is ubiqutous in the UK, but wasn't a problem for buildings themselves until the advent of contemporary (English) construction techniques. Those hundreds-of-year-old homes built of stone don't have issues with mold (their inhabitants might). It doesn't take much more than ensuring proper airflow to mitigate mold issues in actually temperate climates.
Now, mold in, say, the SE USA (or even the mid-Atlantic region), is a different story, but once again, well built homes there from the 1800s can and have avoided mold issues with good air flow design.
Assuming you can afford to source those better materials. It’s a trade off, thermal quality of materials vs cost/access. We’re heading in a direction where most people will not be able to afford the best insulation and so learning from the past becomes essential.
