AM broadcast uses a "band" around the carrier. Whatever tricks you use Shannon's law is going to determine the upper limit of how much data you can carry which depends on the bandwidth and the signal to noise ratio. ML makes no difference from a theoretical perspective. Compression just reduces the bandwidth you'll need assuming your data compresses (with whatever method you choose, ML can be part of that)...
I guess my usual experience is that AM transmissions sound pretty horrible, especially compared to FM
So, my question is more like, can we just upgrade the equipment and keep the channels? And can AM be used more like for signal/indexing rather than full data dump? (eg. instead of transmitting the whole song, transmitting just an id of the song)
AM sounds bad because it’s amplitude modulated. So variation in amplitude (i.e. signal strength) results in noise when decoding. FM however is not sensitive to this issue because its frequency modulated - slight variations in signal strength have no effect on the encoded signal.
And yes, we can always upgrade the equipment. That’s what we did when we transitioned TV to digital signals and FM to digital FM - we reserved some bandwidth for over-the-air while freeing up the rest to be used for other purposes. But it’s an expensive proposition and one that happens rarely (in fact those are the only two times I’m aware of it, they happened at basically the same time, took forever, & there was insane opposition to it).
The limit is Shannon's law which depends on bandwidth and signal to noise. A more advanced modulation technique can get more "information" through up to that limit. A song for this matter can just be considered information as well. So the answer is almost certainly yes, one can get more out of existing channels by upgrading the equipment assuming the current equipment doesn't approach the theoretical limit which is almost certainly the case for this "old" technology we're talking about.
An FM radio channel has a bandwidth of 200Khz. AM radio has a bandwidth of of 10Khz. That's a big part of the reason why AM sounds horrible compared to FM. The way noise impacts AM and FM is also different, especially given human perception. IIRC both these methods were picked for practical reasons not necessarily any optimal criteria. AM is fairly crude, there's only so much you can change the amplitude for any practical purpose. Changing the frequency makes it easier to use more bandwidth so it was a natural evolution. Depending on how you're modulating the amplitude (the naive old school radio IIRC is just using the signal you're transmitting to modulate it) you can make the bandwidth larger or smaller. Random e.g. something like Ethernet on copper can be viewed as a very very fast, high bandwidth, AM. All these things boil down to the same Shannon law just with somewhat different efficiencies depending on how well you can control the frequency domain.
This problem of cramming the most bits into a given channel is something people have been working on for a long time. Some coverage of this here: https://en.wikipedia.org/wiki/Modulation
That's how we got from 300bps modems to 56k modems over the same phone lines way back and how we push more out of radio spectrum today.
One thing you forgot to mention is that in contrast to an FM receiver, or even more a digital one, AM receivers are incredibly easy to build even with very basic "household items", which I believe is at least part of the motivation to provide emergency services and similar broadcasts over AM.
How gratuitous with modern tech. FM needs tens of decibels of signal to noise ratio to sound reasonable, especially if you want stereo. But a digital signal could fit perfect audio into that range at less than one bit per Hz, which would let it tolerate radio noise that's as strong as the actual signal.
