I think one should be really careful making statements like this.
In the case where an HSM is used for e.g. database encryption, one absolutely still cares.
The attacker can politely ask the HSM to decrypt all the data. On the bright side, you didn't lose the key, and your HSM should at least securely log the fact that all your data just walked out the door.
For SSL offload the situation is less dire, but it is totally realistic that heartbleed might cause issues for a system with HSM assisted offload (e.g. leaking plaintext) because the protocol processing happens in software. Any sane architecture should protect the actual private key though.
In general, the right way to think about HSMs is that they protect your keys, not your data. If you care more about the data than the keys, you absolutely need to care about the security of your host.
I used HSMs for a code signing project, and once someone can borrow your key to sign malicious payloads, it really doesn't matter if they have your signing key or not. You still have to get a new key, and clean up the mess.
As far as I understand heartbleed, the core issue was that you could search memory for the private key and then do whatever you wanted with it. With an HSM, that wouldn't be an issue. However, it's true that you could search for other data as well and that data could be sensitive, just not quite as sensitive (depending on your industry and architecture, of course).
I agree with you -- it depends on your threat model.
If state actors or universal mitm or coffee shop attacks are primary threats then loss of private key is a very bad outcome for SSL offload. And this is what SSL is meant to help woth.
For many (more?) sites though, the issue was loss of session plaintext (e.g api creds, basic auth creds, pii or credit cards). For a lot of applications where just hitting the webserver remotely is a lot easier than MITM, this was actually the more severe problem.
I have nation states as part of my threat model so stealing keys enables passive and active monitoring vs popping a box and getting some limited subset of connections intercepted.
In the case where an HSM is used for e.g. database encryption, one absolutely still cares.
The attacker can politely ask the HSM to decrypt all the data. On the bright side, you didn't lose the key, and your HSM should at least securely log the fact that all your data just walked out the door.
For SSL offload the situation is less dire, but it is totally realistic that heartbleed might cause issues for a system with HSM assisted offload (e.g. leaking plaintext) because the protocol processing happens in software. Any sane architecture should protect the actual private key though.
In general, the right way to think about HSMs is that they protect your keys, not your data. If you care more about the data than the keys, you absolutely need to care about the security of your host.