Wouldn't it be enough to enter an invalid audience when configuring the IDP? If the audience is ignored the sign-in flow still allows you to log on and you know the SP is broken.
Sure, but two reasons that's not quite optimal and you might not want to do that in practice:
1. This tells you the SP is broken; just using individual keys means that doesn't matter anymore if the SP is broken or not. Individual keys is in your control, fixing the SP much less likely so. And you can just set up a practice of doing it for everything and now it's one less thing to test for.
2. That still requires a bit of testing that's somewhat annoying to set up, which most vendorsec practices don't have time for. It's also only one of dozens of things you need to test for. Ignoring audiences is super common, but a more subtle problem is that you can sign a valid SAML assertion _for the wrong domain_, and now you can sign in as a competitor's staff.
As you hint at, having an SP that'll just self-service accept any random metadata.xml at least gives you a fighting chance :)
My question was more related to it being tedious. And now you say it requires a bit of testing which is annoying to set up. Isn't testing this just a matter of changing the audience field to something incorrect and try to sign on? This should take like 2 minutes?
If you just change the audience field, the signature will be invalid, so it might tell you the SP won't accept a bad signature, but it doesn't tell you that the SP would accept a correct-signature-for-wrong-audience assertion. And now we've explored two states in that very big tedium space I mentioned; it still doesn't tell you anything about e.g. canonicalization bugs or cross-domain bugs. Those are much harder to test, because they require your IdP to sign specifically crafted assertions malicious, so you can't test them with your standard Okta install or whatever.
So, sure: you can test this one specific bug by replaying an assertion for a different SP. Or you can make your IdP use new key pairs every time and then you're definitionally immune to the entire bug class forever with every SP. Even if replaying the SP takes 2 minutes, getting the tester to a place where they can exploit it takes way longer for most companies, so it's much more effective to just eliminate entire classes of bugs via policy.
TL;DR: you're right (modulo the amount of time) for this particular bug, but why bother? And if you're going to bother testing, why test for this one specific bug that's cheaper to avoid a different way? (I can think of a reason to test; but then the tedium comes in :))
It's a theoretical yes but a practical no, because the amount of people on this planet who understand SAML enough and want to work on it and are available for auditing, is damn close to zero, if not zero.
This is not an outlandish thing to ask of your security auditor _for your own app_. (It's something we do for clients.)
The challenging part is doing it for vendorsec, when you are vetting _other apps_. The timelines that stakeholders (other people in the company who want to use the app) are willing to accept are like, a week, and even if you somehow had a SAML testing praxis at the ready that enumerates all of the problems SAML has historically had, there's a lot more to test than just the SAML bits.
So: in summary: I don't think that number is anywhere near zero, though sure, it's not huge. The hard part is failures being silent and being in parties you don't control.
