If you publish API version 1.0.0, and then internally you fix something and publish 1.0.1 (remember: a patch doesn't change the interface at all), should you continue to serve both the 1.0.0 and 1.0.1 APIs? How are they different from the consumer's perspective? What if your reason to release 1.0.1 is to fix a security issue - in what world is it ethical to continue to serve 1.0.0? If you can discontinue serving 1.0.0 at any time (because of a security patch released with 1.0.1), then you can't offer any long-term durability guarantees for early patch versions, and so indeed, offering older patch versions (when newer patches didn't fix security issues) is more likely to break your consumers when you're forced to discontinue older patch versions for security reasons, than if you refused to serve patch-level variants in the first place.
Because you, as upstream, have no control over whether the pure addition of fields will break downstream (since downstream may or may not be strict about what they accept), you should only offer semantically versioned minor releases if you're willing to guarantee durability for earlier minor versions, and can maintain multiple minor versions of your API in parallel. If not, then be explicit about the potential of your changes to break downstream - use timestamps and document sunset dates.
If you publish API version 1.0.0, and then internally you fix something and publish 1.0.1 (remember: a patch doesn't change the interface at all), should you continue to serve both the 1.0.0 and 1.0.1 APIs? How are they different from the consumer's perspective? What if your reason to release 1.0.1 is to fix a security issue - in what world is it ethical to continue to serve 1.0.0? If you can discontinue serving 1.0.0 at any time (because of a security patch released with 1.0.1), then you can't offer any long-term durability guarantees for early patch versions, and so indeed, offering older patch versions (when newer patches didn't fix security issues) is more likely to break your consumers when you're forced to discontinue older patch versions for security reasons, than if you refused to serve patch-level variants in the first place.
Because you, as upstream, have no control over whether the pure addition of fields will break downstream (since downstream may or may not be strict about what they accept), you should only offer semantically versioned minor releases if you're willing to guarantee durability for earlier minor versions, and can maintain multiple minor versions of your API in parallel. If not, then be explicit about the potential of your changes to break downstream - use timestamps and document sunset dates.