In hindsight I missed a few obvious ones (completely forgot lists), but would have expected HTML 4 tags to still be HTML tags (font, etc.). And I do wonder why SVG is on that list, which can be used inside HTML, but is pretty much not an HTML element, to my knowledge (same with math).
It's a valid HTML tag indicating that its contents are not HTML but instead SVG. The contents of a <script> tag are also not HTML, but we still consider <script> to be an HTML tag.
It's not just the contents; it's the tag itself. The only thing that is special to my knowledge is that the HTML 5 parser understands it and applies the correct XML namespace.
...
Ah, but thinking about it while I type these lines tells me where I'm wrong: It is an HTML element that's by default rendered as a block element, takes part in all the typical layout stuff. I can apply border, border-radius, background, etc. to it and it behaves like a div. Just that I can also put graphics inside.