Text-based formats like XML and JSON are good if most of your data is strings, but storing numeric data in decimal strings is both inefficient and inaccurate. That's why I think that HTML is an appropriate use of XML-like markup but perhaps vector images would need a true binary version.
However I agree with saagarjha that it is very easy to write code to emit SVG, which is nice.
I don't think it would really be any more difficult if it were binary. The only real reason to use XML is better integration with HTML, CSS and JavaScript, which is admittedly a compelling reason.
But I think he's right - it would be nice if there was a "proper" binary vector format that was widely supported.
That's a pretty error prone way of writing SVGs though. In well written code you would use a strongly typed API and that API would be exactly the same for a binary format.
Think about how you write Protobuf files for example.
In the circumstances where I actually cared I would use a strongly typed language with nice wrappers, of course ;) The reason why I love SVGs is you can just belt them out from scripts and it's the easiest thing in the world. Same reason why I love GraphViz and Netpbm.
One could conceivably make a binary format based on Messagepack or Flatbuffers for vector data, which would be much more efficient and eliminate rounding problems of converting between floats and decimal numbers in text representations.