I don't think it would really be any more difficult if it were binary. The only real reason to use XML is better integration with HTML, CSS and JavaScript, which is admittedly a compelling reason.
But I think he's right - it would be nice if there was a "proper" binary vector format that was widely supported.
That's a pretty error prone way of writing SVGs though. In well written code you would use a strongly typed API and that API would be exactly the same for a binary format.
Think about how you write Protobuf files for example.
In the circumstances where I actually cared I would use a strongly typed language with nice wrappers, of course ;) The reason why I love SVGs is you can just belt them out from scripts and it's the easiest thing in the world. Same reason why I love GraphViz and Netpbm.