It's unusual and I guess a lot of people actually would find it annoying, but I actually like it. The reason is there're many characters that are actually useful or even required if you intend to do typography "right", but not usually present on keyboard. I prefer "λ" to "lambda" (and I'm not greek), for instance. But I hate working with some abstractions over pure text, I want to see exactly what I typed, so I'm not big fan of all these vim plugins that show "λ" instead of "lambda" in Haskell, etc. God, I hate even tabs, because they look differently depending on their position in the line!

And typing unicode isn't annoying at all after you are used to it. I'm using XCompose all the time, so typing ½ or «» or λ is completely natural to me.