"lets me express myself faster than a couple of quick doodles on a piece of paper."
... and then take a digital photo. Which is the process I've adopted. This is a hard problem to solve with the clumsy digital tools we have at our disposal at the moment. Probably not with this generation of input.
I just use my finger on my tablet. The app recognizes my skewed lines and transforms them into regular forms and connections. Then I can easily export to SVG and finish it up on the computer.
For block diagrams, graphviz is the fastest for me. If I need anything more complicated than node->node relations, pretty much any editor akin to visio is still faster than doing it by hand and looks much prettier because I have infinite space and can rearrange things on the fly. By hand is useful when all I have for collaboration is a whiteboard, though.
I find myself loathing to explain something with a "simple" chart that can take 5 minutes to draw by hand but far more on a computer.