For me it was when I ran out of savings, sold all my stocks, ran out of that money, and still didn't have a shipped product. My cofounders are still going at it, and I wish them all the best, but we spent the critical months of our startup designing and redesigning a thing without ever getting user feedback.

For all the other folks out there at Google or other largish companies, who are considering quitting and forming a startup: Don't do it until you have at least once seen a project through from conception to launch.