That's what finally got me to come to terms with my extreme night owl shift. When I get up earlier than 9am (using light therapy etc), I get anxiety/panic attacks and have a hard time sleeping more than 6 hours. My natural schedule is sleeping around 5am - 1pm and I sleep like a baby and feel great.
Ha! That's just about my perfect schedule as well, though for practicality I try to make sure I'm winding down by 3:00 am so I can get up before noon. In college I'd regularly go get breakfast before heading to bed and marvel at the morning people around me.
My sleep shifts about an hour later every week, so that last week or so before it flips back is pretty tough, especially during the winter. Waking up at like 4 or 5 pm is definitely not good. I still need light!