To push him out of office would require impeachment, and that's basically an impossibility, even if there's a significant shift blue. At best there'd be some checks on his power, but he's ultimately still going to be POTUS with all the rights and (denied) responsibilities therein.