Scottie's last point is quite significant here, and i think overlooked. For a sitting incumbent presiden to lose has taken major, major screwups on their part, or a temporary collapse of the existing political order. The only exception i can think of is the first John Adams, who while he did some things that were unpopular, hosted no severe screw-ups on his part.
Others who were defeated:
John Q. Adams. He literally stole the election away from Andrew Jackson by cobbling together an electoral majority through a corrupt bargain, and was bound to lose the next election to him, since he didn't really belong there in the first place.
Martin Van Buren. Collapsed the US economy in 1837
Grover Cleveland: Defied big business at the time of big business' greatest direct influence on American politics with his anti-tariff stance.
Benjamin Harrison: pandered too much to big business and imposed a ruinous tariff.
William Taft: the Republican party eviscerated itself between him and Roosevelt, paving the way for Wilson
Herbert Hooever: collapsed the US economy.
Gerald Ford: pardoned Nixon in what, despite what he said, looked like blatant croneyism.
George Bush: another case of Republican evisceration with Perot.
If Obama were to be challenged from the Left (possible, but highly unlikely), it could pose a problem, or if his financial policies do end up wrecking the country, but they have to do so in the next two years. That's really pretty much it.
Monster Hunter: pissing me off since 2010.