You have to sort also remember Scott Browns historic win in Massachusetts in Jan 2010 spooked and demoralized democrats and energized Republicans.
So you had a few conservative democrats going against the President and Republicans feeling they can do well in 2010.
That special election had a huge impact.
To many this is ancient history but that was the first major defeat for Obama.