This has been a problem since the game launched, the net code just wasn't up to par. They say they've been working on a patch to fix the problem, but I don't think its up yet. Anyway, I was able to still connect eventually, but its almost too frustrating to be worth it.
http://kotaku.com/5045238/castle-crashers-title-update-is-coming-its-coming







