I guess its a combination of critical acclaim, public demand, profit, and ease of development. Look at Killzone. Killzone 2 took years to develop, received a ton of acclaim, sold very well. It took a very short amount of time to pump out Killzone 3. Yes, the game is an improvement upon the original but the improvement isn't as pronounced, the amount of acclaim has dwindled a bit, the sales aren't quite as good, but the engine was in place so it was completed fairly quickly. Isuspect many would hope that they take a little time before releasing Killzone 4. This kind of thing happens often this gen.
Examples of games that got it right: Uncharted series, Gears of War series, Forza series, Halo series, Call of Duty series (yes, even though they milk the franchise, fans continue to be excited for the next one and the quaility is still there).
Examples of series that got it wrong: Tony Hawk series, Guitar Hero, Street Fighter 4 (yeah, the game is still good but Capcom is killing/splitting its userbase between too many similar fighting games that we aren't really asking for), Fable series.
Examples of game series that are "on the border": Resistance (R3 will make or break that series), Resident Evil, inFamous (2nd game hasn't been released--hope it's a good one!), Assassins Creed (too much of a good thin?g), Saints Row (again, a great series that will either be received with open arms or overstay its welcome), and BioShock.
It's not easy to strike that balance and I guess a lot of game companies over-estimate the appeal of their franchise and under-estimate how quickly the attention span of gamers move from one thing to another. There's an old saying: "Absence makes the heart grow fonder." How the hell are you supposed to be excited about something coming back if it never leaves?