To be honest if I were Sony if Shadowfall doesn't prove to be successful, and by that I mean at least sale 2,500,000 without bundles then I'd cancel the series and have Guerilla work on something else.
Killzone never became the flagship exclusive Sony was hoping for, I mean none of the games have sold over 3,000,000 and they had great marketing and big ass budgets, but anyways.
Honestly I wouldn't be surprised if Mercenary was better considering another studio is working on it, a studio that used to be R&D or some shit.
Considering that the two PS3 games combined sold enough to pull in 200 to 300 million dollars depending on how many sold at $60, I'd say Sony made their money back and then some on those two games. Few games take much more than 20 to 30 million to make and when we're talking single platform titles, it's typically less. Killzone 2's original budget was 20 million and after delays it eventually ballooned to 45 million. I can assure you, Sony still made a decent profit on that game and Killzone 3 likely made them more despite selling less since its budget didn't become as large as Killzone 2's. The Killzone franchise has established itself as THE FPS series for Sony. Unless a much better FPS is made or Killzone starts losing them money, there is no reason for Sony to stop the franchise.
As far as which I think will do better? Killzone: Mercenary looks like it's going to be a great game but as it's on the Vita and FPS games seem to have a hard time doing well on portable platforms, I'd say Killzone: Shadow Fall is the safer bet for which will be the better game.